2024-05-09T04:55:26.230 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T04:55:26.230 DEBUG:teuthology.run:Teuthology command: teuthology --name pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi --description fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} --verbose --owner scheduled_pdonnell@teuthology --archive /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092 -- /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/orig.config.yaml 2024-05-09T04:55:26.283 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T04:55:26.371 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092 branch: wip-pdonnell-testing-20240508.183908-debug description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} email: pdonnell@redhat.com first_in_suite: false job_id: '7699092' kernel: kdb: 1 sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-pdonnell-testing-20240508.183908-debug ceph: cephfs: max_mds: 5 session_timeout: 300 conf: client: client mount timeout: 600 client_shutdown_timeout: 120 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds oft prefetch dirfrags: true mds op complaint time: 180 mds verify scatter: true mds_dir_prefetch: true mds_sleep_rank_change: 5000000.0 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd blocked scrub grace period: 3600 osd failsafe full ratio: 0.95 osd max markdown count: 1000 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - do not have an application enabled - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - Degraded data redundancy - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(POOL_ - \(CACHE_POOL_ - \(SMALLER_PGP_NUM\) - \(OBJECT_ - \(SLOW_OPS\) - \(REQUEST_SLOW\) - \(TOO_FEW_PGS\) - slow request - timeout on replica - late reservation from - MON_DOWN - OSDMAP_FLAGS - OSD_DOWN - PG_DEGRADED - PG_AVAILABILITY - POOL_APP_NOT_ENABLED - mons down - mon down - out of quorum - overall HEALTH_ - \(MON_DOWN\) - SLOW_OPS - slow request sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs check-counter: counters: mds: - mds.exported - mds.imported install: ceph: flavor: default sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 selinux: allowlist: - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-pdonnell-testing-20240508.183908-debug sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 owner: scheduled_pdonnell@teuthology priority: 90 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - mds.c - mds.e - osd.0 - osd.1 - osd.2 - osd.3 - client.0 - - mon.b - mon.c - mgr.x - mds.b - mds.d - mds.f - osd.4 - osd.5 - osd.6 - osd.7 seed: 1722 sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 sleep_before_teardown: 0 subset: 67/512 suite: fs suite_branch: wip-pdonnell-testing-20240508.183908-debug suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_c9da4049f638c0f84bdd9b4e33c42e86046ac554/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi001.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO/n+q42gdSoJkaKX5+1CUYGh1jjCsWdZwguogGkEOIS8ePxUmO7X1TwMMLkKgSgrTOFI1Ppel1IVYmQLnV3CEk= smithi196.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNaeC+RcduGGUB84dsZNrzFIzXDMcxwZXW7HahP+aGXOIYoBzOzBTwJFEHLXfCWU9DZgB5Ig+oje3Sy4x3PEn4o= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - ceph: create_rbd_pool: false - sequential: - mgrmodules - ceph-fuse: null - mon_thrash: check_mds_failover: true revive_delay: 20 thrash_delay: 10 - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return false end - local function is_kdistro() return false end - local function is_fuse() return true end - local function syntax_version() return '' end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_22:06:20 tube: smithi user: pdonnell verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T04:55:26.371 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_c9da4049f638c0f84bdd9b4e33c42e86046ac554/qa; will attempt to use it 2024-05-09T04:55:26.372 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_c9da4049f638c0f84bdd9b4e33c42e86046ac554/qa/tasks 2024-05-09T04:55:26.372 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T04:55:26.373 INFO:teuthology.task.internal:Checking packages... 2024-05-09T04:55:26.396 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'c9da4049f638c0f84bdd9b4e33c42e86046ac554' 2024-05-09T04:55:26.396 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T04:55:26.396 INFO:teuthology.packaging:ref: None 2024-05-09T04:55:26.396 INFO:teuthology.packaging:tag: None 2024-05-09T04:55:26.397 INFO:teuthology.packaging:branch: wip-pdonnell-testing-20240508.183908-debug 2024-05-09T04:55:26.397 INFO:teuthology.packaging:sha1: c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T04:55:26.397 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-pdonnell-testing-20240508.183908-debug 2024-05-09T04:55:26.615 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3594.gc9da4049 2024-05-09T04:55:26.617 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T04:55:26.623 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T04:55:26.623 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T04:55:26.631 INFO:teuthology.task.internal:Saving configuration 2024-05-09T04:55:26.650 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T04:55:26.656 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T04:55:26.680 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi001.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 04:50:18.416051', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO/n+q42gdSoJkaKX5+1CUYGh1jjCsWdZwguogGkEOIS8ePxUmO7X1TwMMLkKgSgrTOFI1Ppel1IVYmQLnV3CEk='} 2024-05-09T04:55:26.701 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi196.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 04:50:18.418104', 'locked_by': 'scheduled_pdonnell@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNaeC+RcduGGUB84dsZNrzFIzXDMcxwZXW7HahP+aGXOIYoBzOzBTwJFEHLXfCWU9DZgB5Ig+oje3Sy4x3PEn4o='} 2024-05-09T04:55:26.701 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T04:55:26.708 INFO:teuthology.task.internal:roles: ubuntu@smithi001.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'] 2024-05-09T04:55:26.708 INFO:teuthology.task.internal:roles: ubuntu@smithi196.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2024-05-09T04:55:26.708 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T04:55:26.780 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f44271bdf70>, signals=[15]) 2024-05-09T04:55:26.781 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T04:55:26.787 INFO:teuthology.task.internal:Opening connections... 2024-05-09T04:55:26.787 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T04:55:26.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:55:26.868 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T04:55:26.869 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:55:26.942 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T04:55:26.949 DEBUG:teuthology.orchestra.run.smithi001:> uname -m 2024-05-09T04:55:26.968 INFO:teuthology.orchestra.run.smithi001.stdout:x86_64 2024-05-09T04:55:26.968 DEBUG:teuthology.orchestra.run.smithi001:> cat /etc/os-release 2024-05-09T04:55:27.026 INFO:teuthology.orchestra.run.smithi001.stdout:NAME="CentOS Stream" 2024-05-09T04:55:27.026 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION="9" 2024-05-09T04:55:27.026 INFO:teuthology.orchestra.run.smithi001.stdout:ID="centos" 2024-05-09T04:55:27.026 INFO:teuthology.orchestra.run.smithi001.stdout:ID_LIKE="rhel fedora" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:VERSION_ID="9" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:PLATFORM_ID="platform:el9" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:ANSI_COLOR="0;31" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:LOGO="fedora-logo-icon" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:HOME_URL="https://centos.org/" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T04:55:27.027 INFO:teuthology.orchestra.run.smithi001.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T04:55:27.028 INFO:teuthology.lock.ops:Updating smithi001.front.sepia.ceph.com on lock server 2024-05-09T04:55:27.057 DEBUG:teuthology.orchestra.run.smithi196:> uname -m 2024-05-09T04:55:27.075 INFO:teuthology.orchestra.run.smithi196.stdout:x86_64 2024-05-09T04:55:27.076 DEBUG:teuthology.orchestra.run.smithi196:> cat /etc/os-release 2024-05-09T04:55:27.133 INFO:teuthology.orchestra.run.smithi196.stdout:NAME="CentOS Stream" 2024-05-09T04:55:27.133 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION="9" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:ID="centos" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:ID_LIKE="rhel fedora" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:VERSION_ID="9" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:PLATFORM_ID="platform:el9" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:ANSI_COLOR="0;31" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:LOGO="fedora-logo-icon" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:HOME_URL="https://centos.org/" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T04:55:27.134 INFO:teuthology.orchestra.run.smithi196.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T04:55:27.135 INFO:teuthology.lock.ops:Updating smithi196.front.sepia.ceph.com on lock server 2024-05-09T04:55:27.157 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T04:55:27.167 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T04:55:27.172 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T04:55:27.172 DEBUG:teuthology.orchestra.run.smithi001:> test '!' -e /home/ubuntu/cephtest 2024-05-09T04:55:27.175 DEBUG:teuthology.orchestra.run.smithi196:> test '!' -e /home/ubuntu/cephtest 2024-05-09T04:55:27.191 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T04:55:27.197 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T04:55:27.197 DEBUG:teuthology.orchestra.run.smithi001:> test -z $(ls -A /var/lib/ceph) 2024-05-09T04:55:27.233 DEBUG:teuthology.orchestra.run.smithi196:> test -z $(ls -A /var/lib/ceph) 2024-05-09T04:55:27.316 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T04:55:27.372 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T04:55:27.412 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-05-09T04:55:27.412 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': 1, 'sha1': 'distro'}, 'mon.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-05-09T04:55:27.413 DEBUG:teuthology.orchestra.run.smithi001:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T04:55:27.413 DEBUG:teuthology.orchestra.run.smithi196:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T04:55:27.430 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T04:55:27.430 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-05-09T04:55:27.431 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T04:55:27.431 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-05-09T04:55:27.488 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-437.el9.x86_64 2024-05-09T04:55:27.488 INFO:teuthology.task.kernel:Running kernel on smithi196: 5.14.0-437.el9.x86_64 2024-05-09T04:55:27.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum install -y kernel 2024-05-09T04:55:27.489 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-437.el9.x86_64 2024-05-09T04:55:27.489 INFO:teuthology.task.kernel:Running kernel on smithi001: 5.14.0-437.el9.x86_64 2024-05-09T04:55:27.489 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum install -y kernel 2024-05-09T04:55:29.638 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 17 MB/s | 8.1 MB 00:00 2024-05-09T04:55:30.636 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 5.1 MB/s | 8.1 MB 00:01 2024-05-09T04:55:33.105 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 8.1 MB/s | 19 MB 00:02 2024-05-09T04:55:34.187 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 7.8 MB/s | 19 MB 00:02 2024-05-09T04:55:38.680 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 3.5 MB/s | 6.2 MB 00:01 2024-05-09T04:55:40.073 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 72 kB/s | 16 kB 00:00 2024-05-09T04:55:40.903 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.2 MB 00:03 2024-05-09T04:55:41.510 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-05-09T04:55:43.579 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 11 kB/s | 16 kB 00:01 2024-05-09T04:55:45.342 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-05-09T04:55:46.725 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 17 kB/s | 1.7 kB 00:00 2024-05-09T04:55:48.697 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T04:55:48.697 INFO:teuthology.orchestra.run.smithi196.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T04:55:48.799 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-05-09T04:55:48.817 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-05-09T04:55:48.817 INFO:teuthology.orchestra.run.smithi196.stdout: Package Architecture Version Repository Size 2024-05-09T04:55:48.817 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-05-09T04:55:48.817 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-05-09T04:55:48.817 INFO:teuthology.orchestra.run.smithi196.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T04:55:48.817 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-05-09T04:55:48.817 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T04:55:48.817 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T04:55:48.818 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T04:55:48.818 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T04:55:48.818 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-05-09T04:55:48.818 INFO:teuthology.orchestra.run.smithi196.stdout:================================================================================ 2024-05-09T04:55:48.818 INFO:teuthology.orchestra.run.smithi196.stdout:Install 4 Packages 2024-05-09T04:55:48.818 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T04:55:48.818 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 78 M 2024-05-09T04:55:48.819 INFO:teuthology.orchestra.run.smithi196.stdout:Installed size: 123 M 2024-05-09T04:55:48.819 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-05-09T04:55:49.533 INFO:teuthology.orchestra.run.smithi196.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 721 kB/s | 484 kB 00:00 2024-05-09T04:55:50.655 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-05-09T04:55:52.828 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T04:55:52.829 INFO:teuthology.orchestra.run.smithi001.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T04:55:52.929 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-05-09T04:55:52.935 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-05-09T04:55:52.935 INFO:teuthology.orchestra.run.smithi001.stdout: Package Architecture Version Repository Size 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout:================================================================================ 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout:Install 4 Packages 2024-05-09T04:55:52.936 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T04:55:52.937 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 78 M 2024-05-09T04:55:52.937 INFO:teuthology.orchestra.run.smithi001.stdout:Installed size: 123 M 2024-05-09T04:55:52.937 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-05-09T04:55:53.874 INFO:teuthology.orchestra.run.smithi001.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 557 kB/s | 484 kB 00:00 2024-05-09T04:55:54.574 INFO:teuthology.orchestra.run.smithi001.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 9.8 MB/s | 15 MB 00:01 2024-05-09T04:55:55.309 INFO:teuthology.orchestra.run.smithi001.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 15 MB/s | 34 MB 00:02 2024-05-09T04:55:56.308 INFO:teuthology.orchestra.run.smithi001.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 11 MB/s | 28 MB 00:02 2024-05-09T04:55:56.308 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-05-09T04:55:56.308 INFO:teuthology.orchestra.run.smithi001.stdout:Total 23 MB/s | 78 MB 00:03 2024-05-09T04:55:57.016 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-05-09T04:55:57.431 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-05-09T04:55:57.431 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-05-09T04:55:58.411 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-05-09T04:55:58.411 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-05-09T04:56:00.963 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-05-09T04:56:01.611 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T04:56:01.724 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:56:04.173 INFO:teuthology.orchestra.run.smithi196.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 1.0 MB/s | 15 MB 00:15 2024-05-09T04:56:04.515 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:56:04.652 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:56:09.723 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:56:09.763 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:56:14.865 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:56:21.779 INFO:teuthology.orchestra.run.smithi196.stdout:(3/4): kernel-modules-core-5.14.0-444.el9.x86_6 888 kB/s | 28 MB 00:32 2024-05-09T04:56:26.212 INFO:teuthology.orchestra.run.smithi196.stdout:(4/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 930 kB/s | 34 MB 00:37 2024-05-09T04:56:26.212 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-05-09T04:56:26.212 INFO:teuthology.orchestra.run.smithi196.stdout:Total 2.1 MB/s | 78 MB 00:37 2024-05-09T04:56:26.894 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-05-09T04:56:27.221 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-05-09T04:56:27.221 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-05-09T04:56:28.141 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-05-09T04:56:28.141 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-05-09T04:56:30.562 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-05-09T04:56:31.297 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T04:56:31.377 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:56:34.058 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:56:34.186 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:56:38.903 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:56:38.947 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:56:43.697 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:56:45.839 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:56:45.880 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:56:46.188 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:56:46.188 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T04:56:46.188 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:56:46.188 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:56:46.970 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:56:46.970 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T04:56:46.970 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-05-09T04:56:46.971 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T04:56:46.971 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T04:56:46.971 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T04:56:46.971 INFO:teuthology.orchestra.run.smithi001.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T04:56:46.971 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T04:56:46.971 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-05-09T04:56:47.353 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T04:56:47.432 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T04:56:47.433 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T04:56:47.433 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T04:56:47.433 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T04:56:47.433 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T04:56:47.433 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2024-05-09T04:56:47.433 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi001.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T04:56:47.433 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T04:56:47.484 INFO:teuthology.orchestra.run.smithi001.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T04:56:47.485 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T04:56:47.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo rpm -qi grub2-tools 2024-05-09T04:56:47.516 INFO:teuthology.orchestra.run.smithi001.stdout:Name : grub2-tools 2024-05-09T04:56:47.516 INFO:teuthology.orchestra.run.smithi001.stdout:Epoch : 1 2024-05-09T04:56:47.516 INFO:teuthology.orchestra.run.smithi001.stdout:Version : 2.06 2024-05-09T04:56:47.516 INFO:teuthology.orchestra.run.smithi001.stdout:Release : 68.el9 2024-05-09T04:56:47.516 INFO:teuthology.orchestra.run.smithi001.stdout:Architecture: x86_64 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Group : Unspecified 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Size : 8264254 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:License : GPLv3+ 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Packager : builder@centos.org 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Vendor : CentOS 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Summary : Support tools for GRUB. 2024-05-09T04:56:47.517 INFO:teuthology.orchestra.run.smithi001.stdout:Description : 2024-05-09T04:56:47.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T04:56:47.518 INFO:teuthology.orchestra.run.smithi001.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T04:56:47.518 INFO:teuthology.orchestra.run.smithi001.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T04:56:47.518 INFO:teuthology.orchestra.run.smithi001.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T04:56:47.518 INFO:teuthology.orchestra.run.smithi001.stdout:hardware devices. 2024-05-09T04:56:47.518 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T04:56:47.518 INFO:teuthology.orchestra.run.smithi001.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T04:56:47.519 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T04:56:47.519 INFO:teuthology.task.kernel:Updating grub on smithi001 to boot 5.14.0-444.el9.x86_64 2024-05-09T04:56:47.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T04:56:48.159 INFO:teuthology.orchestra.run.smithi001.stderr:Generating grub configuration file ... 2024-05-09T04:56:49.511 INFO:teuthology.orchestra.run.smithi001.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T04:56:49.540 INFO:teuthology.orchestra.run.smithi001.stderr:done 2024-05-09T04:56:49.543 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-05-09T04:56:49.560 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.ZHsweQsrXk 2024-05-09T04:56:49.560 DEBUG:teuthology.orchestra.run.smithi001:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.ZHsweQsrXk 2024-05-09T04:56:49.628 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.ZHsweQsrXk 2024-05-09T04:56:49.769 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.ZHsweQsrXk is 6KB 2024-05-09T04:56:49.790 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.ZHsweQsrXk 2024-05-09T04:56:49.805 DEBUG:teuthology.orchestra.run.smithi001:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T04:56:49.872 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T04:56:49.873 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T04:56:49.873 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T04:56:49.873 INFO:teuthology.orchestra.run.smithi001.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T04:56:49.874 DEBUG:teuthology.orchestra.run.smithi001:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T04:56:50.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo shutdown -r now 2024-05-09T04:57:13.248 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:57:13.287 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:57:13.637 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:57:13.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T04:57:13.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T04:57:13.638 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T04:57:14.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T04:57:14.439 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T04:57:14.439 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-05-09T04:57:14.439 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T04:57:14.439 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T04:57:14.439 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T04:57:14.439 INFO:teuthology.orchestra.run.smithi196.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T04:57:14.439 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T04:57:14.440 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-05-09T04:57:14.793 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T04:57:14.863 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T04:57:14.863 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T04:57:14.863 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T04:57:14.863 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T04:57:14.863 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T04:57:14.863 INFO:teuthology.task.kernel:Installing distro kernel on mon.b... 2024-05-09T04:57:14.863 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi196.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T04:57:14.864 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T04:57:14.913 INFO:teuthology.orchestra.run.smithi196.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T04:57:14.913 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T04:57:14.913 DEBUG:teuthology.orchestra.run.smithi196:> sudo rpm -qi grub2-tools 2024-05-09T04:57:14.944 INFO:teuthology.orchestra.run.smithi196.stdout:Name : grub2-tools 2024-05-09T04:57:14.944 INFO:teuthology.orchestra.run.smithi196.stdout:Epoch : 1 2024-05-09T04:57:14.944 INFO:teuthology.orchestra.run.smithi196.stdout:Version : 2.06 2024-05-09T04:57:14.944 INFO:teuthology.orchestra.run.smithi196.stdout:Release : 68.el9 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Architecture: x86_64 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Group : Unspecified 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Size : 8264254 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:License : GPLv3+ 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Packager : builder@centos.org 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Vendor : CentOS 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Summary : Support tools for GRUB. 2024-05-09T04:57:14.945 INFO:teuthology.orchestra.run.smithi196.stdout:Description : 2024-05-09T04:57:14.946 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T04:57:14.946 INFO:teuthology.orchestra.run.smithi196.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T04:57:14.946 INFO:teuthology.orchestra.run.smithi196.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T04:57:14.946 INFO:teuthology.orchestra.run.smithi196.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T04:57:14.946 INFO:teuthology.orchestra.run.smithi196.stdout:hardware devices. 2024-05-09T04:57:14.946 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T04:57:14.946 INFO:teuthology.orchestra.run.smithi196.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T04:57:14.947 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T04:57:14.947 INFO:teuthology.task.kernel:Updating grub on smithi196 to boot 5.14.0-444.el9.x86_64 2024-05-09T04:57:14.947 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T04:57:15.571 INFO:teuthology.orchestra.run.smithi196.stderr:Generating grub configuration file ... 2024-05-09T04:57:16.915 INFO:teuthology.orchestra.run.smithi196.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T04:57:16.946 INFO:teuthology.orchestra.run.smithi196.stderr:done 2024-05-09T04:57:16.949 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-05-09T04:57:16.964 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.FiqnmLSWth 2024-05-09T04:57:16.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FiqnmLSWth 2024-05-09T04:57:17.031 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.FiqnmLSWth 2024-05-09T04:57:17.166 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.FiqnmLSWth is 6KB 2024-05-09T04:57:17.186 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.FiqnmLSWth 2024-05-09T04:57:17.200 DEBUG:teuthology.orchestra.run.smithi196:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T04:57:17.268 INFO:teuthology.orchestra.run.smithi196.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T04:57:17.268 INFO:teuthology.orchestra.run.smithi196.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T04:57:17.269 INFO:teuthology.orchestra.run.smithi196.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T04:57:17.269 INFO:teuthology.orchestra.run.smithi196.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T04:57:17.270 DEBUG:teuthology.orchestra.run.smithi196:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T04:57:17.407 DEBUG:teuthology.orchestra.run.smithi196:> sudo shutdown -r now 2024-05-09T04:57:20.030 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2024-05-09T04:57:20.031 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-05-09T04:57:20.031 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:57:38.591 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-05-09T04:57:47.418 INFO:teuthology.task.kernel:Checking client mon.b for new kernel version... 2024-05-09T04:57:47.418 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-05-09T04:57:47.419 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:57:47.592 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-05-09T04:57:47.592 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:57:50.658 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.1 2024-05-09T04:58:02.671 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-05-09T04:58:02.671 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:58:05.982 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-05-09T04:58:14.986 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-05-09T04:58:14.987 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:58:18.046 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-05-09T04:58:30.059 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-05-09T04:58:30.060 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:58:34.603 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-05-09T04:58:35.166 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-05-09T04:58:35.166 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "5.14.0-444.el9.x86_64"... 2024-05-09T04:58:35.166 DEBUG:teuthology.orchestra.run.smithi001:> uname -r 2024-05-09T04:58:35.182 INFO:teuthology.orchestra.run.smithi001.stdout:5.14.0-444.el9.x86_64 2024-05-09T04:58:35.183 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T04:58:35.183 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T04:58:35.183 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T04:58:36.184 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2024-05-09T04:58:36.184 DEBUG:teuthology.orchestra.run.smithi001:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T04:58:36.259 INFO:teuthology.orchestra.run.smithi001.stdout:ttyS1 2024-05-09T04:58:36.298 DEBUG:teuthology.parallel:result is None 2024-05-09T04:58:48.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.196 2024-05-09T04:59:03.479 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-05-09T04:59:03.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T04:59:03.853 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-05-09T04:59:04.327 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-05-09T04:59:04.327 INFO:teuthology.task.kernel:Checking kernel version of mon.b, want "5.14.0-444.el9.x86_64"... 2024-05-09T04:59:04.327 DEBUG:teuthology.orchestra.run.smithi196:> uname -r 2024-05-09T04:59:04.342 INFO:teuthology.orchestra.run.smithi196.stdout:5.14.0-444.el9.x86_64 2024-05-09T04:59:04.342 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T04:59:04.342 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T04:59:04.342 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T04:59:05.343 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2024-05-09T04:59:05.344 DEBUG:teuthology.orchestra.run.smithi196:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-05-09T04:59:05.469 INFO:teuthology.orchestra.run.smithi196.stdout:ttyS1 2024-05-09T04:59:05.581 DEBUG:teuthology.parallel:result is None 2024-05-09T04:59:05.582 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T04:59:05.589 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T04:59:05.589 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T04:59:05.592 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T04:59:05.609 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T04:59:05.635 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T04:59:05.670 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T04:59:05.670 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T04:59:05.672 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T04:59:05.733 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T04:59:05.740 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T04:59:05.740 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T04:59:05.750 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T04:59:05.782 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T04:59:05.794 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T04:59:06.067 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T04:59:06.083 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T04:59:06.086 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T04:59:06.093 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T04:59:06.093 DEBUG:teuthology.orchestra.run.smithi001:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T04:59:06.096 DEBUG:teuthology.orchestra.run.smithi196:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T04:59:06.146 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T04:59:06.156 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T04:59:06.156 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T04:59:06.166 DEBUG:teuthology.orchestra.run.smithi196:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T04:59:06.196 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T04:59:06.269 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T04:59:06.323 DEBUG:teuthology.orchestra.run.smithi001:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T04:59:06.407 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T04:59:06.451 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T04:59:06.451 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T04:59:06.519 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T04:59:06.575 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T04:59:06.648 DEBUG:teuthology.orchestra.run.smithi196:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T04:59:06.702 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T04:59:06.743 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T04:59:06.744 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T04:59:06.808 DEBUG:teuthology.orchestra.run.smithi001:> sudo service rsyslog restart 2024-05-09T04:59:06.810 DEBUG:teuthology.orchestra.run.smithi196:> sudo service rsyslog restart 2024-05-09T04:59:06.849 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T04:59:06.896 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T04:59:07.225 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T04:59:07.232 INFO:teuthology.task.internal:Starting timer... 2024-05-09T04:59:07.232 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T04:59:07.242 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T04:59:07.250 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:getty_t:s0']} 2024-05-09T04:59:07.251 DEBUG:teuthology.orchestra.run.smithi001:> sudo service auditd rotate 2024-05-09T04:59:07.343 INFO:teuthology.orchestra.run.smithi001.stdout:Rotating logs: 2024-05-09T04:59:07.345 DEBUG:teuthology.orchestra.run.smithi196:> sudo service auditd rotate 2024-05-09T04:59:07.404 INFO:teuthology.orchestra.run.smithi196.stdout:Rotating logs: 2024-05-09T04:59:07.406 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T04:59:07.406 DEBUG:teuthology.orchestra.run.smithi001:> /usr/sbin/getenforce 2024-05-09T04:59:07.444 INFO:teuthology.orchestra.run.smithi001.stdout:Permissive 2024-05-09T04:59:07.444 DEBUG:teuthology.orchestra.run.smithi196:> /usr/sbin/getenforce 2024-05-09T04:59:07.471 INFO:teuthology.orchestra.run.smithi196.stdout:Permissive 2024-05-09T04:59:07.471 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi001.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi196.front.sepia.ceph.com': 'permissive'} 2024-05-09T04:59:07.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T04:59:07.514 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T04:59:07.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T04:59:07.541 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T04:59:07.541 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T04:59:07.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo /usr/sbin/setenforce permissive 2024-05-09T04:59:07.591 DEBUG:teuthology.orchestra.run.smithi196:> sudo /usr/sbin/setenforce permissive 2024-05-09T04:59:07.632 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T04:59:07.640 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-05-09T04:59:07.645 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-05-09T04:59:07.769 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T04:59:07.783 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T04:59:07.784 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi001.front.sepia.ceph.com,smithi196.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T05:03:15.767 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi001.front.sepia.ceph.com'), Remote(name='ubuntu@smithi196.front.sepia.ceph.com')] 2024-05-09T05:03:15.769 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-05-09T05:03:15.770 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi001.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T05:03:15.851 DEBUG:teuthology.orchestra.run.smithi001:> true 2024-05-09T05:03:15.935 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi001.front.sepia.ceph.com' 2024-05-09T05:03:15.935 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-05-09T05:03:15.936 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi196.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T05:03:16.053 DEBUG:teuthology.orchestra.run.smithi196:> true 2024-05-09T05:03:16.130 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi196.front.sepia.ceph.com' 2024-05-09T05:03:16.131 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T05:03:16.141 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T05:03:16.142 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T05:03:16.142 DEBUG:teuthology.orchestra.run.smithi001:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T05:03:16.144 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T05:03:16.145 DEBUG:teuthology.orchestra.run.smithi196:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T05:03:16.173 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T05:03:16.188 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T05:03:16.200 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T05:03:16.213 INFO:teuthology.orchestra.run.smithi001.stderr:sudo: ntpd: command not found 2024-05-09T05:03:16.215 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T05:03:16.225 INFO:teuthology.orchestra.run.smithi001.stdout:506 Cannot talk to daemon 2024-05-09T05:03:16.239 INFO:teuthology.orchestra.run.smithi196.stderr:sudo: ntpd: command not found 2024-05-09T05:03:16.239 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T05:03:16.249 INFO:teuthology.orchestra.run.smithi196.stdout:506 Cannot talk to daemon 2024-05-09T05:03:16.254 INFO:teuthology.orchestra.run.smithi001.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T05:03:16.263 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T05:03:16.277 INFO:teuthology.orchestra.run.smithi196.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T05:03:16.314 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-05-09T05:03:16.317 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T05:03:16.317 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-05-09T05:03:16.318 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:03:16.318 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:03:16.318 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:03:16.318 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:03:16.340 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-05-09T05:03:16.343 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T05:03:16.344 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-05-09T05:03:16.344 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:03:16.344 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:03:16.344 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:03:16.344 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:03:16.345 INFO:teuthology.run_tasks:Running task install... 2024-05-09T05:03:16.353 DEBUG:teuthology.task.install:project ceph 2024-05-09T05:03:16.353 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}} 2024-05-09T05:03:16.354 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'} 2024-05-09T05:03:16.354 INFO:teuthology.task.install:Using flavor: default 2024-05-09T05:03:16.360 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T05:03:16.360 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-09T05:03:16.361 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'tag': None, 'wait_for_package': False} 2024-05-09T05:03:16.361 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T05:03:16.364 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'tag': None, 'wait_for_package': False} 2024-05-09T05:03:16.364 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T05:03:16.524 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/ 2024-05-09T05:03:16.524 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3594.gc9da4049 2024-05-09T05:03:16.594 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/ 2024-05-09T05:03:16.594 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3594.gc9da4049 2024-05-09T05:03:16.644 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T05:03:16.644 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:03:16.644 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T05:03:16.671 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T05:03:16.672 DEBUG:teuthology.orchestra.run.smithi196:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c9da4049f638c0f84bdd9b4e33c42e86046ac554/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T05:03:16.739 DEBUG:teuthology.orchestra.run.smithi196:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T05:03:16.764 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-pdonnell-testing-20240508.183908-debug/c9da4049f638c0f84bdd9b4e33c42e86046ac554/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T05:03:16.765 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:03:16.765 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T05:03:16.792 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T05:03:16.792 DEBUG:teuthology.orchestra.run.smithi001:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/c9da4049f638c0f84bdd9b4e33c42e86046ac554/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T05:03:16.815 DEBUG:teuthology.orchestra.run.smithi196:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T05:03:16.859 DEBUG:teuthology.orchestra.run.smithi001:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T05:03:16.880 INFO:teuthology.orchestra.run.smithi196.stdout:check_obsoletes = 1 2024-05-09T05:03:16.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum clean all 2024-05-09T05:03:16.935 DEBUG:teuthology.orchestra.run.smithi001:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T05:03:17.001 INFO:teuthology.orchestra.run.smithi001.stdout:check_obsoletes = 1 2024-05-09T05:03:17.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum clean all 2024-05-09T05:03:17.113 INFO:teuthology.orchestra.run.smithi196.stdout:54 files removed 2024-05-09T05:03:17.140 DEBUG:teuthology.orchestra.run.smithi196:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T05:03:17.249 INFO:teuthology.orchestra.run.smithi001.stdout:54 files removed 2024-05-09T05:03:17.276 DEBUG:teuthology.orchestra.run.smithi001:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T05:03:17.740 INFO:teuthology.orchestra.run.smithi196.stdout:ceph packages for x86_64 253 kB/s | 84 kB 00:00 2024-05-09T05:03:17.970 INFO:teuthology.orchestra.run.smithi001.stdout:ceph packages for x86_64 205 kB/s | 84 kB 00:00 2024-05-09T05:03:18.116 INFO:teuthology.orchestra.run.smithi196.stdout:ceph noarch packages 36 kB/s | 12 kB 00:00 2024-05-09T05:03:18.429 INFO:teuthology.orchestra.run.smithi001.stdout:ceph noarch packages 29 kB/s | 12 kB 00:00 2024-05-09T05:03:18.450 INFO:teuthology.orchestra.run.smithi196.stdout:ceph source packages 6.5 kB/s | 1.9 kB 00:00 2024-05-09T05:03:18.813 INFO:teuthology.orchestra.run.smithi001.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-05-09T05:03:19.596 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-05-09T05:03:20.117 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - BaseOS 5.0 MB/s | 8.1 MB 00:01 2024-05-09T05:03:22.101 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-05-09T05:03:22.314 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-05-09T05:03:26.994 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.2 MB 00:01 2024-05-09T05:03:28.369 INFO:teuthology.orchestra.run.smithi196.stdout:CentOS Stream 9 - Extras packages 89 kB/s | 16 kB 00:00 2024-05-09T05:03:29.165 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - CRB 2.0 MB/s | 6.2 MB 00:03 2024-05-09T05:03:29.482 INFO:teuthology.orchestra.run.smithi196.stdout:Extra Packages for Enterprise Linux 22 MB/s | 21 MB 00:00 2024-05-09T05:03:30.875 INFO:teuthology.orchestra.run.smithi001.stdout:CentOS Stream 9 - Extras packages 34 kB/s | 16 kB 00:00 2024-05-09T05:03:34.630 INFO:teuthology.orchestra.run.smithi196.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-05-09T05:03:34.693 INFO:teuthology.orchestra.run.smithi001.stdout:Extra Packages for Enterprise Linux 5.7 MB/s | 21 MB 00:03 2024-05-09T05:03:36.998 INFO:teuthology.orchestra.run.smithi196.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T05:03:36.999 INFO:teuthology.orchestra.run.smithi196.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T05:03:37.010 INFO:teuthology.orchestra.run.smithi196.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T05:03:37.011 INFO:teuthology.orchestra.run.smithi196.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T05:03:37.011 INFO:teuthology.orchestra.run.smithi196.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T05:03:37.012 INFO:teuthology.orchestra.run.smithi196.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T05:03:37.014 INFO:teuthology.orchestra.run.smithi196.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T05:03:37.016 INFO:teuthology.orchestra.run.smithi196.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T05:03:37.017 INFO:teuthology.orchestra.run.smithi196.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T05:03:37.080 INFO:teuthology.orchestra.run.smithi196.stdout:Dependencies resolved. 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: Package Arch Version Repository Size 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout:Installing: 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: ceph x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 6.6 k 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 5.4 M 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 863 k 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 147 k 2024-05-09T05:03:37.099 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 1.5 M 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 142 k 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 3.5 M 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 7.4 M 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 49 k 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 10 M 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 43 M 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 273 k 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 763 k 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-mirror x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 224 k 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-shell x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 27 k 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-top noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 18 k 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T05:03:37.100 INFO:teuthology.orchestra.run.smithi196.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 32 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 756 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 127 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 162 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 321 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 301 k 2024-05-09T05:03:37.101 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 99 k 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 86 k 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.0 M 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 169 k 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrading: 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: librados2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.3 M 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.1 M 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T05:03:37.102 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout:Installing dependencies: 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 20 M 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 34 k 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 2.3 M 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 274 k 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 4.9 M 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 18 M 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 16 k 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 25 k 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T05:03:37.103 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 158 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 497 k 2024-05-09T05:03:37.104 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 5.2 M 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T05:03:37.105 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 45 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 135 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T05:03:37.106 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T05:03:37.107 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T05:03:37.108 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T05:03:37.109 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T05:03:37.110 INFO:teuthology.orchestra.run.smithi196.stdout:Installing weak dependencies: 2024-05-09T05:03:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T05:03:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:03:37.111 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction Summary 2024-05-09T05:03:37.111 INFO:teuthology.orchestra.run.smithi196.stdout:======================================================================================= 2024-05-09T05:03:37.111 INFO:teuthology.orchestra.run.smithi196.stdout:Install 148 Packages 2024-05-09T05:03:37.111 INFO:teuthology.orchestra.run.smithi196.stdout:Upgrade 8 Packages 2024-05-09T05:03:37.111 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:03:37.112 INFO:teuthology.orchestra.run.smithi196.stdout:Total download size: 219 M 2024-05-09T05:03:37.112 INFO:teuthology.orchestra.run.smithi196.stdout:Downloading Packages: 2024-05-09T05:03:37.772 INFO:teuthology.orchestra.run.smithi196.stdout:(1/156): ceph-19.0.0-3594.gc9da4049.el9.x86_64. 31 kB/s | 6.6 kB 00:00 2024-05-09T05:03:38.098 INFO:teuthology.orchestra.run.smithi196.stdout:(2/156): ceph-fuse-19.0.0-3594.gc9da4049.el9.x8 2.6 MB/s | 863 kB 00:00 2024-05-09T05:03:38.214 INFO:teuthology.orchestra.run.smithi196.stdout:(3/156): ceph-immutable-object-cache-19.0.0-359 1.2 MB/s | 147 kB 00:00 2024-05-09T05:03:38.304 INFO:teuthology.orchestra.run.smithi196.stdout:(4/156): ceph-base-19.0.0-3594.gc9da4049.el9.x8 7.3 MB/s | 5.4 MB 00:00 2024-05-09T05:03:38.473 INFO:teuthology.orchestra.run.smithi196.stdout:(5/156): ceph-mgr-19.0.0-3594.gc9da4049.el9.x86 8.9 MB/s | 1.5 MB 00:00 2024-05-09T05:03:38.523 INFO:teuthology.orchestra.run.smithi196.stdout:(6/156): ceph-mds-19.0.0-3594.gc9da4049.el9.x86 7.3 MB/s | 2.3 MB 00:00 2024-05-09T05:03:38.899 INFO:teuthology.orchestra.run.smithi196.stdout:(7/156): ceph-mon-19.0.0-3594.gc9da4049.el9.x86 11 MB/s | 4.9 MB 00:00 2024-05-09T05:03:39.918 INFO:teuthology.orchestra.run.smithi196.stdout:(8/156): ceph-radosgw-19.0.0-3594.gc9da4049.el9 9.9 MB/s | 10 MB 00:01 2024-05-09T05:03:39.999 INFO:teuthology.orchestra.run.smithi196.stdout:(9/156): ceph-selinux-19.0.0-3594.gc9da4049.el9 313 kB/s | 25 kB 00:00 2024-05-09T05:03:40.009 INFO:teuthology.orchestra.run.smithi001.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-05-09T05:03:40.500 INFO:teuthology.orchestra.run.smithi196.stdout:(10/156): ceph-osd-19.0.0-3594.gc9da4049.el9.x8 9.0 MB/s | 18 MB 00:01 2024-05-09T05:03:40.683 INFO:teuthology.orchestra.run.smithi196.stdout:(11/156): ceph-common-19.0.0-3594.gc9da4049.el9 6.3 MB/s | 20 MB 00:03 2024-05-09T05:03:40.708 INFO:teuthology.orchestra.run.smithi196.stdout:(12/156): cephfs-mirror-19.0.0-3594.gc9da4049.e 1.0 MB/s | 224 kB 00:00 2024-05-09T05:03:40.734 INFO:teuthology.orchestra.run.smithi196.stdout:(13/156): cephfs-shell-19.0.0-3594.gc9da4049.el 541 kB/s | 27 kB 00:00 2024-05-09T05:03:40.759 INFO:teuthology.orchestra.run.smithi196.stdout:(14/156): libcephfs-devel-19.0.0-3594.gc9da4049 631 kB/s | 32 kB 00:00 2024-05-09T05:03:40.842 INFO:teuthology.orchestra.run.smithi196.stdout:(15/156): libcephsqlite-19.0.0-3594.gc9da4049.e 1.8 MB/s | 158 kB 00:00 2024-05-09T05:03:40.876 INFO:teuthology.orchestra.run.smithi196.stdout:(16/156): libcephfs2-19.0.0-3594.gc9da4049.el9. 5.2 MB/s | 756 kB 00:00 2024-05-09T05:03:40.918 INFO:teuthology.orchestra.run.smithi196.stdout:(17/156): librados-devel-19.0.0-3594.gc9da4049. 1.6 MB/s | 127 kB 00:00 2024-05-09T05:03:40.968 INFO:teuthology.orchestra.run.smithi196.stdout:(18/156): libradosstriper1-19.0.0-3594.gc9da404 5.3 MB/s | 497 kB 00:00 2024-05-09T05:03:41.060 INFO:teuthology.orchestra.run.smithi196.stdout:(19/156): python3-ceph-argparse-19.0.0-3594.gc9 492 kB/s | 45 kB 00:00 2024-05-09T05:03:41.152 INFO:teuthology.orchestra.run.smithi196.stdout:(20/156): python3-ceph-common-19.0.0-3594.gc9da 1.4 MB/s | 135 kB 00:00 2024-05-09T05:03:41.253 INFO:teuthology.orchestra.run.smithi196.stdout:(21/156): python3-cephfs-19.0.0-3594.gc9da4049. 1.6 MB/s | 162 kB 00:00 2024-05-09T05:03:41.353 INFO:teuthology.orchestra.run.smithi196.stdout:(22/156): python3-rados-19.0.0-3594.gc9da4049.e 3.1 MB/s | 321 kB 00:00 2024-05-09T05:03:41.445 INFO:teuthology.orchestra.run.smithi196.stdout:(23/156): librgw2-19.0.0-3594.gc9da4049.el9.x86 9.8 MB/s | 5.2 MB 00:00 2024-05-09T05:03:41.470 INFO:teuthology.orchestra.run.smithi196.stdout:(24/156): python3-rbd-19.0.0-3594.gc9da4049.el9 2.5 MB/s | 301 kB 00:00 2024-05-09T05:03:41.504 INFO:teuthology.orchestra.run.smithi196.stdout:(25/156): python3-rgw-19.0.0-3594.gc9da4049.el9 1.7 MB/s | 99 kB 00:00 2024-05-09T05:03:41.546 INFO:teuthology.orchestra.run.smithi196.stdout:(26/156): rbd-fuse-19.0.0-3594.gc9da4049.el9.x8 1.1 MB/s | 86 kB 00:00 2024-05-09T05:03:41.638 INFO:teuthology.orchestra.run.smithi196.stdout:(27/156): rbd-nbd-19.0.0-3594.gc9da4049.el9.x86 1.8 MB/s | 169 kB 00:00 2024-05-09T05:03:41.721 INFO:teuthology.orchestra.run.smithi196.stdout:(28/156): ceph-grafana-dashboards-19.0.0-3594.g 410 kB/s | 34 kB 00:00 2024-05-09T05:03:41.805 INFO:teuthology.orchestra.run.smithi196.stdout:(29/156): rbd-mirror-19.0.0-3594.gc9da4049.el9. 10 MB/s | 3.0 MB 00:00 2024-05-09T05:03:41.830 INFO:teuthology.orchestra.run.smithi196.stdout:(30/156): ceph-mgr-cephadm-19.0.0-3594.gc9da404 1.3 MB/s | 142 kB 00:00 2024-05-09T05:03:42.239 INFO:teuthology.orchestra.run.smithi196.stdout:(31/156): ceph-mgr-dashboard-19.0.0-3594.gc9da4 8.2 MB/s | 3.5 MB 00:00 2024-05-09T05:03:42.339 INFO:teuthology.orchestra.run.smithi196.stdout:(32/156): ceph-mgr-modules-core-19.0.0-3594.gc9 2.7 MB/s | 274 kB 00:00 2024-05-09T05:03:42.431 INFO:teuthology.orchestra.run.smithi196.stdout:(33/156): ceph-mgr-rook-19.0.0-3594.gc9da4049.e 538 kB/s | 49 kB 00:00 2024-05-09T05:03:42.470 INFO:teuthology.orchestra.run.smithi001.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T05:03:42.471 INFO:teuthology.orchestra.run.smithi001.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T05:03:42.482 INFO:teuthology.orchestra.run.smithi001.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T05:03:42.483 INFO:teuthology.orchestra.run.smithi001.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T05:03:42.484 INFO:teuthology.orchestra.run.smithi001.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T05:03:42.484 INFO:teuthology.orchestra.run.smithi001.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T05:03:42.486 INFO:teuthology.orchestra.run.smithi001.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T05:03:42.488 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T05:03:42.489 INFO:teuthology.orchestra.run.smithi001.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T05:03:42.514 INFO:teuthology.orchestra.run.smithi196.stdout:(34/156): ceph-prometheus-alerts-19.0.0-3594.gc 198 kB/s | 16 kB 00:00 2024-05-09T05:03:42.555 INFO:teuthology.orchestra.run.smithi001.stdout:Dependencies resolved. 2024-05-09T05:03:42.572 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-05-09T05:03:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: Package Arch Version Repository Size 2024-05-09T05:03:42.572 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-05-09T05:03:42.572 INFO:teuthology.orchestra.run.smithi001.stdout:Installing: 2024-05-09T05:03:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T05:03:42.572 INFO:teuthology.orchestra.run.smithi001.stdout: ceph x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 6.6 k 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 5.4 M 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 863 k 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 147 k 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 1.5 M 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 142 k 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 3.5 M 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 7.4 M 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 49 k 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 10 M 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 43 M 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 273 k 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 763 k 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 224 k 2024-05-09T05:03:42.573 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 27 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 18 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 32 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 756 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 127 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T05:03:42.574 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 162 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 321 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 301 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 99 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 86 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.0 M 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 169 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrading: 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T05:03:42.575 INFO:teuthology.orchestra.run.smithi001.stdout: librados2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.3 M 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 3.1 M 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout:Installing dependencies: 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 20 M 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 34 k 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 2.3 M 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 274 k 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 4.9 M 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 18 M 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3594.gc9da4049.el9 ceph-noarch 16 k 2024-05-09T05:03:42.576 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 25 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 158 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T05:03:42.577 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 497 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2 x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 5.2 M 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T05:03:42.578 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T05:03:42.579 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 45 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common x86_64 2:19.0.0-3594.gc9da4049.el9 ceph 135 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T05:03:42.580 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T05:03:42.581 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T05:03:42.582 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T05:03:42.583 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T05:03:42.584 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T05:03:42.585 INFO:teuthology.orchestra.run.smithi001.stdout:Installing weak dependencies: 2024-05-09T05:03:42.585 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T05:03:42.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:03:42.585 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction Summary 2024-05-09T05:03:42.585 INFO:teuthology.orchestra.run.smithi001.stdout:======================================================================================= 2024-05-09T05:03:42.585 INFO:teuthology.orchestra.run.smithi001.stdout:Install 148 Packages 2024-05-09T05:03:42.585 INFO:teuthology.orchestra.run.smithi001.stdout:Upgrade 8 Packages 2024-05-09T05:03:42.585 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:03:42.586 INFO:teuthology.orchestra.run.smithi001.stdout:Total download size: 219 M 2024-05-09T05:03:42.586 INFO:teuthology.orchestra.run.smithi001.stdout:Downloading Packages: 2024-05-09T05:03:42.606 INFO:teuthology.orchestra.run.smithi196.stdout:(35/156): ceph-volume-19.0.0-3594.gc9da4049.el9 2.9 MB/s | 273 kB 00:00 2024-05-09T05:03:42.723 INFO:teuthology.orchestra.run.smithi196.stdout:(36/156): cephadm-19.0.0-3594.gc9da4049.el9.noa 6.4 MB/s | 763 kB 00:00 2024-05-09T05:03:42.807 INFO:teuthology.orchestra.run.smithi196.stdout:(37/156): cephfs-top-19.0.0-3594.gc9da4049.el9. 218 kB/s | 18 kB 00:00 2024-05-09T05:03:43.183 INFO:teuthology.orchestra.run.smithi196.stdout:(38/156): ceph-test-19.0.0-3594.gc9da4049.el9.x 14 MB/s | 43 MB 00:03 2024-05-09T05:03:43.274 INFO:teuthology.orchestra.run.smithi196.stdout:(39/156): ceph-mgr-diskprediction-local-19.0.0- 5.1 MB/s | 7.4 MB 00:01 2024-05-09T05:03:43.299 INFO:teuthology.orchestra.run.smithi196.stdout:(40/156): fuse-2.9.9-15.el9.x86_64.rpm 163 kB/s | 80 kB 00:00 2024-05-09T05:03:43.408 INFO:teuthology.orchestra.run.smithi196.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-05-09T05:03:43.433 INFO:teuthology.orchestra.run.smithi196.stdout:(42/156): libquadmath-11.4.1-3.el9.x86_64.rpm 1.2 MB/s | 195 kB 00:00 2024-05-09T05:03:43.458 INFO:teuthology.orchestra.run.smithi196.stdout:(43/156): libgfortran-11.4.1-3.el9.x86_64.rpm 2.8 MB/s | 803 kB 00:00 2024-05-09T05:03:43.484 INFO:teuthology.orchestra.run.smithi196.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-09T05:03:43.526 INFO:teuthology.orchestra.run.smithi196.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-09T05:03:43.553 INFO:teuthology.orchestra.run.smithi001.stdout:(1/156): ceph-19.0.0-3594.gc9da4049.el9.x86_64. 29 kB/s | 6.6 kB 00:00 2024-05-09T05:03:43.559 INFO:teuthology.orchestra.run.smithi196.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.6 MB/s | 125 kB 00:00 2024-05-09T05:03:43.584 INFO:teuthology.orchestra.run.smithi196.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 3.6 MB/s | 215 kB 00:00 2024-05-09T05:03:43.627 INFO:teuthology.orchestra.run.smithi196.stdout:(48/156): python3-cryptography-36.0.1-4.el9.x86 5.6 MB/s | 1.2 MB 00:00 2024-05-09T05:03:43.752 INFO:teuthology.orchestra.run.smithi196.stdout:(49/156): boost-program-options-1.75.0-8.el9.x8 856 kB/s | 107 kB 00:00 2024-05-09T05:03:43.802 INFO:teuthology.orchestra.run.smithi196.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 4.2 MB/s | 943 kB 00:00 2024-05-09T05:03:43.828 INFO:teuthology.orchestra.run.smithi196.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 286 kB/s | 21 kB 00:00 2024-05-09T05:03:43.878 INFO:teuthology.orchestra.run.smithi196.stdout:(52/156): automake-1.16.2-8.el9.noarch.rpm 2.1 MB/s | 689 kB 00:00 2024-05-09T05:03:43.904 INFO:teuthology.orchestra.run.smithi196.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 435 kB/s | 33 kB 00:00 2024-05-09T05:03:43.954 INFO:teuthology.orchestra.run.smithi196.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 2.0 MB/s | 313 kB 00:00 2024-05-09T05:03:43.979 INFO:teuthology.orchestra.run.smithi196.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 242 kB/s | 18 kB 00:00 2024-05-09T05:03:44.046 INFO:teuthology.orchestra.run.smithi196.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 121 kB/s | 11 kB 00:00 2024-05-09T05:03:44.071 INFO:teuthology.orchestra.run.smithi196.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.7 MB/s | 160 kB 00:00 2024-05-09T05:03:44.129 INFO:teuthology.orchestra.run.smithi196.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 543 kB/s | 45 kB 00:00 2024-05-09T05:03:44.238 INFO:teuthology.orchestra.run.smithi196.stdout:(59/156): libtool-2.4.6-45.el9.x86_64.rpm 5.2 MB/s | 582 kB 00:00 2024-05-09T05:03:44.280 INFO:teuthology.orchestra.run.smithi196.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 3.1 MB/s | 662 kB 00:00 2024-05-09T05:03:44.305 INFO:teuthology.orchestra.run.smithi196.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 381 kB/s | 25 kB 00:00 2024-05-09T05:03:44.372 INFO:teuthology.orchestra.run.smithi196.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 517 kB/s | 47 kB 00:00 2024-05-09T05:03:44.456 INFO:teuthology.orchestra.run.smithi196.stdout:(63/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 5.2 MB/s | 3.0 MB 00:00 2024-05-09T05:03:44.481 INFO:teuthology.orchestra.run.smithi196.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.6 MB/s | 292 kB 00:00 2024-05-09T05:03:44.486 INFO:teuthology.orchestra.run.smithi001.stdout:(2/156): ceph-common-19.0.0-3594.gc9da4049.el9. 17 MB/s | 20 MB 00:01 2024-05-09T05:03:44.506 INFO:teuthology.orchestra.run.smithi196.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.4 MB/s | 188 kB 00:00 2024-05-09T05:03:44.530 INFO:teuthology.orchestra.run.smithi001.stdout:(3/156): ceph-fuse-19.0.0-3594.gc9da4049.el9.x8 885 kB/s | 863 kB 00:00 2024-05-09T05:03:44.532 INFO:teuthology.orchestra.run.smithi196.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 467 kB/s | 35 kB 00:00 2024-05-09T05:03:44.553 INFO:teuthology.orchestra.run.smithi001.stdout:(4/156): ceph-immutable-object-cache-19.0.0-359 2.1 MB/s | 147 kB 00:00 2024-05-09T05:03:44.654 INFO:teuthology.orchestra.run.smithi001.stdout:(5/156): ceph-mgr-19.0.0-3594.gc9da4049.el9.x86 15 MB/s | 1.5 MB 00:00 2024-05-09T05:03:44.774 INFO:teuthology.orchestra.run.smithi196.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 16 MB/s | 4.7 MB 00:00 2024-05-09T05:03:44.937 INFO:teuthology.orchestra.run.smithi001.stdout:(6/156): ceph-mon-19.0.0-3594.gc9da4049.el9.x86 17 MB/s | 4.9 MB 00:00 2024-05-09T05:03:44.990 INFO:teuthology.orchestra.run.smithi196.stdout:(68/156): postgresql-contrib-13.14-1.el9.x86_64 4.1 MB/s | 911 kB 00:00 2024-05-09T05:03:45.074 INFO:teuthology.orchestra.run.smithi196.stdout:(69/156): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-05-09T05:03:45.191 INFO:teuthology.orchestra.run.smithi196.stdout:(70/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 4.4 MB 00:00 2024-05-09T05:03:45.341 INFO:teuthology.orchestra.run.smithi196.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 6.7 MB/s | 1.0 MB 00:00 2024-05-09T05:03:45.433 INFO:teuthology.orchestra.run.smithi196.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 957 kB/s | 87 kB 00:00 2024-05-09T05:03:45.600 INFO:teuthology.orchestra.run.smithi196.stdout:(73/156): postgresql-server-13.14-1.el9.x86_64. 11 MB/s | 5.8 MB 00:00 2024-05-09T05:03:45.642 INFO:teuthology.orchestra.run.smithi196.stdout:(74/156): postgresql-13.14-1.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-05-09T05:03:45.742 INFO:teuthology.orchestra.run.smithi196.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 1.7 MB/s | 244 kB 00:00 2024-05-09T05:03:45.776 INFO:teuthology.orchestra.run.smithi196.stdout:(76/156): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-05-09T05:03:45.788 INFO:teuthology.orchestra.run.smithi001.stdout:(7/156): ceph-osd-19.0.0-3594.gc9da4049.el9.x86 21 MB/s | 18 MB 00:00 2024-05-09T05:03:45.893 INFO:teuthology.orchestra.run.smithi196.stdout:(77/156): python3-babel-2.9.1-2.el9.noarch.rpm 13 MB/s | 6.0 MB 00:00 2024-05-09T05:03:45.910 INFO:teuthology.orchestra.run.smithi196.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 153 kB/s | 26 kB 00:00 2024-05-09T05:03:45.926 INFO:teuthology.orchestra.run.smithi196.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 127 kB/s | 19 kB 00:00 2024-05-09T05:03:45.994 INFO:teuthology.orchestra.run.smithi196.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 172 kB 00:00 2024-05-09T05:03:46.010 INFO:teuthology.orchestra.run.smithi196.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 415 kB/s | 35 kB 00:00 2024-05-09T05:03:46.094 INFO:teuthology.orchestra.run.smithi196.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.4 MB/s | 250 kB 00:00 2024-05-09T05:03:46.145 INFO:teuthology.orchestra.run.smithi196.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.6 MB/s | 222 kB 00:00 2024-05-09T05:03:46.170 INFO:teuthology.orchestra.run.smithi196.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 77 kB 00:00 2024-05-09T05:03:46.245 INFO:teuthology.orchestra.run.smithi196.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 422 kB/s | 42 kB 00:00 2024-05-09T05:03:46.288 INFO:teuthology.orchestra.run.smithi001.stdout:(8/156): ceph-radosgw-19.0.0-3594.gc9da4049.el9 20 MB/s | 10 MB 00:00 2024-05-09T05:03:46.304 INFO:teuthology.orchestra.run.smithi196.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 2.0 MB/s | 269 kB 00:00 2024-05-09T05:03:46.329 INFO:teuthology.orchestra.run.smithi196.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 159 kB 00:00 2024-05-09T05:03:46.363 INFO:teuthology.orchestra.run.smithi001.stdout:(9/156): ceph-base-19.0.0-3594.gc9da4049.el9.x8 1.8 MB/s | 5.4 MB 00:03 2024-05-09T05:03:46.388 INFO:teuthology.orchestra.run.smithi001.stdout:(10/156): ceph-selinux-19.0.0-3594.gc9da4049.el 251 kB/s | 25 kB 00:00 2024-05-09T05:03:46.396 INFO:teuthology.orchestra.run.smithi196.stdout:(88/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 11 MB/s | 5.1 MB 00:00 2024-05-09T05:03:46.439 INFO:teuthology.orchestra.run.smithi001.stdout:(11/156): cephfs-mirror-19.0.0-3594.gc9da4049.e 4.4 MB/s | 224 kB 00:00 2024-05-09T05:03:46.446 INFO:teuthology.orchestra.run.smithi196.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 1.9 MB/s | 279 kB 00:00 2024-05-09T05:03:46.472 INFO:teuthology.orchestra.run.smithi196.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 358 kB/s | 51 kB 00:00 2024-05-09T05:03:46.480 INFO:teuthology.orchestra.run.smithi001.stdout:(12/156): cephfs-shell-19.0.0-3594.gc9da4049.el 651 kB/s | 27 kB 00:00 2024-05-09T05:03:46.497 INFO:teuthology.orchestra.run.smithi196.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 534 kB/s | 54 kB 00:00 2024-05-09T05:03:46.531 INFO:teuthology.orchestra.run.smithi001.stdout:(13/156): libcephfs-devel-19.0.0-3594.gc9da4049 632 kB/s | 32 kB 00:00 2024-05-09T05:03:46.547 INFO:teuthology.orchestra.run.smithi196.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 555 kB/s | 42 kB 00:00 2024-05-09T05:03:46.572 INFO:teuthology.orchestra.run.smithi196.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 587 kB/s | 44 kB 00:00 2024-05-09T05:03:46.606 INFO:teuthology.orchestra.run.smithi001.stdout:(14/156): libcephfs2-19.0.0-3594.gc9da4049.el9. 9.9 MB/s | 756 kB 00:00 2024-05-09T05:03:46.631 INFO:teuthology.orchestra.run.smithi196.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 617 kB/s | 36 kB 00:00 2024-05-09T05:03:46.657 INFO:teuthology.orchestra.run.smithi001.stdout:(15/156): libcephsqlite-19.0.0-3594.gc9da4049.e 3.1 MB/s | 158 kB 00:00 2024-05-09T05:03:46.673 INFO:teuthology.orchestra.run.smithi196.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-05-09T05:03:46.698 INFO:teuthology.orchestra.run.smithi196.stdout:(96/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 711 kB/s | 47 kB 00:00 2024-05-09T05:03:46.707 INFO:teuthology.orchestra.run.smithi001.stdout:(16/156): librados-devel-19.0.0-3594.gc9da4049. 2.5 MB/s | 127 kB 00:00 2024-05-09T05:03:46.749 INFO:teuthology.orchestra.run.smithi001.stdout:(17/156): ceph-mds-19.0.0-3594.gc9da4049.el9.x8 1.0 MB/s | 2.3 MB 00:02 2024-05-09T05:03:46.765 INFO:teuthology.orchestra.run.smithi196.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 868 kB/s | 58 kB 00:00 2024-05-09T05:03:46.782 INFO:teuthology.orchestra.run.smithi001.stdout:(18/156): libradosstriper1-19.0.0-3594.gc9da404 6.5 MB/s | 497 kB 00:00 2024-05-09T05:03:46.807 INFO:teuthology.orchestra.run.smithi196.stdout:(98/156): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-05-09T05:03:46.824 INFO:teuthology.orchestra.run.smithi001.stdout:(19/156): python3-ceph-argparse-19.0.0-3594.gc9 1.1 MB/s | 45 kB 00:00 2024-05-09T05:03:46.849 INFO:teuthology.orchestra.run.smithi196.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.1 MB/s | 94 kB 00:00 2024-05-09T05:03:46.866 INFO:teuthology.orchestra.run.smithi001.stdout:(20/156): python3-ceph-common-19.0.0-3594.gc9da 3.2 MB/s | 135 kB 00:00 2024-05-09T05:03:46.874 INFO:teuthology.orchestra.run.smithi196.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 954 kB/s | 64 kB 00:00 2024-05-09T05:03:46.917 INFO:teuthology.orchestra.run.smithi001.stdout:(21/156): python3-cephfs-19.0.0-3594.gc9da4049. 3.2 MB/s | 162 kB 00:00 2024-05-09T05:03:46.924 INFO:teuthology.orchestra.run.smithi196.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 447 kB/s | 22 kB 00:00 2024-05-09T05:03:46.950 INFO:teuthology.orchestra.run.smithi196.stdout:(102/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 446 kB/s | 45 kB 00:00 2024-05-09T05:03:46.984 INFO:teuthology.orchestra.run.smithi001.stdout:(22/156): python3-rados-19.0.0-3594.gc9da4049.e 4.7 MB/s | 321 kB 00:00 2024-05-09T05:03:47.034 INFO:teuthology.orchestra.run.smithi001.stdout:(23/156): python3-rbd-19.0.0-3594.gc9da4049.el9 5.9 MB/s | 301 kB 00:00 2024-05-09T05:03:47.042 INFO:teuthology.orchestra.run.smithi196.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 7.2 MB/s | 863 kB 00:00 2024-05-09T05:03:47.084 INFO:teuthology.orchestra.run.smithi001.stdout:(24/156): python3-rgw-19.0.0-3594.gc9da4049.el9 1.9 MB/s | 99 kB 00:00 2024-05-09T05:03:47.126 INFO:teuthology.orchestra.run.smithi001.stdout:(25/156): rbd-fuse-19.0.0-3594.gc9da4049.el9.x8 2.0 MB/s | 86 kB 00:00 2024-05-09T05:03:47.292 INFO:teuthology.orchestra.run.smithi196.stdout:(104/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 19 MB/s | 16 MB 00:00 2024-05-09T05:03:47.317 INFO:teuthology.orchestra.run.smithi196.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.5 MB/s | 548 kB 00:00 2024-05-09T05:03:47.318 INFO:teuthology.orchestra.run.smithi001.stdout:(26/156): rbd-mirror-19.0.0-3594.gc9da4049.el9. 16 MB/s | 3.0 MB 00:00 2024-05-09T05:03:47.359 INFO:teuthology.orchestra.run.smithi196.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 975 kB/s | 309 kB 00:00 2024-05-09T05:03:47.368 INFO:teuthology.orchestra.run.smithi001.stdout:(27/156): rbd-nbd-19.0.0-3594.gc9da4049.el9.x86 3.3 MB/s | 169 kB 00:00 2024-05-09T05:03:47.384 INFO:teuthology.orchestra.run.smithi196.stdout:(107/156): grpc-data-1.46.7-10.el9.noarch.rpm 211 kB/s | 19 kB 00:00 2024-05-09T05:03:47.410 INFO:teuthology.orchestra.run.smithi196.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 516 kB/s | 26 kB 00:00 2024-05-09T05:03:47.410 INFO:teuthology.orchestra.run.smithi001.stdout:(28/156): ceph-grafana-dashboards-19.0.0-3594.g 824 kB/s | 34 kB 00:00 2024-05-09T05:03:47.452 INFO:teuthology.orchestra.run.smithi196.stdout:(109/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.6 MB/s | 67 kB 00:00 2024-05-09T05:03:47.485 INFO:teuthology.orchestra.run.smithi196.stdout:(110/156): liboath-2.6.7-2.el9.x86_64.rpm 481 kB/s | 49 kB 00:00 2024-05-09T05:03:47.527 INFO:teuthology.orchestra.run.smithi196.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 2.0 MB/s | 151 kB 00:00 2024-05-09T05:03:47.560 INFO:teuthology.orchestra.run.smithi001.stdout:(29/156): ceph-mgr-cephadm-19.0.0-3594.gc9da404 953 kB/s | 142 kB 00:00 2024-05-09T05:03:47.653 INFO:teuthology.orchestra.run.smithi196.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 4.3 MB/s | 548 kB 00:00 2024-05-09T05:03:47.678 INFO:teuthology.orchestra.run.smithi196.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 4.3 MB/s | 839 kB 00:00 2024-05-09T05:03:47.703 INFO:teuthology.orchestra.run.smithi196.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 863 kB/s | 43 kB 00:00 2024-05-09T05:03:47.729 INFO:teuthology.orchestra.run.smithi196.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-05-09T05:03:47.754 INFO:teuthology.orchestra.run.smithi196.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-05-09T05:03:47.761 INFO:teuthology.orchestra.run.smithi001.stdout:(30/156): ceph-mgr-dashboard-19.0.0-3594.gc9da4 18 MB/s | 3.5 MB 00:00 2024-05-09T05:03:47.788 INFO:teuthology.orchestra.run.smithi196.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.9 MB/s | 172 kB 00:00 2024-05-09T05:03:47.839 INFO:teuthology.orchestra.run.smithi196.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-05-09T05:03:47.871 INFO:teuthology.orchestra.run.smithi196.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.5 MB/s | 303 kB 00:00 2024-05-09T05:03:47.896 INFO:teuthology.orchestra.run.smithi196.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 602 kB/s | 35 kB 00:00 2024-05-09T05:03:47.930 INFO:teuthology.orchestra.run.smithi196.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 3.4 MB/s | 206 kB 00:00 2024-05-09T05:03:47.980 INFO:teuthology.orchestra.run.smithi196.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-05-09T05:03:48.047 INFO:teuthology.orchestra.run.smithi196.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 161 kB/s | 11 kB 00:00 2024-05-09T05:03:48.369 INFO:teuthology.orchestra.run.smithi001.stdout:(31/156): ceph-mgr-diskprediction-local-19.0.0- 19 MB/s | 7.4 MB 00:00 2024-05-09T05:03:48.369 INFO:teuthology.orchestra.run.smithi001.stdout:(32/156): ceph-mgr-modules-core-19.0.0-3594.gc9 4.6 MB/s | 274 kB 00:00 2024-05-09T05:03:48.369 INFO:teuthology.orchestra.run.smithi001.stdout:(33/156): ceph-mgr-rook-19.0.0-3594.gc9da4049.e 988 kB/s | 49 kB 00:00 2024-05-09T05:03:48.369 INFO:teuthology.orchestra.run.smithi001.stdout:(34/156): ceph-prometheus-alerts-19.0.0-3594.gc 395 kB/s | 16 kB 00:00 2024-05-09T05:03:48.369 INFO:teuthology.orchestra.run.smithi001.stdout:(35/156): ceph-volume-19.0.0-3594.gc9da4049.el9 5.4 MB/s | 273 kB 00:00 2024-05-09T05:03:48.369 INFO:teuthology.orchestra.run.smithi196.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 529 kB/s | 18 kB 00:00 2024-05-09T05:03:48.369 INFO:teuthology.orchestra.run.smithi196.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 698 kB/s | 23 kB 00:00 2024-05-09T05:03:48.369 INFO:teuthology.orchestra.run.smithi196.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 582 kB/s | 19 kB 00:00 2024-05-09T05:03:48.370 INFO:teuthology.orchestra.run.smithi196.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 471 kB/s | 20 kB 00:00 2024-05-09T05:03:48.370 INFO:teuthology.orchestra.run.smithi196.stdout:(128/156): python3-grpcio-1.46.7-10.el9.x86_64. 4.4 MB/s | 2.0 MB 00:00 2024-05-09T05:03:48.399 INFO:teuthology.orchestra.run.smithi196.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 1.1 MB/s | 46 kB 00:00 2024-05-09T05:03:48.421 INFO:teuthology.orchestra.run.smithi001.stdout:(36/156): cephadm-19.0.0-3594.gc9da4049.el9.noa 10 MB/s | 763 kB 00:00 2024-05-09T05:03:48.458 INFO:teuthology.orchestra.run.smithi196.stdout:(130/156): libarrow-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 4.4 MB 00:01 2024-05-09T05:03:48.471 INFO:teuthology.orchestra.run.smithi001.stdout:(37/156): cephfs-top-19.0.0-3594.gc9da4049.el9. 365 kB/s | 18 kB 00:00 2024-05-09T05:03:48.483 INFO:teuthology.orchestra.run.smithi196.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 943 kB/s | 79 kB 00:00 2024-05-09T05:03:48.508 INFO:teuthology.orchestra.run.smithi196.stdout:(132/156): python3-kubernetes-26.1.0-2.el9.noar 3.2 MB/s | 1.0 MB 00:00 2024-05-09T05:03:48.533 INFO:teuthology.orchestra.run.smithi196.stdout:(133/156): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-05-09T05:03:48.559 INFO:teuthology.orchestra.run.smithi196.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 327 kB/s | 16 kB 00:00 2024-05-09T05:03:48.584 INFO:teuthology.orchestra.run.smithi196.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.8 MB/s | 90 kB 00:00 2024-05-09T05:03:48.609 INFO:teuthology.orchestra.run.smithi196.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 561 kB/s | 28 kB 00:00 2024-05-09T05:03:48.613 INFO:teuthology.orchestra.run.smithi001.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 568 kB/s | 80 kB 00:00 2024-05-09T05:03:48.634 INFO:teuthology.orchestra.run.smithi196.stdout:(137/156): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-05-09T05:03:48.660 INFO:teuthology.orchestra.run.smithi196.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 407 kB/s | 31 kB 00:00 2024-05-09T05:03:48.694 INFO:teuthology.orchestra.run.smithi196.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 1.0 MB/s | 59 kB 00:00 2024-05-09T05:03:48.719 INFO:teuthology.orchestra.run.smithi196.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 1.7 MB/s | 188 kB 00:00 2024-05-09T05:03:48.744 INFO:teuthology.orchestra.run.smithi196.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 1.0 MB/s | 51 kB 00:00 2024-05-09T05:03:48.770 INFO:teuthology.orchestra.run.smithi196.stdout:(142/156): python3-tempora-5.0.0-2.el9.noarch.r 327 kB/s | 36 kB 00:00 2024-05-09T05:03:48.787 INFO:teuthology.orchestra.run.smithi196.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-05-09T05:03:48.812 INFO:teuthology.orchestra.run.smithi196.stdout:(144/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-05-09T05:03:48.814 INFO:teuthology.orchestra.run.smithi001.stdout:(39/156): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-05-09T05:03:48.838 INFO:teuthology.orchestra.run.smithi196.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 397 kB/s | 20 kB 00:00 2024-05-09T05:03:48.862 INFO:teuthology.orchestra.run.smithi196.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-05-09T05:03:48.896 INFO:teuthology.orchestra.run.smithi196.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-09T05:03:48.897 INFO:teuthology.orchestra.run.smithi001.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-05-09T05:03:48.990 INFO:teuthology.orchestra.run.smithi001.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.7 MB/s | 253 kB 00:00 2024-05-09T05:03:49.105 INFO:teuthology.orchestra.run.smithi196.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 5.9 MB/s | 1.6 MB 00:00 2024-05-09T05:03:49.231 INFO:teuthology.orchestra.run.smithi001.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 5.1 MB/s | 1.2 MB 00:00 2024-05-09T05:03:49.282 INFO:teuthology.orchestra.run.smithi001.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-05-09T05:03:49.305 INFO:teuthology.orchestra.run.smithi196.stdout:(149/156): librados2-19.0.0-3594.gc9da4049.el9. 7.5 MB/s | 3.3 MB 00:00 2024-05-09T05:03:49.340 INFO:teuthology.orchestra.run.smithi001.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.3 MB/s | 135 kB 00:00 2024-05-09T05:03:49.355 INFO:teuthology.orchestra.run.smithi196.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 9.2 MB/s | 2.3 MB 00:00 2024-05-09T05:03:49.391 INFO:teuthology.orchestra.run.smithi001.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-05-09T05:03:49.397 INFO:teuthology.orchestra.run.smithi196.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 1.0 MB/s | 42 kB 00:00 2024-05-09T05:03:49.474 INFO:teuthology.orchestra.run.smithi001.stdout:(46/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.5 MB/s | 215 kB 00:00 2024-05-09T05:03:49.522 INFO:teuthology.orchestra.run.smithi196.stdout:(152/156): NetworkManager-libnm-1.47.5-1.el9.x8 8.4 MB/s | 1.8 MB 00:00 2024-05-09T05:03:49.541 INFO:teuthology.orchestra.run.smithi001.stdout:(47/156): librgw2-19.0.0-3594.gc9da4049.el9.x86 1.8 MB/s | 5.2 MB 00:02 2024-05-09T05:03:49.589 INFO:teuthology.orchestra.run.smithi196.stdout:(153/156): openssl-3.2.1-1.el9.x86_64.rpm 20 MB/s | 1.3 MB 00:00 2024-05-09T05:03:49.640 INFO:teuthology.orchestra.run.smithi196.stdout:(154/156): librbd1-19.0.0-3594.gc9da4049.el9.x8 4.2 MB/s | 3.1 MB 00:00 2024-05-09T05:03:49.665 INFO:teuthology.orchestra.run.smithi196.stdout:(155/156): NetworkManager-tui-1.47.5-1.el9.x86_ 929 kB/s | 248 kB 00:00 2024-05-09T05:03:49.706 INFO:teuthology.orchestra.run.smithi196.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 20 MB/s | 2.4 MB 00:00 2024-05-09T05:03:49.709 INFO:teuthology.orchestra.run.smithi196.stdout:-------------------------------------------------------------------------------- 2024-05-09T05:03:49.709 INFO:teuthology.orchestra.run.smithi196.stdout:Total 17 MB/s | 219 MB 00:12 2024-05-09T05:03:49.717 INFO:teuthology.orchestra.run.smithi001.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 2.8 MB/s | 689 kB 00:00 2024-05-09T05:03:49.750 INFO:teuthology.orchestra.run.smithi001.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 4.4 MB/s | 943 kB 00:00 2024-05-09T05:03:49.775 INFO:teuthology.orchestra.run.smithi001.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 107 kB 00:00 2024-05-09T05:03:49.801 INFO:teuthology.orchestra.run.smithi001.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 427 kB/s | 21 kB 00:00 2024-05-09T05:03:49.826 INFO:teuthology.orchestra.run.smithi001.stdout:(52/156): flex-2.6.4-9.el9.x86_64.rpm 6.1 MB/s | 313 kB 00:00 2024-05-09T05:03:49.851 INFO:teuthology.orchestra.run.smithi001.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 652 kB/s | 33 kB 00:00 2024-05-09T05:03:49.943 INFO:teuthology.orchestra.run.smithi001.stdout:(54/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 26 MB/s | 3.0 MB 00:00 2024-05-09T05:03:49.985 INFO:teuthology.orchestra.run.smithi001.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 267 kB/s | 11 kB 00:00 2024-05-09T05:03:50.036 INFO:teuthology.orchestra.run.smithi001.stdout:(56/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 160 kB 00:00 2024-05-09T05:03:50.077 INFO:teuthology.orchestra.run.smithi001.stdout:(57/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 1.1 MB/s | 45 kB 00:00 2024-05-09T05:03:50.094 INFO:teuthology.orchestra.run.smithi001.stdout:(58/156): flexiblas-openblas-openmp-3.0.4-8.el9 75 kB/s | 18 kB 00:00 2024-05-09T05:03:50.136 INFO:teuthology.orchestra.run.smithi001.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-05-09T05:03:50.187 INFO:teuthology.orchestra.run.smithi001.stdout:(60/156): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 582 kB 00:00 2024-05-09T05:03:50.204 INFO:teuthology.orchestra.run.smithi001.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 381 kB/s | 25 kB 00:00 2024-05-09T05:03:50.229 INFO:teuthology.orchestra.run.smithi001.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-05-09T05:03:50.254 INFO:teuthology.orchestra.run.smithi001.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-05-09T05:03:50.280 INFO:teuthology.orchestra.run.smithi001.stdout:(64/156): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-05-09T05:03:50.297 INFO:teuthology.orchestra.run.smithi001.stdout:(65/156): openblas-0.3.21-2.el9.x86_64.rpm 841 kB/s | 35 kB 00:00 2024-05-09T05:03:50.413 INFO:teuthology.orchestra.run.smithi001.stdout:(66/156): openblas-openmp-0.3.21-2.el9.x86_64.r 36 MB/s | 4.7 MB 00:00 2024-05-09T05:03:50.481 INFO:teuthology.orchestra.run.smithi001.stdout:(67/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 24 MB/s | 4.4 MB 00:00 2024-05-09T05:03:50.531 INFO:teuthology.orchestra.run.smithi001.stdout:(68/156): postgresql-13.14-1.el9.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-05-09T05:03:50.565 INFO:teuthology.orchestra.run.smithi001.stdout:(69/156): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-05-09T05:03:50.590 INFO:teuthology.orchestra.run.smithi001.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-05-09T05:03:50.648 INFO:teuthology.orchestra.run.smithi001.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 17 MB/s | 1.0 MB 00:00 2024-05-09T05:03:50.699 INFO:teuthology.orchestra.run.smithi001.stdout:(72/156): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-05-09T05:03:50.782 INFO:teuthology.orchestra.run.smithi001.stdout:(73/156): postgresql-server-13.14-1.el9.x86_64. 27 MB/s | 5.8 MB 00:00 2024-05-09T05:03:50.832 INFO:teuthology.orchestra.run.smithi001.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 4.8 MB/s | 244 kB 00:00 2024-05-09T05:03:50.883 INFO:teuthology.orchestra.run.smithi001.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-05-09T05:03:50.939 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction check 2024-05-09T05:03:50.958 INFO:teuthology.orchestra.run.smithi001.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-05-09T05:03:50.983 INFO:teuthology.orchestra.run.smithi001.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 256 kB/s | 26 kB 00:00 2024-05-09T05:03:51.009 INFO:teuthology.orchestra.run.smithi001.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 382 kB/s | 19 kB 00:00 2024-05-09T05:03:51.089 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction check succeeded. 2024-05-09T05:03:51.089 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction test 2024-05-09T05:03:51.094 INFO:teuthology.orchestra.run.smithi001.stdout:(79/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-05-09T05:03:51.118 INFO:teuthology.orchestra.run.smithi001.stdout:(80/156): python3-markupsafe-1.1.1-12.el9.x86_6 319 kB/s | 35 kB 00:00 2024-05-09T05:03:51.168 INFO:teuthology.orchestra.run.smithi001.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.9 MB/s | 250 kB 00:00 2024-05-09T05:03:51.302 INFO:teuthology.orchestra.run.smithi001.stdout:(82/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 24 MB/s | 5.1 MB 00:00 2024-05-09T05:03:51.327 INFO:teuthology.orchestra.run.smithi001.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 1.4 MB/s | 222 kB 00:00 2024-05-09T05:03:51.352 INFO:teuthology.orchestra.run.smithi001.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 1.5 MB/s | 77 kB 00:00 2024-05-09T05:03:51.378 INFO:teuthology.orchestra.run.smithi001.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 839 kB/s | 42 kB 00:00 2024-05-09T05:03:51.411 INFO:teuthology.orchestra.run.smithi001.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 4.5 MB/s | 269 kB 00:00 2024-05-09T05:03:51.436 INFO:teuthology.orchestra.run.smithi001.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.7 MB/s | 159 kB 00:00 2024-05-09T05:03:51.490 INFO:teuthology.orchestra.run.smithi001.stdout:(88/156): python3-pyasn1-modules-0.4.8-6.el9.no 3.5 MB/s | 279 kB 00:00 2024-05-09T05:03:51.512 INFO:teuthology.orchestra.run.smithi001.stdout:(89/156): python3-pytz-2021.1-5.el9.noarch.rpm 678 kB/s | 51 kB 00:00 2024-05-09T05:03:51.629 INFO:teuthology.orchestra.run.smithi001.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 386 kB/s | 54 kB 00:00 2024-05-09T05:03:51.671 INFO:teuthology.orchestra.run.smithi001.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-05-09T05:03:51.730 INFO:teuthology.orchestra.run.smithi001.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 758 kB/s | 44 kB 00:00 2024-05-09T05:03:51.780 INFO:teuthology.orchestra.run.smithi001.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 4.3 MB/s | 221 kB 00:00 2024-05-09T05:03:51.813 INFO:teuthology.orchestra.run.smithi001.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 1.1 MB/s | 36 kB 00:00 2024-05-09T05:03:51.847 INFO:teuthology.orchestra.run.smithi001.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.4 MB/s | 47 kB 00:00 2024-05-09T05:03:51.898 INFO:teuthology.orchestra.run.smithi001.stdout:(96/156): socat-1.7.4.1-5.el9.x86_64.rpm 6.0 MB/s | 305 kB 00:00 2024-05-09T05:03:52.106 INFO:teuthology.orchestra.run.smithi001.stdout:(97/156): uuid-1.6.2-55.el9.x86_64.rpm 277 kB/s | 58 kB 00:00 2024-05-09T05:03:52.156 INFO:teuthology.orchestra.run.smithi001.stdout:(98/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-05-09T05:03:52.190 INFO:teuthology.orchestra.run.smithi001.stdout:(99/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.9 MB/s | 64 kB 00:00 2024-05-09T05:03:52.224 INFO:teuthology.orchestra.run.smithi001.stdout:(100/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 1.3 MB/s | 45 kB 00:00 2024-05-09T05:03:52.807 INFO:teuthology.orchestra.run.smithi001.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 38 kB/s | 22 kB 00:00 2024-05-09T05:03:53.182 INFO:teuthology.orchestra.run.smithi196.stdout:Transaction test succeeded. 2024-05-09T05:03:53.182 INFO:teuthology.orchestra.run.smithi196.stdout:Running transaction 2024-05-09T05:03:53.441 INFO:teuthology.orchestra.run.smithi001.stdout:(102/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 8.4 MB/s | 16 MB 00:01 2024-05-09T05:03:53.891 INFO:teuthology.orchestra.run.smithi001.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 797 kB/s | 863 kB 00:01 2024-05-09T05:03:54.041 INFO:teuthology.orchestra.run.smithi001.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 914 kB/s | 548 kB 00:00 2024-05-09T05:03:54.142 INFO:teuthology.orchestra.run.smithi001.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-05-09T05:03:54.417 INFO:teuthology.orchestra.run.smithi001.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 588 kB/s | 309 kB 00:00 2024-05-09T05:03:54.509 INFO:teuthology.orchestra.run.smithi001.stdout:(107/156): libarrow-9.0.0-11.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-05-09T05:03:54.534 INFO:teuthology.orchestra.run.smithi001.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-05-09T05:03:54.601 INFO:teuthology.orchestra.run.smithi001.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-05-09T05:03:54.635 INFO:teuthology.orchestra.run.smithi001.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-05-09T05:03:54.693 INFO:teuthology.orchestra.run.smithi001.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 1.6 MB/s | 151 kB 00:00 2024-05-09T05:03:54.794 INFO:teuthology.orchestra.run.smithi001.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 5.4 MB/s | 548 kB 00:00 2024-05-09T05:03:54.869 INFO:teuthology.orchestra.run.smithi001.stdout:(113/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.5 MB/s | 839 kB 00:00 2024-05-09T05:03:54.894 INFO:teuthology.orchestra.run.smithi001.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 433 kB/s | 43 kB 00:00 2024-05-09T05:03:54.961 INFO:teuthology.orchestra.run.smithi001.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 350 kB/s | 32 kB 00:00 2024-05-09T05:03:54.986 INFO:teuthology.orchestra.run.smithi001.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 163 kB/s | 15 kB 00:00 2024-05-09T05:03:55.070 INFO:teuthology.orchestra.run.smithi001.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-05-09T05:03:55.095 INFO:teuthology.orchestra.run.smithi001.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 3.2 MB/s | 358 kB 00:00 2024-05-09T05:03:55.179 INFO:teuthology.orchestra.run.smithi001.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-05-09T05:03:55.204 INFO:teuthology.orchestra.run.smithi001.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 325 kB/s | 35 kB 00:00 2024-05-09T05:03:55.288 INFO:teuthology.orchestra.run.smithi001.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 1.8 MB/s | 206 kB 00:00 2024-05-09T05:03:55.347 INFO:teuthology.orchestra.run.smithi001.stdout:(122/156): python3-grpcio-1.46.7-10.el9.x86_64. 14 MB/s | 2.0 MB 00:00 2024-05-09T05:03:55.405 INFO:teuthology.orchestra.run.smithi001.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 1.2 MB/s | 144 kB 00:00 2024-05-09T05:03:55.439 INFO:teuthology.orchestra.run.smithi001.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-05-09T05:03:55.498 INFO:teuthology.orchestra.run.smithi001.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-05-09T05:03:55.531 INFO:teuthology.orchestra.run.smithi001.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-05-09T05:03:55.590 INFO:teuthology.orchestra.run.smithi001.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-05-09T05:03:55.624 INFO:teuthology.orchestra.run.smithi001.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-05-09T05:03:55.716 INFO:teuthology.orchestra.run.smithi001.stdout:(129/156): python3-logutils-0.3.5-21.el9.noarch 505 kB/s | 46 kB 00:00 2024-05-09T05:03:55.749 INFO:teuthology.orchestra.run.smithi001.stdout:(130/156): python3-kubernetes-26.1.0-2.el9.noar 6.4 MB/s | 1.0 MB 00:00 2024-05-09T05:03:55.808 INFO:teuthology.orchestra.run.smithi001.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 859 kB/s | 79 kB 00:00 2024-05-09T05:03:55.833 INFO:teuthology.orchestra.run.smithi001.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 691 kB/s | 58 kB 00:00 2024-05-09T05:03:56.225 INFO:teuthology.orchestra.run.smithi001.stdout:(133/156): ceph-test-19.0.0-3594.gc9da4049.el9. 4.4 MB/s | 43 MB 00:09 2024-05-09T05:03:56.242 INFO:teuthology.orchestra.run.smithi001.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 40 kB/s | 16 kB 00:00 2024-05-09T05:03:56.267 INFO:teuthology.orchestra.run.smithi001.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 593 kB/s | 272 kB 00:00 2024-05-09T05:03:56.342 INFO:teuthology.orchestra.run.smithi001.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 282 kB/s | 28 kB 00:00 2024-05-09T05:03:56.359 INFO:teuthology.orchestra.run.smithi001.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 335 kB/s | 31 kB 00:00 2024-05-09T05:03:56.443 INFO:teuthology.orchestra.run.smithi001.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 1.8 MB/s | 188 kB 00:00 2024-05-09T05:03:56.460 INFO:teuthology.orchestra.run.smithi001.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 588 kB/s | 59 kB 00:00 2024-05-09T05:03:56.527 INFO:teuthology.orchestra.run.smithi001.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 428 kB/s | 36 kB 00:00 2024-05-09T05:03:56.544 INFO:teuthology.orchestra.run.smithi001.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 610 kB/s | 51 kB 00:00 2024-05-09T05:03:56.602 INFO:teuthology.orchestra.run.smithi001.stdout:(142/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 239 kB/s | 90 kB 00:00 2024-05-09T05:03:56.628 INFO:teuthology.orchestra.run.smithi001.stdout:(143/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-05-09T05:03:56.653 INFO:teuthology.orchestra.run.smithi001.stdout:(144/156): python3-websocket-client-1.2.3-2.el9 825 kB/s | 90 kB 00:00 2024-05-09T05:03:56.720 INFO:teuthology.orchestra.run.smithi001.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-05-09T05:03:56.745 INFO:teuthology.orchestra.run.smithi001.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 2.0 MB/s | 192 kB 00:00 2024-05-09T05:03:56.879 INFO:teuthology.orchestra.run.smithi001.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 1.5 MB/s | 427 kB 00:00 2024-05-09T05:03:56.912 INFO:teuthology.orchestra.run.smithi001.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 8.2 MB/s | 1.6 MB 00:00 2024-05-09T05:03:57.304 INFO:teuthology.orchestra.run.smithi001.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 5.9 MB/s | 2.3 MB 00:00 2024-05-09T05:03:57.496 INFO:teuthology.orchestra.run.smithi001.stdout:(150/156): librbd1-19.0.0-3594.gc9da4049.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-05-09T05:03:57.538 INFO:teuthology.orchestra.run.smithi001.stdout:(151/156): NetworkManager-libnm-1.47.5-1.el9.x8 7.8 MB/s | 1.8 MB 00:00 2024-05-09T05:03:57.597 INFO:teuthology.orchestra.run.smithi001.stdout:(152/156): NetworkManager-tui-1.47.5-1.el9.x86_ 4.2 MB/s | 248 kB 00:00 2024-05-09T05:03:57.622 INFO:teuthology.orchestra.run.smithi001.stdout:(153/156): NetworkManager-team-1.47.5-1.el9.x86 333 kB/s | 42 kB 00:00 2024-05-09T05:03:57.756 INFO:teuthology.orchestra.run.smithi001.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 8.5 MB/s | 1.3 MB 00:00 2024-05-09T05:03:57.981 INFO:teuthology.orchestra.run.smithi001.stdout:(155/156): librados2-19.0.0-3594.gc9da4049.el9. 2.7 MB/s | 3.3 MB 00:01 2024-05-09T05:03:58.131 INFO:teuthology.orchestra.run.smithi001.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 4.7 MB/s | 2.4 MB 00:00 2024-05-09T05:03:58.134 INFO:teuthology.orchestra.run.smithi001.stdout:-------------------------------------------------------------------------------- 2024-05-09T05:03:58.135 INFO:teuthology.orchestra.run.smithi001.stdout:Total 14 MB/s | 219 MB 00:15 2024-05-09T05:03:59.376 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction check 2024-05-09T05:03:59.425 INFO:teuthology.orchestra.run.smithi196.stdout: Preparing : 1/1 2024-05-09T05:03:59.503 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T05:03:59.541 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T05:03:59.543 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction check succeeded. 2024-05-09T05:03:59.543 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction test 2024-05-09T05:03:59.610 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T05:03:59.653 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T05:03:59.678 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T05:03:59.733 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T05:04:00.064 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T05:04:00.171 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T05:04:00.296 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T05:04:00.322 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T05:04:00.413 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T05:04:00.443 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 10/164 2024-05-09T05:04:00.615 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T05:04:00.665 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T05:04:00.693 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T05:04:00.798 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T05:04:00.827 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 14/164 2024-05-09T05:04:00.882 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 14/164 2024-05-09T05:04:00.912 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T05:04:00.940 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T05:04:01.131 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T05:04:01.317 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T05:04:01.483 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T05:04:01.546 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T05:04:01.672 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T05:04:01.742 INFO:teuthology.orchestra.run.smithi001.stdout:Transaction test succeeded. 2024-05-09T05:04:01.742 INFO:teuthology.orchestra.run.smithi001.stdout:Running transaction 2024-05-09T05:04:01.745 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T05:04:01.961 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T05:04:02.077 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T05:04:02.108 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T05:04:02.142 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/164 2024-05-09T05:04:02.217 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 22/164 2024-05-09T05:04:02.259 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 23/164 2024-05-09T05:04:02.303 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 24/164 2024-05-09T05:04:02.335 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 25/164 2024-05-09T05:04:02.363 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-5.4.4-4.el9.x86_64 26/164 2024-05-09T05:04:02.431 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 27/164 2024-05-09T05:04:02.484 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 28/164 2024-05-09T05:04:02.516 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 29/164 2024-05-09T05:04:02.559 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/164 2024-05-09T05:04:02.634 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/164 2024-05-09T05:04:02.670 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-common-2:19.0.0-3594.gc9da4049.el9. 32/164 2024-05-09T05:04:02.713 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el 33/164 2024-05-09T05:04:02.742 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_6 34/164 2024-05-09T05:04:02.811 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 35/164 2024-05-09T05:04:02.856 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : luarocks-3.9.2-1.el9.noarch 36/164 2024-05-09T05:04:02.930 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 37/164 2024-05-09T05:04:03.062 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 38/164 2024-05-09T05:04:03.117 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 39/164 2024-05-09T05:04:03.653 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rsa-4.9-2.el9.noarch 40/164 2024-05-09T05:04:03.729 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 41/164 2024-05-09T05:04:03.759 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 42/164 2024-05-09T05:04:03.802 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 43/164 2024-05-09T05:04:03.831 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_6 44/164 2024-05-09T05:04:03.873 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 45/164 2024-05-09T05:04:04.017 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 46/164 2024-05-09T05:04:04.075 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 47/164 2024-05-09T05:04:04.166 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/164 2024-05-09T05:04:04.231 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 49/164 2024-05-09T05:04:04.304 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T05:04:04.335 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T05:04:04.371 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T05:04:04.418 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T05:04:04.462 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T05:04:04.505 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T05:04:04.547 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T05:04:04.683 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T05:04:04.728 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T05:04:04.758 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T05:04:04.800 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T05:04:04.894 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T05:04:05.219 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T05:04:05.277 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T05:04:05.317 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T05:04:05.341 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T05:04:05.445 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T05:04:05.578 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T05:04:05.631 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T05:04:05.659 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T05:04:05.694 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T05:04:05.732 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T05:04:05.764 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T05:04:05.794 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T05:04:05.828 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T05:04:05.861 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T05:04:05.903 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T05:04:05.929 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T05:04:06.033 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T05:04:06.076 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T05:04:06.108 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T05:04:06.154 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T05:04:06.253 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T05:04:06.300 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T05:04:07.023 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T05:04:07.064 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T05:04:07.097 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T05:04:07.196 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T05:04:07.248 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T05:04:07.335 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T05:04:07.383 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T05:04:07.561 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T05:04:07.589 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T05:04:07.674 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T05:04:08.054 INFO:teuthology.orchestra.run.smithi001.stdout: Preparing : 1/1 2024-05-09T05:04:08.153 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T05:04:08.196 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T05:04:08.279 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T05:04:08.320 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T05:04:08.332 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T05:04:08.346 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T05:04:08.405 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T05:04:08.445 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T05:04:08.759 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T05:04:08.880 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T05:04:08.996 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T05:04:09.023 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T05:04:09.116 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T05:04:09.144 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 10/164 2024-05-09T05:04:09.327 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T05:04:09.375 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T05:04:09.393 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T05:04:09.505 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T05:04:09.528 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 14/164 2024-05-09T05:04:09.591 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 14/164 2024-05-09T05:04:09.620 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T05:04:09.648 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T05:04:09.687 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T05:04:09.776 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T05:04:09.803 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T05:04:09.828 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T05:04:09.849 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T05:04:10.019 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T05:04:10.146 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T05:04:10.150 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T05:04:10.191 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 101/164 2024-05-09T05:04:10.211 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T05:04:10.282 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 101/164 2024-05-09T05:04:10.312 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 102/164 2024-05-09T05:04:10.326 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T05:04:10.355 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T05:04:10.421 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T05:04:10.641 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T05:04:10.757 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T05:04:10.795 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T05:04:10.837 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : liboath-2.6.7-2.el9.x86_64 21/164 2024-05-09T05:04:10.853 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T05:04:10.886 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 105/164 2024-05-09T05:04:10.912 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 22/164 2024-05-09T05:04:10.957 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 105/164 2024-05-09T05:04:10.960 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 23/164 2024-05-09T05:04:10.987 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 106/164 2024-05-09T05:04:11.004 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 24/164 2024-05-09T05:04:11.046 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 25/164 2024-05-09T05:04:11.074 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-5.4.4-4.el9.x86_64 26/164 2024-05-09T05:04:11.143 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 27/164 2024-05-09T05:04:11.196 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 28/164 2024-05-09T05:04:11.225 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 29/164 2024-05-09T05:04:11.269 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 30/164 2024-05-09T05:04:11.347 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 31/164 2024-05-09T05:04:11.380 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-common-2:19.0.0-3594.gc9da4049.el9. 32/164 2024-05-09T05:04:11.424 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el 33/164 2024-05-09T05:04:11.452 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_6 34/164 2024-05-09T05:04:11.526 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 35/164 2024-05-09T05:04:11.575 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : luarocks-3.9.2-1.el9.noarch 36/164 2024-05-09T05:04:11.654 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 37/164 2024-05-09T05:04:11.815 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 38/164 2024-05-09T05:04:11.871 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 39/164 2024-05-09T05:04:12.424 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rsa-4.9-2.el9.noarch 40/164 2024-05-09T05:04:12.517 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 41/164 2024-05-09T05:04:12.552 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 42/164 2024-05-09T05:04:12.602 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 43/164 2024-05-09T05:04:12.641 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_6 44/164 2024-05-09T05:04:12.690 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 45/164 2024-05-09T05:04:12.853 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 46/164 2024-05-09T05:04:12.911 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 47/164 2024-05-09T05:04:13.006 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 48/164 2024-05-09T05:04:13.068 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 49/164 2024-05-09T05:04:13.142 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T05:04:13.178 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T05:04:13.215 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T05:04:13.256 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T05:04:13.271 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T05:04:13.291 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T05:04:13.306 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T05:04:13.350 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T05:04:13.392 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T05:04:13.536 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T05:04:13.833 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T05:04:13.879 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T05:04:13.887 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T05:04:13.912 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 108/164 2024-05-09T05:04:13.929 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T05:04:14.005 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 108/164 2024-05-09T05:04:14.005 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T05:04:14.005 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T05:04:14.005 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:14.030 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T05:04:14.046 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 109/164 2024-05-09T05:04:14.061 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T05:04:14.103 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T05:04:14.134 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T05:04:14.159 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T05:04:14.247 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T05:04:14.382 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T05:04:14.425 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T05:04:14.456 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T05:04:14.487 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T05:04:14.527 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T05:04:14.567 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T05:04:14.596 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T05:04:14.630 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T05:04:14.664 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T05:04:14.706 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T05:04:14.740 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T05:04:14.830 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T05:04:14.885 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T05:04:14.935 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T05:04:14.983 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T05:04:15.094 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T05:04:15.146 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T05:04:15.911 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T05:04:15.967 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T05:04:16.001 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T05:04:16.106 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T05:04:16.160 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T05:04:16.251 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T05:04:16.286 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T05:04:16.470 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T05:04:16.500 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T05:04:16.589 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T05:04:17.291 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T05:04:17.411 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T05:04:18.739 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T05:04:18.824 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T05:04:18.857 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T05:04:18.890 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T05:04:19.224 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T05:04:19.269 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 101/164 2024-05-09T05:04:19.361 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 101/164 2024-05-09T05:04:19.390 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 102/164 2024-05-09T05:04:19.438 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T05:04:19.952 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T05:04:19.988 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 105/164 2024-05-09T05:04:20.060 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 105/164 2024-05-09T05:04:20.090 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 106/164 2024-05-09T05:04:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T05:04:22.495 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T05:04:23.098 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 107/164 2024-05-09T05:04:23.140 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 108/164 2024-05-09T05:04:23.234 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 108/164 2024-05-09T05:04:23.234 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T05:04:23.234 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T05:04:23.234 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:23.274 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 109/164 2024-05-09T05:04:25.471 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 109/164 2024-05-09T05:04:25.472 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /sys 2024-05-09T05:04:25.472 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /proc 2024-05-09T05:04:25.472 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /mnt 2024-05-09T05:04:25.472 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /var/tmp 2024-05-09T05:04:25.472 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /home 2024-05-09T05:04:25.472 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /root 2024-05-09T05:04:25.472 INFO:teuthology.orchestra.run.smithi196.stdout:skipping the directory /tmp 2024-05-09T05:04:25.472 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:25.644 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 110/164 2024-05-09T05:04:25.660 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 110/164 2024-05-09T05:04:25.660 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:25.660 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T05:04:25.660 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:27.272 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 111/164 2024-05-09T05:04:27.302 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 111/164 2024-05-09T05:04:27.302 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:27.302 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T05:04:27.303 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T05:04:27.303 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T05:04:27.303 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:27.532 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 112/164 2024-05-09T05:04:27.557 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 112/164 2024-05-09T05:04:27.557 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:27.557 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T05:04:27.558 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T05:04:27.558 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T05:04:27.558 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:28.024 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 113/164 2024-05-09T05:04:28.048 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 113/164 2024-05-09T05:04:28.048 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:28.048 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T05:04:28.048 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T05:04:28.049 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T05:04:28.049 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:28.154 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T05:04:28.210 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T05:04:28.265 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T05:04:28.460 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T05:04:28.508 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T05:04:28.585 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T05:04:28.674 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T05:04:28.821 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T05:04:28.871 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T05:04:28.907 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T05:04:28.934 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T05:04:29.415 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 125/164 2024-05-09T05:04:29.433 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 125/164 2024-05-09T05:04:29.465 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.e 126/164 2024-05-09T05:04:29.535 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049. 127/164 2024-05-09T05:04:29.564 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 128/164 2024-05-09T05:04:30.603 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 128/164 2024-05-09T05:04:30.641 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 129/164 2024-05-09T05:04:31.604 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 129/164 2024-05-09T05:04:31.633 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 130/164 2024-05-09T05:04:31.744 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 130/164 2024-05-09T05:04:31.908 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el 131/164 2024-05-09T05:04:31.943 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 132/164 2024-05-09T05:04:31.968 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 132/164 2024-05-09T05:04:31.968 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:31.968 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T05:04:31.968 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T05:04:31.968 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T05:04:31.968 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:32.019 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 133/164 2024-05-09T05:04:32.032 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 133/164 2024-05-09T05:04:32.130 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 134/164 2024-05-09T05:04:32.182 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 135/164 2024-05-09T05:04:33.121 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T05:04:33.155 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 137/164 2024-05-09T05:04:33.179 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 137/164 2024-05-09T05:04:33.179 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:33.179 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T05:04:33.179 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T05:04:33.180 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T05:04:33.180 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:33.222 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 138/164 2024-05-09T05:04:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 138/164 2024-05-09T05:04:33.245 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:33.245 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T05:04:33.245 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:33.298 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 139/164 2024-05-09T05:04:33.321 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 139/164 2024-05-09T05:04:33.322 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:33.322 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T05:04:33.322 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T05:04:33.322 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T05:04:33.322 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:33.631 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 140/164 2024-05-09T05:04:33.655 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 140/164 2024-05-09T05:04:33.655 INFO:teuthology.orchestra.run.smithi196.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:33.655 INFO:teuthology.orchestra.run.smithi196.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T05:04:33.655 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T05:04:33.656 INFO:teuthology.orchestra.run.smithi196.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T05:04:33.656 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 109/164 2024-05-09T05:04:35.204 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /sys 2024-05-09T05:04:35.204 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /proc 2024-05-09T05:04:35.204 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /mnt 2024-05-09T05:04:35.204 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /var/tmp 2024-05-09T05:04:35.204 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /home 2024-05-09T05:04:35.205 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /root 2024-05-09T05:04:35.205 INFO:teuthology.orchestra.run.smithi001.stdout:skipping the directory /tmp 2024-05-09T05:04:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:35.382 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 110/164 2024-05-09T05:04:35.399 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 110/164 2024-05-09T05:04:35.399 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:35.399 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T05:04:35.399 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:37.052 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 111/164 2024-05-09T05:04:37.081 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 111/164 2024-05-09T05:04:37.081 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:37.081 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T05:04:37.081 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T05:04:37.082 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T05:04:37.082 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:37.319 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 112/164 2024-05-09T05:04:37.344 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 112/164 2024-05-09T05:04:37.344 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:37.344 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T05:04:37.344 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T05:04:37.344 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T05:04:37.344 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:37.828 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 113/164 2024-05-09T05:04:37.853 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 113/164 2024-05-09T05:04:37.853 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:37.854 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T05:04:37.854 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T05:04:37.854 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T05:04:37.854 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:37.918 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 141/164 2024-05-09T05:04:37.960 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 142/164 2024-05-09T05:04:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T05:04:38.009 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 143/164 2024-05-09T05:04:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T05:04:38.041 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T05:04:38.068 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T05:04:38.089 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T05:04:38.260 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 146/164 2024-05-09T05:04:38.282 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T05:04:38.306 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T05:04:38.345 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_ 148/164 2024-05-09T05:04:38.354 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T05:04:38.374 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T05:04:38.420 INFO:teuthology.orchestra.run.smithi196.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T05:04:38.435 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T05:04:38.533 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T05:04:38.673 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T05:04:38.730 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T05:04:38.770 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T05:04:38.796 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T05:04:38.919 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T05:04:38.960 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T05:04:38.980 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T05:04:39.261 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 125/164 2024-05-09T05:04:39.279 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 125/164 2024-05-09T05:04:39.303 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.e 126/164 2024-05-09T05:04:39.366 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049. 127/164 2024-05-09T05:04:39.393 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 128/164 2024-05-09T05:04:39.822 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 152/164 2024-05-09T05:04:39.922 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T05:04:39.973 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T05:04:40.066 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T05:04:40.100 INFO:teuthology.orchestra.run.smithi196.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T05:04:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T05:04:40.126 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T05:04:40.211 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T05:04:40.252 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T05:04:40.252 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T05:04:40.345 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T05:04:40.370 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T05:04:40.380 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T05:04:40.380 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T05:04:40.491 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 128/164 2024-05-09T05:04:40.528 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 129/164 2024-05-09T05:04:40.563 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T05:04:40.564 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T05:04:40.648 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T05:04:40.713 INFO:teuthology.orchestra.run.smithi196.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T05:04:41.546 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 129/164 2024-05-09T05:04:41.578 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 130/164 2024-05-09T05:04:41.697 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 130/164 2024-05-09T05:04:41.869 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el 131/164 2024-05-09T05:04:41.897 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 132/164 2024-05-09T05:04:41.923 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 132/164 2024-05-09T05:04:41.923 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:41.923 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T05:04:41.923 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T05:04:41.923 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T05:04:41.923 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:41.973 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 133/164 2024-05-09T05:04:41.985 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 133/164 2024-05-09T05:04:42.078 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 134/164 2024-05-09T05:04:42.148 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 135/164 2024-05-09T05:04:43.135 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T05:04:43.175 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 137/164 2024-05-09T05:04:43.201 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 137/164 2024-05-09T05:04:43.201 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:43.202 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T05:04:43.202 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T05:04:43.202 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T05:04:43.202 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:43.242 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 138/164 2024-05-09T05:04:43.266 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 138/164 2024-05-09T05:04:43.267 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:43.267 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T05:04:43.267 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:43.318 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 139/164 2024-05-09T05:04:43.342 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 139/164 2024-05-09T05:04:43.343 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:43.343 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T05:04:43.343 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T05:04:43.343 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T05:04:43.343 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:43.669 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 140/164 2024-05-09T05:04:43.694 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 140/164 2024-05-09T05:04:43.694 INFO:teuthology.orchestra.run.smithi001.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T05:04:43.694 INFO:teuthology.orchestra.run.smithi001.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T05:04:43.694 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T05:04:43.695 INFO:teuthology.orchestra.run.smithi001.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T05:04:43.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:46.437 INFO:teuthology.orchestra.run.smithi196.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 1/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 2/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 3/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 4/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 5/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 6/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 7/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T05:04:46.438 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 10/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 11/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 12/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 14/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_ 15/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 16/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 17/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_6 18/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 19/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 20/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el 21/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ceph-common-2:19.0.0-3594.gc9da4049.el9. 22/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_6 23/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 24/164 2024-05-09T05:04:46.439 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 25/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 26/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 27/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 28/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 29/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049. 30/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 31/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 32/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 33/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el 34/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 35/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.e 36/164 2024-05-09T05:04:46.453 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 37/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 38/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 39/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T05:04:46.454 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T05:04:46.455 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T05:04:46.456 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T05:04:46.457 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T05:04:46.458 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T05:04:46.459 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T05:04:46.460 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T05:04:46.461 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 149/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 151/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T05:04:46.462 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T05:04:46.463 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T05:04:46.463 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T05:04:46.463 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T05:04:48.206 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 141/164 2024-05-09T05:04:48.256 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 142/164 2024-05-09T05:04:48.297 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 143/164 2024-05-09T05:04:48.328 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T05:04:48.356 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T05:04:48.566 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 146/164 2024-05-09T05:04:48.618 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T05:04:48.648 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_ 148/164 2024-05-09T05:04:48.678 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T05:04:48.733 INFO:teuthology.orchestra.run.smithi001.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout:Upgraded: 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout: librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.883 INFO:teuthology.orchestra.run.smithi196.stdout: librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout:Installed: 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.884 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-immutable-object-cache-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.885 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T05:04:48.886 INFO:teuthology.orchestra.run.smithi196.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.887 INFO:teuthology.orchestra.run.smithi196.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T05:04:48.888 INFO:teuthology.orchestra.run.smithi196.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.889 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T05:04:48.890 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T05:04:48.891 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T05:04:48.892 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T05:04:48.893 INFO:teuthology.orchestra.run.smithi196.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T05:04:48.894 INFO:teuthology.orchestra.run.smithi196.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T05:04:48.895 INFO:teuthology.orchestra.run.smithi196.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T05:04:48.895 INFO:teuthology.orchestra.run.smithi196.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T05:04:48.895 INFO:teuthology.orchestra.run.smithi196.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T05:04:48.895 INFO:teuthology.orchestra.run.smithi196.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T05:04:48.895 INFO:teuthology.orchestra.run.smithi196.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T05:04:48.895 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:04:48.895 INFO:teuthology.orchestra.run.smithi196.stdout:Complete! 2024-05-09T05:04:49.273 DEBUG:teuthology.parallel:result is None 2024-05-09T05:04:49.354 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T05:04:49.389 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T05:04:49.410 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T05:04:50.307 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 152/164 2024-05-09T05:04:50.402 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T05:04:50.453 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T05:04:50.547 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T05:04:50.580 INFO:teuthology.orchestra.run.smithi001.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T05:04:50.606 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T05:04:50.606 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T05:04:50.690 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T05:04:50.732 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T05:04:50.732 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T05:04:50.816 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T05:04:50.841 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T05:04:50.852 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T05:04:50.852 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T05:04:51.035 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T05:04:51.035 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T05:04:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T05:04:51.195 INFO:teuthology.orchestra.run.smithi001.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T05:04:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T05:04:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 1/164 2024-05-09T05:04:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 2/164 2024-05-09T05:04:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 3/164 2024-05-09T05:04:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 4/164 2024-05-09T05:04:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3594.gc9da4 5/164 2024-05-09T05:04:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 6/164 2024-05-09T05:04:56.648 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 7/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 8/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 9/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 10/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 11/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 12/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 13/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 14/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_ 15/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 16/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 17/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_6 18/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86 19/164 2024-05-09T05:04:56.649 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 20/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el 21/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ceph-common-2:19.0.0-3594.gc9da4049.el9. 22/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_6 23/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 24/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 25/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 26/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 27/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 28/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 29/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049. 30/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noa 31/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.n 32/164 2024-05-09T05:04:56.650 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9d 33/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el 34/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 35/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.e 36/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 37/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 38/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 39/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T05:04:56.651 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T05:04:56.652 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T05:04:56.653 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T05:04:56.654 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T05:04:56.655 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T05:04:56.656 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T05:04:56.657 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T05:04:56.658 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 149/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 151/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T05:04:56.659 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T05:04:56.660 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T05:04:59.349 INFO:teuthology.orchestra.run.smithi001.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T05:04:59.350 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:59.350 INFO:teuthology.orchestra.run.smithi001.stdout:Upgraded: 2024-05-09T05:04:59.350 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T05:04:59.350 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T05:04:59.350 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T05:04:59.350 INFO:teuthology.orchestra.run.smithi001.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T05:04:59.351 INFO:teuthology.orchestra.run.smithi001.stdout: librados2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.351 INFO:teuthology.orchestra.run.smithi001.stdout: librbd1-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.351 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T05:04:59.351 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T05:04:59.351 INFO:teuthology.orchestra.run.smithi001.stdout:Installed: 2024-05-09T05:04:59.351 INFO:teuthology.orchestra.run.smithi001.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T05:04:59.352 INFO:teuthology.orchestra.run.smithi001.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T05:04:59.352 INFO:teuthology.orchestra.run.smithi001.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T05:04:59.352 INFO:teuthology.orchestra.run.smithi001.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T05:04:59.352 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.352 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-base-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.352 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.352 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.353 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-grafana-dashboards-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.353 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-immutable-object-cache-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.353 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mds-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.353 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.353 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.353 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-dashboard-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.353 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.354 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-modules-core-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.354 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mgr-rook-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.354 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-mon-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.354 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-osd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.355 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-prometheus-alerts-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.355 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-radosgw-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.355 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-selinux-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.355 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-test-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.355 INFO:teuthology.orchestra.run.smithi001.stdout: ceph-volume-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.355 INFO:teuthology.orchestra.run.smithi001.stdout: cephadm-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.355 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.355 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-shell-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: cephfs-top-2:19.0.0-3594.gc9da4049.el9.noarch 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T05:04:59.356 INFO:teuthology.orchestra.run.smithi001.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs-devel-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: libcephfs2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: libcephsqlite-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: librados-devel-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: libradosstriper1-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.357 INFO:teuthology.orchestra.run.smithi001.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: librgw2-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T05:04:59.358 INFO:teuthology.orchestra.run.smithi001.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T05:04:59.359 INFO:teuthology.orchestra.run.smithi001.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-argparse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ceph-common-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cephfs-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T05:04:59.360 INFO:teuthology.orchestra.run.smithi001.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T05:04:59.361 INFO:teuthology.orchestra.run.smithi001.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T05:04:59.362 INFO:teuthology.orchestra.run.smithi001.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T05:04:59.363 INFO:teuthology.orchestra.run.smithi001.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rados-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rbd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rgw-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T05:04:59.364 INFO:teuthology.orchestra.run.smithi001.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-fuse-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.365 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-mirror-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: rbd-nbd-2:19.0.0-3594.gc9da4049.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:04:59.366 INFO:teuthology.orchestra.run.smithi001.stdout:Complete! 2024-05-09T05:04:59.796 DEBUG:teuthology.parallel:result is None 2024-05-09T05:04:59.800 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T05:04:59.958 DEBUG:teuthology.orchestra.run.smithi001:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T05:04:59.981 INFO:teuthology.orchestra.run.smithi001.stdout:19.0.0-3594.gc9da4049.el9 2024-05-09T05:04:59.982 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3594.gc9da4049.el9 2024-05-09T05:04:59.982 INFO:teuthology.task.install:The correct ceph version 19.0.0-3594.gc9da4049 is installed. 2024-05-09T05:04:59.984 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T05:05:00.129 DEBUG:teuthology.orchestra.run.smithi196:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T05:05:00.156 INFO:teuthology.orchestra.run.smithi196.stdout:19.0.0-3594.gc9da4049.el9 2024-05-09T05:05:00.156 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3594.gc9da4049.el9 2024-05-09T05:05:00.156 INFO:teuthology.task.install:The correct ceph version 19.0.0-3594.gc9da4049 is installed. 2024-05-09T05:05:00.159 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T05:05:00.160 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:00.160 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T05:05:00.189 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:00.189 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T05:05:00.228 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T05:05:00.228 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:00.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T05:05:00.256 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T05:05:00.322 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:00.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T05:05:00.350 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T05:05:00.417 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T05:05:00.418 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:00.418 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T05:05:00.443 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T05:05:00.509 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:00.509 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T05:05:00.536 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T05:05:00.601 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T05:05:00.602 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:00.602 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T05:05:00.628 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T05:05:00.694 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:00.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T05:05:00.721 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T05:05:00.788 INFO:teuthology.run_tasks:Running task ceph... 2024-05-09T05:05:00.876 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-05-09T05:05:00.876 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 777 /var/log/ceph 2024-05-09T05:05:00.878 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 777 /var/log/ceph 2024-05-09T05:05:00.905 INFO:tasks.ceph:Disabling ceph logrotate... 2024-05-09T05:05:00.905 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T05:05:00.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/logrotate.d/ceph 2024-05-09T05:05:00.972 INFO:tasks.ceph:Creating extra log directories... 2024-05-09T05:05:00.972 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T05:05:01.012 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-05-09T05:05:01.047 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2024-05-09T05:05:01.047 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2024-05-09T05:05:01.047 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2024-05-09T05:05:01.047 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:01.047 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-09T05:05:01.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-09T05:05:01.205 DEBUG:teuthology.orchestra.run.smithi001:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-09T05:05:01.232 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:01.232 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2024-05-09T05:05:01.249 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2024-05-09T05:05:01.366 DEBUG:teuthology.orchestra.run.smithi196:> sudo chcon system_u:object_r:etc_t:s0 /etc/logrotate.d/ceph-test.conf 2024-05-09T05:05:01.399 INFO:tasks.ceph:Creating ceph cluster ceph... 2024-05-09T05:05:01.399 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-05-09T05:05:01.399 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092', 'branch': 'wip-pdonnell-testing-20240508.183908-debug', 'description': 'fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}}', 'email': 'pdonnell@redhat.com', 'first_in_suite': False, 'job_id': '7699092', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'mgrmodules': {'sequential': [{'print': 'Enabling mgr modules'}]}, 'name': 'pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 20}}, {'machine': {'disk': 200}}], 'os_type': 'centos', 'os_version': '9.stream', 'overrides': {'admin_socket': {'branch': 'wip-pdonnell-testing-20240508.183908-debug'}, 'ceph': {'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}, 'ceph-deploy': {'bluestore': True, 'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs'}, 'check-counter': {'counters': {'mds': ['mds.exported', 'mds.imported']}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}}, 'selinux': {'allowlist': ['scontext=system_u:system_r:getty_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-pdonnell-testing-20240508.183908-debug', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}}, 'owner': 'scheduled_pdonnell@teuthology', 'priority': 90, 'repo': 'https://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'mds.c', 'mds.e', 'osd.0', 'osd.1', 'osd.2', 'osd.3', 'client.0'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'mds.d', 'mds.f', 'osd.4', 'osd.5', 'osd.6', 'osd.7']], 'seed': 1722, 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'sleep_before_teardown': 0, 'subset': '67/512', 'suite': 'fs', 'suite_branch': 'wip-pdonnell-testing-20240508.183908-debug', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_c9da4049f638c0f84bdd9b4e33c42e86046ac554/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph-ci.git', 'suite_sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'syslog': {'ignorelist': ['WARNING*.*check_session_state', 'WARNING*.*__ceph_remove_cap']}, 'targets': {'smithi001.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBO/n+q42gdSoJkaKX5+1CUYGh1jjCsWdZwguogGkEOIS8ePxUmO7X1TwMMLkKgSgrTOFI1Ppel1IVYmQLnV3CEk=', 'smithi196.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNaeC+RcduGGUB84dsZNrzFIzXDMcxwZXW7HahP+aGXOIYoBzOzBTwJFEHLXfCWU9DZgB5Ig+oje3Sy4x3PEn4o='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'}}, {'ceph': {'create_rbd_pool': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300}, 'conf': {'client': {'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'do not have an application enabled', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'Degraded data redundancy', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(POOL_', '\\(CACHE_POOL_', '\\(SMALLER_PGP_NUM\\)', '\\(OBJECT_', '\\(SLOW_OPS\\)', '\\(REQUEST_SLOW\\)', '\\(TOO_FEW_PGS\\)', 'slow request', 'timeout on replica', 'late reservation from', 'MON_DOWN', 'OSDMAP_FLAGS', 'OSD_DOWN', 'PG_DEGRADED', 'PG_AVAILABILITY', 'POOL_APP_NOT_ENABLED', 'mons down', 'mon down', 'out of quorum', 'overall HEALTH_', '\\(MON_DOWN\\)', 'SLOW_OPS', 'slow request'], 'sha1': 'c9da4049f638c0f84bdd9b4e33c42e86046ac554', 'cluster': 'ceph'}}, {'sequential': ['mgrmodules']}, {'ceph-fuse': None}, {'mon_thrash': {'check_mds_failover': True, 'revive_delay': 20, 'thrash_delay': 10}}, {'workunit': {'clients': {'all': ['suites/ffsb.sh']}}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': ['local function is_kupstream() return false end', 'local function is_kdistro() return false end', 'local function is_fuse() return true end', "local function syntax_version() return '' end"]}, 'teuthology_branch': 'main', 'teuthology_sha1': '1ae7ad82388e92a475afff437d49054826c019a1', 'timestamp': '2024-05-08_22:06:20', 'tube': 'smithi', 'user': 'pdonnell', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885'} 2024-05-09T05:05:01.400 DEBUG:teuthology.orchestra.run.smithi001:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T05:05:01.403 DEBUG:teuthology.orchestra.run.smithi196:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2024-05-09T05:05:01.449 DEBUG:teuthology.orchestra.run.smithi001:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T05:05:01.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo install -d -m0777 -- /var/run/ceph 2024-05-09T05:05:01.515 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:01.515 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T05:05:01.541 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T05:05:01.541 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_1 2024-05-09T05:05:01.597 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T05:05:01.597 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T05:05:01.597 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-09T05:05:01.598 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:05:01.598 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T05:05:01.598 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-05-09 05:04:51.538117212 +0000 2024-05-09T05:05:01.598 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-05-09 05:02:55.823069393 +0000 2024-05-09T05:05:01.598 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-05-09 05:02:55.823069393 +0000 2024-05-09T05:05:01.598 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-05-09 05:02:55.823069393 +0000 2024-05-09T05:05:01.599 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T05:05:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-05-09T05:05:01.662 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-05-09T05:05:01.662 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 0.000101483 s, 5.0 MB/s 2024-05-09T05:05:01.663 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T05:05:01.720 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_2 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-05-09 05:04:51.538117212 +0000 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-05-09 05:02:56.176065062 +0000 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-05-09 05:02:56.176065062 +0000 2024-05-09T05:05:01.777 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-05-09 05:02:56.176065062 +0000 2024-05-09T05:05:01.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T05:05:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-05-09T05:05:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-05-09T05:05:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.4834e-05 s, 5.4 MB/s 2024-05-09T05:05:01.842 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T05:05:01.899 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_3 2024-05-09T05:05:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T05:05:01.955 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T05:05:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-09T05:05:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:05:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T05:05:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-05-09 05:04:51.539117200 +0000 2024-05-09T05:05:01.955 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-05-09 05:02:56.535060657 +0000 2024-05-09T05:05:01.956 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-05-09 05:02:56.535060657 +0000 2024-05-09T05:05:01.956 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-05-09 05:02:56.535060657 +0000 2024-05-09T05:05:01.956 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T05:05:02.019 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-05-09T05:05:02.019 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-05-09T05:05:02.019 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 9.955e-05 s, 5.1 MB/s 2024-05-09T05:05:02.021 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T05:05:02.078 DEBUG:teuthology.orchestra.run.smithi001:> stat /dev/vg_nvme/lv_4 2024-05-09T05:05:02.134 INFO:teuthology.orchestra.run.smithi001.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T05:05:02.134 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T05:05:02.134 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T05:05:02.134 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:05:02.134 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T05:05:02.134 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 2024-05-09 05:04:51.539117200 +0000 2024-05-09T05:05:02.134 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-05-09 05:02:56.881056411 +0000 2024-05-09T05:05:02.134 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-05-09 05:02:56.881056411 +0000 2024-05-09T05:05:02.135 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: 2024-05-09 05:02:56.881056411 +0000 2024-05-09T05:05:02.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T05:05:02.198 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records in 2024-05-09T05:05:02.198 INFO:teuthology.orchestra.run.smithi001.stderr:1+0 records out 2024-05-09T05:05:02.198 INFO:teuthology.orchestra.run.smithi001.stderr:512 bytes copied, 7.9695e-05 s, 6.4 MB/s 2024-05-09T05:05:02.199 DEBUG:teuthology.orchestra.run.smithi001:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T05:05:02.256 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:02.256 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:02.256 DEBUG:teuthology.orchestra.run.smithi196:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T05:05:02.275 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T05:05:02.275 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_1 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-05-09 05:04:41.054834095 +0000 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-05-09 05:02:43.612217464 +0000 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-05-09 05:02:43.612217464 +0000 2024-05-09T05:05:02.333 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-05-09 05:02:43.612217464 +0000 2024-05-09T05:05:02.334 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T05:05:02.399 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-05-09T05:05:02.400 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-05-09T05:05:02.400 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000235917 s, 2.2 MB/s 2024-05-09T05:05:02.401 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T05:05:02.460 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_2 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-05-09 05:04:41.055834075 +0000 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-05-09 05:02:43.987209662 +0000 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-05-09 05:02:43.987209662 +0000 2024-05-09T05:05:02.518 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-05-09 05:02:43.987209662 +0000 2024-05-09T05:05:02.519 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T05:05:02.583 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-05-09T05:05:02.583 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-05-09T05:05:02.583 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000165419 s, 3.1 MB/s 2024-05-09T05:05:02.585 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T05:05:02.641 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_3 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 704 Links: 1 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-05-09 05:04:41.055834075 +0000 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-05-09 05:02:44.339202339 +0000 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-05-09 05:02:44.339202339 +0000 2024-05-09T05:05:02.697 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-05-09 05:02:44.339202339 +0000 2024-05-09T05:05:02.698 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T05:05:02.761 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-05-09T05:05:02.762 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-05-09T05:05:02.762 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000201943 s, 2.5 MB/s 2024-05-09T05:05:02.763 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T05:05:02.820 DEBUG:teuthology.orchestra.run.smithi196:> stat /dev/vg_nvme/lv_4 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout:Access: 2024-05-09 05:04:41.056834054 +0000 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout:Modify: 2024-05-09 05:02:44.680195245 +0000 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout:Change: 2024-05-09 05:02:44.680195245 +0000 2024-05-09T05:05:02.876 INFO:teuthology.orchestra.run.smithi196.stdout: Birth: 2024-05-09 05:02:44.680195245 +0000 2024-05-09T05:05:02.877 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T05:05:02.940 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records in 2024-05-09T05:05:02.940 INFO:teuthology.orchestra.run.smithi196.stderr:1+0 records out 2024-05-09T05:05:02.940 INFO:teuthology.orchestra.run.smithi196.stderr:512 bytes copied, 0.000194237 s, 2.6 MB/s 2024-05-09T05:05:02.942 DEBUG:teuthology.orchestra.run.smithi196:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T05:05:02.998 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:02.999 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-09T05:05:02.999 INFO:tasks.ceph:Generating config... 2024-05-09T05:05:03.000 INFO:tasks.ceph:[client] client mount timeout = 600 2024-05-09T05:05:03.000 INFO:tasks.ceph:[client] client_shutdown_timeout = 120 2024-05-09T05:05:03.000 INFO:tasks.ceph:[client] debug client = 20 2024-05-09T05:05:03.000 INFO:tasks.ceph:[client] debug ms = 1 2024-05-09T05:05:03.000 INFO:tasks.ceph:[client] rados mon op timeout = 900 2024-05-09T05:05:03.000 INFO:tasks.ceph:[client] rados osd op timeout = 900 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] debug mds = 20 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] debug mds balancer = 20 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] debug ms = 1 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] mds bal merge size = 5 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] mds bal split bits = 3 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] mds bal split size = 100 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] mds debug frag = True 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] mds oft prefetch dirfrags = True 2024-05-09T05:05:03.001 INFO:tasks.ceph:[mds] mds op complaint time = 180 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mds] mds verify scatter = True 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mds] mds_dir_prefetch = True 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mds] mds_sleep_rank_change = 5000000.0 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mds] osd op complaint time = 180 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mds] rados mon op timeout = 900 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mds] rados osd op timeout = 900 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mgr] debug client = 20 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mgr] debug ms = 1 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mon] debug mon = 20 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mon] debug ms = 1 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mon] debug paxos = 20 2024-05-09T05:05:03.002 INFO:tasks.ceph:[mon] mon down mkfs grace = 300 2024-05-09T05:05:03.003 INFO:tasks.ceph:[mon] mon op complaint time = 120 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] bdev async discard = True 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] bluestore allocator = bitmap 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] debug ms = 1 2024-05-09T05:05:03.003 INFO:tasks.ceph:[osd] debug osd = 20 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] filestore flush min = 0 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] osd blocked scrub grace period = 3600 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] osd max markdown count = 1000 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-05-09T05:05:03.004 INFO:tasks.ceph:[osd] osd op complaint time = 180 2024-05-09T05:05:03.004 INFO:tasks.ceph:Setting up mon.a... 2024-05-09T05:05:03.005 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2024-05-09T05:05:03.068 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.keyring 2024-05-09T05:05:03.072 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2024-05-09T05:05:03.129 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T05:05:03.155 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.1'), ('mon.b', '172.21.15.196'), ('mon.c', '[v2:172.21.15.196:3301,v1:172.21.15.196:6790]')] 2024-05-09T05:05:03.155 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.1,172.21.15.196,[v2:172.21.15.196:3301,v1:172.21.15.196:6790]'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd blocked scrub grace period': 3600, 'osd failsafe full ratio': 0.95, 'osd max markdown count': 1000, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug client': 20}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false', 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'client mount timeout': 600, 'client_shutdown_timeout': 120, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}, 'mds.a': {}, 'mds.c': {}, 'mds.e': {}, 'mds.b': {}, 'mds.d': {}, 'mds.f': {}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds oft prefetch dirfrags': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_dir_prefetch': True, 'mds_sleep_rank_change': 5000000.0, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}} 2024-05-09T05:05:03.156 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:03.156 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T05:05:03.211 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.1 --add b 172.21.15.196 --addv c '[v2:172.21.15.196:3301,v1:172.21.15.196:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: generated fsid aa0bbdea-07dc-4c3a-9697-b556ae629556 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:setting min_mon_release = quincy 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:epoch 0 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:fsid aa0bbdea-07dc-4c3a-9697-b556ae629556 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:last_changed 2024-05-09T05:05:03.309340+0000 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:created 2024-05-09T05:05:03.309340+0000 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:min_mon_release 17 (quincy) 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:election_strategy: 1 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:0: [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] mon.a 2024-05-09T05:05:03.309 INFO:teuthology.orchestra.run.smithi001.stdout:1: [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] mon.b 2024-05-09T05:05:03.310 INFO:teuthology.orchestra.run.smithi001.stdout:2: [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] mon.c 2024-05-09T05:05:03.310 INFO:teuthology.orchestra.run.smithi001.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2024-05-09T05:05:03.311 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-05-09T05:05:03.326 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID aa0bbdea-07dc-4c3a-9697-b556ae629556... 2024-05-09T05:05:03.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T05:05:03.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2024-05-09T05:05:03.410 INFO:teuthology.orchestra.run.smithi001.stdout:[global] 2024-05-09T05:05:03.410 INFO:teuthology.orchestra.run.smithi001.stdout: chdir = "" 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: auth supported = cephx 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: filestore xattr use omap = true 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: mon clock drift allowed = 1.000 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: osd crush chooseleaf type = 0 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: auth debug = true 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on old message = true 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: ms die on bug = true 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: mon pg warn max object skew = 0 2024-05-09T05:05:03.411 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default size = 2 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow primary affinity = true 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd allow pg remap = true 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on legacy crush tunables = false 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on crush straw calc version zero = false 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on no sortbitwise = false 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on osd down out interval zero = false 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon warn on too few osds = false 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T05:05:03.412 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: mon_allow_pool_size_one = true 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: osd default data pool replay window = 5 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: mon allow pool delete = true 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: mon cluster log file level = debug 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: debug asserts on shutdown = true 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: mon health detail to clog = false 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: mon host = "172.21.15.1,172.21.15.196,[v2:172.21.15.196:3301,v1:172.21.15.196:6790]" 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: fsid = aa0bbdea-07dc-4c3a-9697-b556ae629556 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout:[osd] 2024-05-09T05:05:03.413 INFO:teuthology.orchestra.run.smithi001.stdout: osd journal size = 100 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub load threshold = 5.0 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd scrub max interval = 600 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd mclock profile = high_recovery_ops 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd recover clone overlap = true 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd recovery max chunk = 1048576 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug shutdown = true 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug op order = true 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug verify stray on activate = true 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug trim objects = true 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd open classes on start = true 2024-05-09T05:05:03.414 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug pg log writeout = true 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: osd deep scrub update digest min age = 30 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: osd map max advance = 10 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: journal zero on create = true 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: filestore ondisk finisher threads = 3 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: filestore apply finisher threads = 3 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: bdev debug aio = true 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: osd debug misdirected ops = true 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: bdev async discard = True 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: bdev enable discard = True 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore allocator = bitmap 2024-05-09T05:05:03.415 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore block size = 96636764160 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: bluestore fsck on mount = True 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluefs = 1/20 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: debug bluestore = 1/20 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: debug osd = 20 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: debug rocksdb = 4/10 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: filestore flush min = 0 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd full ratio = 0.9 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd nearfull ratio = 0.8 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: osd blocked scrub grace period = 3600 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: osd failsafe full ratio = 0.95 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: osd max markdown count = 1000 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: osd objectstore = bluestore 2024-05-09T05:05:03.416 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr] 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: debug mgr = 20 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: mgr/telemetry/nag = false 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout:[mon] 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-05-09T05:05:03.417 INFO:teuthology.orchestra.run.smithi001.stdout: debug mon = 20 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: debug paxos = 20 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: debug auth = 20 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: mon data avail warn = 5 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: mon mgr mkfs grace = 240 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min pgs per osd = 4 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd reporter subtree level = osd 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: mon osd prime pg temp = true 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: mon reweight min bytes per osd = 10 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.418 INFO:teuthology.orchestra.run.smithi001.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: # 1m isn't quite enough 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: mon_down_mkfs_grace = 2m 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: mon_warn_on_filestore_osds = false 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: mon down mkfs grace = 300 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: mon op complaint time = 120 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout:[client] 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: rgw cache enabled = true 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable ops log = true 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: rgw enable usage log = true 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T05:05:03.419 INFO:teuthology.orchestra.run.smithi001.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout: client mount timeout = 600 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout: client_shutdown_timeout = 120 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout: debug client = 20 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.a] 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.b] 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:[mon.c] 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.e] 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-05-09T05:05:03.420 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.f] 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout:[mds] 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds = 20 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: debug mds balancer = 20 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: debug ms = 1 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds bal fragment size max = 10000 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds bal merge size = 5 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds bal split bits = 3 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds bal split size = 100 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug frag = True 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds debug scatterstat = True 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds oft prefetch dirfrags = True 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds op complaint time = 180 2024-05-09T05:05:03.421 INFO:teuthology.orchestra.run.smithi001.stdout: mds verify scatter = True 2024-05-09T05:05:03.422 INFO:teuthology.orchestra.run.smithi001.stdout: mds_dir_prefetch = True 2024-05-09T05:05:03.422 INFO:teuthology.orchestra.run.smithi001.stdout: mds_sleep_rank_change = 5000000.0 2024-05-09T05:05:03.422 INFO:teuthology.orchestra.run.smithi001.stdout: osd op complaint time = 180 2024-05-09T05:05:03.422 INFO:teuthology.orchestra.run.smithi001.stdout: rados mon op timeout = 900 2024-05-09T05:05:03.422 INFO:teuthology.orchestra.run.smithi001.stdout: rados osd op timeout = 900 2024-05-09T05:05:03.422 INFO:teuthology.orchestra.run.smithi196.stdout:[global] 2024-05-09T05:05:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: chdir = "" 2024-05-09T05:05:03.422 INFO:teuthology.orchestra.run.smithi196.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: auth supported = cephx 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: filestore xattr use omap = true 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: mon clock drift allowed = 1.000 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: osd crush chooseleaf type = 0 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: auth debug = true 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on old message = true 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: ms die on bug = true 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: mon max pg per osd = 10000 # >= luminous 2024-05-09T05:05:03.423 INFO:teuthology.orchestra.run.smithi196.stdout: mon pg warn max object skew = 0 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: # disable pg_autoscaler by default for new pools 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: osd_pool_default_pg_autoscale_mode = off 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default size = 2 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow primary affinity = true 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd allow pg remap = true 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on legacy crush tunables = false 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on crush straw calc version zero = false 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on no sortbitwise = false 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on osd down out interval zero = false 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: mon warn on too few osds = false 2024-05-09T05:05:03.424 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_pool_no_redundancy = false 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: mon_allow_pool_size_one = true 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: osd default data pool replay window = 5 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: mon allow pool delete = true 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: mon cluster log file level = debug 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: debug asserts on shutdown = true 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: mon health detail to clog = false 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: mon host = "172.21.15.1,172.21.15.196,[v2:172.21.15.196:3301,v1:172.21.15.196:6790]" 2024-05-09T05:05:03.425 INFO:teuthology.orchestra.run.smithi196.stdout: fsid = aa0bbdea-07dc-4c3a-9697-b556ae629556 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout:[osd] 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: osd journal size = 100 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub load threshold = 5.0 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: osd scrub max interval = 600 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: osd mclock profile = high_recovery_ops 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: osd recover clone overlap = true 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: osd recovery max chunk = 1048576 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug shutdown = true 2024-05-09T05:05:03.426 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug op order = true 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug verify stray on activate = true 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug trim objects = true 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: osd open classes on start = true 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug pg log writeout = true 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: osd deep scrub update digest min age = 30 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: osd map max advance = 10 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: journal zero on create = true 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: filestore ondisk finisher threads = 3 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: filestore apply finisher threads = 3 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.427 INFO:teuthology.orchestra.run.smithi196.stdout: bdev debug aio = true 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: osd debug misdirected ops = true 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: bdev async discard = True 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: bdev enable discard = True 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore allocator = bitmap 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore block size = 96636764160 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: bluestore fsck on mount = True 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluefs = 1/20 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: debug bluestore = 1/20 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: debug osd = 20 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: debug rocksdb = 4/10 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: filestore flush min = 0 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd backfillfull_ratio = 0.85 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd full ratio = 0.9 2024-05-09T05:05:03.428 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd nearfull ratio = 0.8 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: osd blocked scrub grace period = 3600 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: osd failsafe full ratio = 0.95 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: osd max markdown count = 1000 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: osd objectstore = bluestore 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: osd op complaint time = 180 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr] 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: debug mgr = 20 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: mgr/telemetry/nag = false 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: debug client = 20 2024-05-09T05:05:03.429 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout:[mon] 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: debug mon = 20 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: debug paxos = 20 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: debug auth = 20 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: mon data avail warn = 5 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: mon mgr mkfs grace = 240 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min pgs per osd = 4 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd reporter subtree level = osd 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: mon osd prime pg temp = true 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: mon reweight min bytes per osd = 10 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: auth mon ticket ttl = 660 # 11m 2024-05-09T05:05:03.430 INFO:teuthology.orchestra.run.smithi196.stdout: auth service ticket ttl = 240 # 4m 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: # don't complain about insecure global_id in the test suite 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: # 1m isn't quite enough 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: mon_down_mkfs_grace = 2m 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: mon_warn_on_filestore_osds = false 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: mon down mkfs grace = 300 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: mon op complaint time = 120 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout: 2024-05-09T05:05:03.431 INFO:teuthology.orchestra.run.smithi196.stdout:[client] 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: rgw cache enabled = true 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable ops log = true 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: rgw enable usage log = true 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: client mount timeout = 600 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: client_shutdown_timeout = 120 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: debug client = 20 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: rados mon op timeout = 900 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout: rados osd op timeout = 900 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.a] 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.b] 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout:[mon.c] 2024-05-09T05:05:03.432 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.a] 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.c] 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.e] 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.b] 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.d] 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.f] 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout:[mds] 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: debug mds = 20 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: debug mds balancer = 20 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: debug ms = 1 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: mds bal fragment size max = 10000 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: mds bal merge size = 5 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: mds bal split bits = 3 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: mds bal split size = 100 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: mds debug frag = True 2024-05-09T05:05:03.433 INFO:teuthology.orchestra.run.smithi196.stdout: mds debug scatterstat = True 2024-05-09T05:05:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: mds oft prefetch dirfrags = True 2024-05-09T05:05:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: mds op complaint time = 180 2024-05-09T05:05:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: mds verify scatter = True 2024-05-09T05:05:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: mds_dir_prefetch = True 2024-05-09T05:05:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: mds_sleep_rank_change = 5000000.0 2024-05-09T05:05:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: osd op complaint time = 180 2024-05-09T05:05:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: rados mon op timeout = 900 2024-05-09T05:05:03.434 INFO:teuthology.orchestra.run.smithi196.stdout: rados osd op timeout = 900 2024-05-09T05:05:03.435 INFO:tasks.ceph:Creating admin key on mon.a... 2024-05-09T05:05:03.436 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2024-05-09T05:05:03.541 INFO:tasks.ceph:Copying monmap to all nodes... 2024-05-09T05:05:03.541 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:03.541 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2024-05-09T05:05:03.557 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:03.557 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2024-05-09T05:05:03.613 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:05:03.613 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:03.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T05:05:03.613 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T05:05:03.689 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:03.689 DEBUG:teuthology.orchestra.run.smithi001:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:03.745 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:05:03.745 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:03.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd of=/etc/ceph/ceph.keyring 2024-05-09T05:05:03.745 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0644 /etc/ceph/ceph.keyring 2024-05-09T05:05:03.780 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:03.780 DEBUG:teuthology.orchestra.run.smithi196:> dd of=/home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:03.836 INFO:tasks.ceph:Setting up mon nodes... 2024-05-09T05:05:03.836 INFO:tasks.ceph:Setting up mgr nodes... 2024-05-09T05:05:03.836 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T05:05:03.898 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2024-05-09T05:05:03.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T05:05:03.966 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2024-05-09T05:05:03.969 INFO:tasks.ceph:Setting up mds nodes... 2024-05-09T05:05:03.970 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2024-05-09T05:05:04.032 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2024-05-09T05:05:04.035 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2024-05-09T05:05:04.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-c && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.c /var/lib/ceph/mds/ceph-c/keyring 2024-05-09T05:05:04.162 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-c/keyring 2024-05-09T05:05:04.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-c 2024-05-09T05:05:04.191 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mds/ceph-e && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.e /var/lib/ceph/mds/ceph-e/keyring 2024-05-09T05:05:04.293 INFO:teuthology.orchestra.run.smithi001.stdout:creating /var/lib/ceph/mds/ceph-e/keyring 2024-05-09T05:05:04.296 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-e 2024-05-09T05:05:04.322 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2024-05-09T05:05:04.384 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2024-05-09T05:05:04.387 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2024-05-09T05:05:04.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mds/ceph-d && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.d /var/lib/ceph/mds/ceph-d/keyring 2024-05-09T05:05:04.511 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mds/ceph-d/keyring 2024-05-09T05:05:04.514 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-d 2024-05-09T05:05:04.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mds/ceph-f && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.f /var/lib/ceph/mds/ceph-f/keyring 2024-05-09T05:05:04.639 INFO:teuthology.orchestra.run.smithi196.stdout:creating /var/lib/ceph/mds/ceph-f/keyring 2024-05-09T05:05:04.642 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-f 2024-05-09T05:05:04.667 INFO:tasks.ceph_client:Setting up client nodes... 2024-05-09T05:05:04.667 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T05:05:04.721 INFO:teuthology.orchestra.run.smithi001.stdout:creating /etc/ceph/ceph.client.0.keyring 2024-05-09T05:05:04.735 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-05-09T05:05:04.735 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi001.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi196.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2024-05-09T05:05:04.735 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2024-05-09T05:05:04.761 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:04.761 INFO:tasks.ceph:role: osd.0 2024-05-09T05:05:04.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:05:04.761 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T05:05:04.826 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T05:05:04.826 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T05:05:04.826 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T05:05:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T05:05:04.827 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T05:05:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-05-09T05:05:04.827 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T05:05:04.827 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T05:05:04.827 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T05:05:04.827 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T05:05:04.859 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-05-09T05:05:04.861 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-05-09T05:05:04.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2024-05-09T05:05:04.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2024-05-09T05:05:05.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2024-05-09T05:05:05.074 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:05.074 INFO:tasks.ceph:role: osd.1 2024-05-09T05:05:05.074 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:05:05.074 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T05:05:05.139 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T05:05:05.140 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T05:05:05.172 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-05-09T05:05:05.174 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-05-09T05:05:05.174 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2024-05-09T05:05:05.244 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2024-05-09T05:05:05.314 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2024-05-09T05:05:05.380 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:05.380 INFO:tasks.ceph:role: osd.2 2024-05-09T05:05:05.380 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:05:05.380 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T05:05:05.445 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T05:05:05.445 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T05:05:05.446 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T05:05:05.446 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T05:05:05.446 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T05:05:05.446 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-05-09T05:05:05.446 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T05:05:05.446 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T05:05:05.446 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T05:05:05.446 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T05:05:05.479 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-05-09T05:05:05.481 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-05-09T05:05:05.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2024-05-09T05:05:05.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2024-05-09T05:05:05.624 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2024-05-09T05:05:05.690 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:05.690 INFO:tasks.ceph:role: osd.3 2024-05-09T05:05:05.690 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:05:05.690 DEBUG:teuthology.orchestra.run.smithi001:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T05:05:05.755 INFO:teuthology.orchestra.run.smithi001.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T05:05:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T05:05:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T05:05:05.755 INFO:teuthology.orchestra.run.smithi001.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T05:05:05.756 INFO:teuthology.orchestra.run.smithi001.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T05:05:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: = sunit=0 swidth=0 blks 2024-05-09T05:05:05.756 INFO:teuthology.orchestra.run.smithi001.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T05:05:05.756 INFO:teuthology.orchestra.run.smithi001.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T05:05:05.756 INFO:teuthology.orchestra.run.smithi001.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T05:05:05.756 INFO:teuthology.orchestra.run.smithi001.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T05:05:05.787 INFO:teuthology.orchestra.run.smithi001.stdout:Discarding blocks...Done. 2024-05-09T05:05:05.789 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi001.front.sepia.ceph.com -o noatime 2024-05-09T05:05:05.789 DEBUG:teuthology.orchestra.run.smithi001:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2024-05-09T05:05:05.859 DEBUG:teuthology.orchestra.run.smithi001:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2024-05-09T05:05:05.932 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:06.040+0000 7fc3b6686640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2024-05-09T05:05:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:06.041+0000 7fc3b6686640 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2024-05-09T05:05:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:06.041+0000 7fc3b6686640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T05:05:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:06.041+0000 7fc3b6686640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T05:05:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:06.041+0000 7fc3b6686640 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2024-05-09T05:05:06.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:06.041+0000 7fc3b6686640 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2024-05-09T05:05:08.897 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2024-05-09T05:05:08.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:09.030+0000 7f35f4bbc640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2024-05-09T05:05:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:09.030+0000 7f35f4bbc640 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2024-05-09T05:05:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:09.030+0000 7f35f4bbc640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T05:05:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:09.030+0000 7f35f4bbc640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T05:05:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:09.030+0000 7f35f4bbc640 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2024-05-09T05:05:09.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:09.030+0000 7f35f4bbc640 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2024-05-09T05:05:11.886 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2024-05-09T05:05:11.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:12.021+0000 7f6902f50640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2024-05-09T05:05:12.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:12.022+0000 7f6902f50640 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2024-05-09T05:05:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:12.022+0000 7f6902f50640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T05:05:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:12.022+0000 7f6902f50640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T05:05:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:12.022+0000 7f6902f50640 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2024-05-09T05:05:12.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:12.022+0000 7f6902f50640 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2024-05-09T05:05:14.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2024-05-09T05:05:14.910 DEBUG:teuthology.orchestra.run.smithi001:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:15.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:15.017+0000 7f4f8700e640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2024-05-09T05:05:15.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:15.018+0000 7f4f8700e640 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2024-05-09T05:05:15.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:15.018+0000 7f4f8700e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T05:05:15.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:15.018+0000 7f4f8700e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T05:05:15.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:15.018+0000 7f4f8700e640 -1 bluestore(/var/lib/ceph/osd/ceph-3/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-3/block: (2) No such file or directory 2024-05-09T05:05:15.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:15.018+0000 7f4f8700e640 -1 bluestore(/var/lib/ceph/osd/ceph-3) _read_fsid unparsable uuid 2024-05-09T05:05:17.869 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2024-05-09T05:05:17.900 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2024-05-09T05:05:17.930 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:17.930 INFO:tasks.ceph:role: osd.4 2024-05-09T05:05:17.931 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:05:17.931 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-05-09T05:05:18.005 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T05:05:18.006 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T05:05:18.524 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-05-09T05:05:18.526 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-05-09T05:05:18.526 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2024-05-09T05:05:18.663 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2024-05-09T05:05:18.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2024-05-09T05:05:18.768 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:18.768 INFO:tasks.ceph:role: osd.5 2024-05-09T05:05:18.768 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:05:18.768 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-05-09T05:05:18.842 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T05:05:18.842 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T05:05:18.842 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T05:05:18.842 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T05:05:18.842 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T05:05:18.842 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-05-09T05:05:18.843 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T05:05:18.843 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T05:05:18.843 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T05:05:18.843 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T05:05:19.289 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-05-09T05:05:19.291 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-05-09T05:05:19.291 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2024-05-09T05:05:19.383 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2024-05-09T05:05:19.412 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2024-05-09T05:05:19.478 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:19.479 INFO:tasks.ceph:role: osd.6 2024-05-09T05:05:19.479 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:05:19.479 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T05:05:19.551 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T05:05:20.042 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-05-09T05:05:20.044 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-05-09T05:05:20.044 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2024-05-09T05:05:20.160 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2024-05-09T05:05:20.191 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2024-05-09T05:05:20.257 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2024-05-09T05:05:20.257 INFO:tasks.ceph:role: osd.7 2024-05-09T05:05:20.257 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:05:20.257 DEBUG:teuthology.orchestra.run.smithi196:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2024-05-09T05:05:20.330 INFO:teuthology.orchestra.run.smithi196.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2024-05-09T05:05:20.330 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 attr=2, projid32bit=1 2024-05-09T05:05:20.330 INFO:teuthology.orchestra.run.smithi196.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-05-09T05:05:20.330 INFO:teuthology.orchestra.run.smithi196.stdout: = reflink=1 bigtime=1 inobtcount=1 nrext64=0 2024-05-09T05:05:20.330 INFO:teuthology.orchestra.run.smithi196.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-05-09T05:05:20.331 INFO:teuthology.orchestra.run.smithi196.stdout: = sunit=0 swidth=0 blks 2024-05-09T05:05:20.331 INFO:teuthology.orchestra.run.smithi196.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-05-09T05:05:20.331 INFO:teuthology.orchestra.run.smithi196.stdout:log =internal log bsize=4096 blocks=16384, version=2 2024-05-09T05:05:20.331 INFO:teuthology.orchestra.run.smithi196.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-05-09T05:05:20.331 INFO:teuthology.orchestra.run.smithi196.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-05-09T05:05:20.843 INFO:teuthology.orchestra.run.smithi196.stdout:Discarding blocks...Done. 2024-05-09T05:05:20.845 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi196.front.sepia.ceph.com -o noatime 2024-05-09T05:05:20.845 DEBUG:teuthology.orchestra.run.smithi196:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2024-05-09T05:05:20.971 DEBUG:teuthology.orchestra.run.smithi196:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2024-05-09T05:05:21.003 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:21.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:21.109+0000 7fdcbf220640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2024-05-09T05:05:21.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:21.110+0000 7fdcbf220640 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2024-05-09T05:05:21.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:21.110+0000 7fdcbf220640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T05:05:21.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:21.110+0000 7fdcbf220640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T05:05:21.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:21.110+0000 7fdcbf220640 -1 bluestore(/var/lib/ceph/osd/ceph-4/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-4/block: (2) No such file or directory 2024-05-09T05:05:21.111 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:21.110+0000 7fdcbf220640 -1 bluestore(/var/lib/ceph/osd/ceph-4) _read_fsid unparsable uuid 2024-05-09T05:05:23.964 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2024-05-09T05:05:23.991 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:24.088 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:24.086+0000 7f7f6b218640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2024-05-09T05:05:24.088 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:24.086+0000 7f7f6b218640 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2024-05-09T05:05:24.088 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:24.086+0000 7f7f6b218640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T05:05:24.088 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:24.086+0000 7f7f6b218640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T05:05:24.089 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:24.086+0000 7f7f6b218640 -1 bluestore(/var/lib/ceph/osd/ceph-5/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-5/block: (2) No such file or directory 2024-05-09T05:05:24.089 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:24.087+0000 7f7f6b218640 -1 bluestore(/var/lib/ceph/osd/ceph-5) _read_fsid unparsable uuid 2024-05-09T05:05:26.949 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2024-05-09T05:05:26.979 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:27.084 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:27.083+0000 7f5075131640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2024-05-09T05:05:27.084 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:27.083+0000 7f5075131640 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2024-05-09T05:05:27.085 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:27.083+0000 7f5075131640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T05:05:27.085 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:27.083+0000 7f5075131640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T05:05:27.085 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:27.083+0000 7f5075131640 -1 bluestore(/var/lib/ceph/osd/ceph-6/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-6/block: (2) No such file or directory 2024-05-09T05:05:27.085 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:27.083+0000 7f5075131640 -1 bluestore(/var/lib/ceph/osd/ceph-6) _read_fsid unparsable uuid 2024-05-09T05:05:29.939 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2024-05-09T05:05:29.966 DEBUG:teuthology.orchestra.run.smithi196:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:30.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:30.063+0000 7f0d5fe2c640 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2024-05-09T05:05:30.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:30.063+0000 7f0d5fe2c640 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2024-05-09T05:05:30.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:30.063+0000 7f0d5fe2c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T05:05:30.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:30.063+0000 7f0d5fe2c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T05:05:30.065 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:30.063+0000 7f0d5fe2c640 -1 bluestore(/var/lib/ceph/osd/ceph-7/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-7/block: (2) No such file or directory 2024-05-09T05:05:30.066 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:30.064+0000 7f0d5fe2c640 -1 bluestore(/var/lib/ceph/osd/ceph-7) _read_fsid unparsable uuid 2024-05-09T05:05:31.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:05:31.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:05:32.910 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2024-05-09T05:05:32.936 INFO:tasks.ceph:Reading keys from all nodes... 2024-05-09T05:05:32.936 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:32.936 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mgr/ceph-y/keyring of=/dev/stdout 2024-05-09T05:05:32.968 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:32.968 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-a/keyring of=/dev/stdout 2024-05-09T05:05:33.040 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:33.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-c/keyring of=/dev/stdout 2024-05-09T05:05:33.110 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:33.110 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/mds/ceph-e/keyring of=/dev/stdout 2024-05-09T05:05:33.180 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:33.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2024-05-09T05:05:33.250 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:33.250 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2024-05-09T05:05:33.320 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:33.321 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2024-05-09T05:05:33.390 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:33.390 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/keyring of=/dev/stdout 2024-05-09T05:05:33.459 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:33.460 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2024-05-09T05:05:33.488 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:33.488 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mds/ceph-b/keyring of=/dev/stdout 2024-05-09T05:05:33.554 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:33.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mds/ceph-d/keyring of=/dev/stdout 2024-05-09T05:05:33.621 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:33.622 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/mds/ceph-f/keyring of=/dev/stdout 2024-05-09T05:05:33.687 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:33.687 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/keyring of=/dev/stdout 2024-05-09T05:05:33.752 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:33.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/keyring of=/dev/stdout 2024-05-09T05:05:33.817 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:33.817 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/keyring of=/dev/stdout 2024-05-09T05:05:33.882 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:33.882 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/keyring of=/dev/stdout 2024-05-09T05:05:33.947 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:33.947 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/etc/ceph/ceph.client.0.keyring of=/dev/stdout 2024-05-09T05:05:33.967 INFO:tasks.ceph:Adding keys to all mons... 2024-05-09T05:05:33.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T05:05:34.010 DEBUG:teuthology.orchestra.run.smithi196:> sudo tee -a /etc/ceph/ceph.keyring 2024-05-09T05:05:34.034 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.y] 2024-05-09T05:05:34.034 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQB/WTxm0QqPNRAANRkjBtkskW3rPiZ3UO65zw== 2024-05-09T05:05:34.034 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.a] 2024-05-09T05:05:34.034 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCAWTxm4szwARAANUZCmnl2EF7hCl6zKHAf4A== 2024-05-09T05:05:34.035 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.c] 2024-05-09T05:05:34.035 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCAWTxmmFu0CRAAQ5uMSxZcH+MTNDnj+PqzRw== 2024-05-09T05:05:34.035 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.e] 2024-05-09T05:05:34.035 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCAWTxmjOyDERAArh+1RtBJJkoGis2wWI9r1A== 2024-05-09T05:05:34.035 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.0] 2024-05-09T05:05:34.035 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCCWTxmrLeMAhAARZyvb9QuapGWrawda9MNUQ== 2024-05-09T05:05:34.036 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.1] 2024-05-09T05:05:34.036 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCFWTxm31PtARAAlGdQHAODO0PiIUIo+VKG0A== 2024-05-09T05:05:34.036 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.2] 2024-05-09T05:05:34.036 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCIWTxmXfJbARAAJRU4b919LVh/Sjd3IzlxIw== 2024-05-09T05:05:34.036 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.3] 2024-05-09T05:05:34.036 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCLWTxmhiQfARAAb9lucVVOV3pAUTVxgie6GA== 2024-05-09T05:05:34.037 INFO:teuthology.orchestra.run.smithi196.stdout:[mgr.x] 2024-05-09T05:05:34.037 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQB/WTxmul+kORAAEuPH6Pe82FTCLTy8m3kvNQ== 2024-05-09T05:05:34.037 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.b] 2024-05-09T05:05:34.037 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCAWTxmU1fuFhAAxTplmhmbjmFqtFqwxx/l9w== 2024-05-09T05:05:34.037 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.d] 2024-05-09T05:05:34.037 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCAWTxmp52EHhAAokE3A5SXuaG8XNVgfzQ4FQ== 2024-05-09T05:05:34.038 INFO:teuthology.orchestra.run.smithi196.stdout:[mds.f] 2024-05-09T05:05:34.038 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCAWTxmrvQkJhAA3AeLeEzupQXLbKyDJhMqPg== 2024-05-09T05:05:34.038 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.4] 2024-05-09T05:05:34.038 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCRWTxmJHyaBhAA5pVavRzgd6mRjiczqsh5Ow== 2024-05-09T05:05:34.038 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.5] 2024-05-09T05:05:34.038 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCUWTxmMFNGBRAA9C0JOeDaIMmb5LOq8FmRIw== 2024-05-09T05:05:34.039 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.6] 2024-05-09T05:05:34.039 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCXWTxm8XMBBRAAi9/3D6yLVU1gWne2QCWjYA== 2024-05-09T05:05:34.039 INFO:teuthology.orchestra.run.smithi196.stdout:[osd.7] 2024-05-09T05:05:34.039 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCaWTxm85HTAxAAf0C4FW/tdyER5jyqv27+6g== 2024-05-09T05:05:34.039 INFO:teuthology.orchestra.run.smithi196.stdout:[client.0] 2024-05-09T05:05:34.039 INFO:teuthology.orchestra.run.smithi196.stdout: key = AQCAWTxmCF0GKxAAPsSAOGlNCN/z4xLqS4xzqQ== 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.y] 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQB/WTxm0QqPNRAANRkjBtkskW3rPiZ3UO65zw== 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.a] 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCAWTxm4szwARAANUZCmnl2EF7hCl6zKHAf4A== 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.c] 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCAWTxmmFu0CRAAQ5uMSxZcH+MTNDnj+PqzRw== 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.e] 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCAWTxmjOyDERAArh+1RtBJJkoGis2wWI9r1A== 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.0] 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCCWTxmrLeMAhAARZyvb9QuapGWrawda9MNUQ== 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.1] 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCFWTxm31PtARAAlGdQHAODO0PiIUIo+VKG0A== 2024-05-09T05:05:34.041 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.2] 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCIWTxmXfJbARAAJRU4b919LVh/Sjd3IzlxIw== 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.3] 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCLWTxmhiQfARAAb9lucVVOV3pAUTVxgie6GA== 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout:[mgr.x] 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQB/WTxmul+kORAAEuPH6Pe82FTCLTy8m3kvNQ== 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.b] 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCAWTxmU1fuFhAAxTplmhmbjmFqtFqwxx/l9w== 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.d] 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCAWTxmp52EHhAAokE3A5SXuaG8XNVgfzQ4FQ== 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout:[mds.f] 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCAWTxmrvQkJhAA3AeLeEzupQXLbKyDJhMqPg== 2024-05-09T05:05:34.042 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.4] 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCRWTxmJHyaBhAA5pVavRzgd6mRjiczqsh5Ow== 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.5] 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCUWTxmMFNGBRAA9C0JOeDaIMmb5LOq8FmRIw== 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.6] 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCXWTxm8XMBBRAAi9/3D6yLVU1gWne2QCWjYA== 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout:[osd.7] 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCaWTxm85HTAxAAf0C4FW/tdyER5jyqv27+6g== 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout:[client.0] 2024-05-09T05:05:34.043 INFO:teuthology.orchestra.run.smithi001.stdout: key = AQCAWTxmCF0GKxAAPsSAOGlNCN/z4xLqS4xzqQ== 2024-05-09T05:05:34.043 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T05:05:34.085 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T05:05:34.156 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.158 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.225 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.228 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.c --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.296 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.e --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.359 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.362 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.422 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.425 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.489 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.554 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.616 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T05:05:34.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-05-09T05:05:34.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.683 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.747 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.749 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.d --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.813 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.f --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2024-05-09T05:05:34.879 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.881 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.942 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:34.945 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:35.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:35.008 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:35.072 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:35.074 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-05-09T05:05:35.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T05:05:35.137 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-05-09T05:05:35.201 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-05-09T05:05:35.201 DEBUG:teuthology.orchestra.run.smithi001:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2024-05-09T05:05:35.227 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T05:05:35.336 DEBUG:teuthology.orchestra.run.smithi001:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2024-05-09T05:05:35.361 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2024-05-09T05:05:35.388 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T05:05:35.495 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2024-05-09T05:05:35.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2024-05-09T05:05:35.587 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2024-05-09T05:05:35.694 DEBUG:teuthology.orchestra.run.smithi196:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2024-05-09T05:05:35.720 DEBUG:teuthology.orchestra.run.smithi001:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:35.723 DEBUG:teuthology.orchestra.run.smithi196:> rm -- /home/ubuntu/cephtest/ceph.monmap 2024-05-09T05:05:35.776 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2024-05-09T05:05:35.777 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T05:05:35.777 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T05:05:35.782 INFO:tasks.ceph.mon.a:Started 2024-05-09T05:05:35.782 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T05:05:35.782 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T05:05:35.820 INFO:tasks.ceph.mon.b:Started 2024-05-09T05:05:35.820 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T05:05:35.820 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T05:05:35.823 INFO:tasks.ceph.mon.c:Started 2024-05-09T05:05:35.823 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2024-05-09T05:05:35.823 INFO:tasks.ceph.mgr.y:Restarting daemon 2024-05-09T05:05:35.824 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2024-05-09T05:05:35.829 INFO:tasks.ceph.mgr.y:Started 2024-05-09T05:05:35.829 INFO:tasks.ceph.mgr.x:Restarting daemon 2024-05-09T05:05:35.829 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2024-05-09T05:05:35.832 INFO:tasks.ceph.mgr.x:Started 2024-05-09T05:05:35.832 INFO:tasks.ceph:Setting crush tunables to default 2024-05-09T05:05:35.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph osd crush tunables default 2024-05-09T05:05:35.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.950+0000 7f3297ac8640 1 Processor -- start 2024-05-09T05:05:35.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.951+0000 7f3297ac8640 1 -- start start 2024-05-09T05:05:35.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3297ac8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:35.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3297ac8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:35.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3297ac8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32900c3be0 con 0x7f32900c4af0 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3297ac8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f32900c9290 con 0x7f32900c46e0 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3297ac8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f329006e890 con 0x7f32900c89a0 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295864640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295063640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295864640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59872/0 (socket says 172.21.15.1:59872) 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 learned_addr learned my addr 172.21.15.1:0/3883070838 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3296065640 1 --1- 172.21.15.1:0/3883070838 >> v1:172.21.15.196:6789/0 conn(0x7f32900c4af0 0x7f3290153490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:45274/0 (socket says 172.21.15.1:45274) 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:35.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:35.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:35.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:35.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:35.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:35.952+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:36.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:36.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:36.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:36.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:36.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:36.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:05:36.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:36.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:36.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:36.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.153+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:05:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:36.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:05:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:36.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:36.554+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:37.356+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:05:38.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f32837fe640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 msgr2=0x7f32900ca930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f32837fe640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32900c89a0 0x7f32900ca930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f32837fe640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 msgr2=0x7f3290159650 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f32837fe640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.196:3300/0 conn(0x7f32900c46e0 0x7f3290159650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f32837fe640 1 -- 172.21.15.1:0/3883070838 >> v1:172.21.15.196:6789/0 conn(0x7f32900c4af0 legacy=0x7f3290153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f32837fe640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f32837fe640 1 -- 172.21.15.1:0/3883070838 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32900c3e60 con 0x7f32740061c0 2024-05-09T05:05:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f32837fe640 1 -- 172.21.15.1:0/3883070838 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f32900c9290 con 0x7f32740036a0 2024-05-09T05:05:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.952+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.953+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.953+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.953+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:38.953+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.154+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.154+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.154+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.154+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:39.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.154+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:05:39.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.555+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:39.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.555+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:39.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.555+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:39.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.555+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:39.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:39.555+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:05:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:40.356+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:40.356+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:40.356+0000 7f3295864640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:40.356+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:40.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:40.356+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:05:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.108+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1497747795 0 0) 0x7f32900c3e60 con 0x7f32740061c0 2024-05-09T05:05:41.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.108+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f327c003520 con 0x7f32740061c0 2024-05-09T05:05:41.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3043440428 0 0) 0x7f327c003520 con 0x7f32740061c0 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 msgr2=0x7f3274005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 --2- 172.21.15.1:0/3883070838 >> v2:172.21.15.1:3300/0 conn(0x7f32740036a0 0x7f3274005b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327400af80 con 0x7f32740061c0 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f328c00d5c0 con 0x7f32740061c0 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 >> v1:172.21.15.1:6789/0 conn(0x7f32740061c0 legacy=0x7f32740086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3296065640 1 -- 172.21.15.1:0/3883070838 reap_dead start 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c003a50 0x7f327c003ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f327c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c0099b0 0x7f327c00bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f32900c9290 con 0x7f327c0099b0 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f327c00c3c0 con 0x7f327c003a50 2024-05-09T05:05:41.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f327c00c5c0 con 0x7f327c004800 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f327c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3295864640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c003a50 0x7f327c003ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3296065640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c0099b0 0x7f327c00bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 msgr2=0x7f327c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3295063640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 msgr2=0x7f327c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.110+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f327c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.111+0000 7f3295063640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f327c004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.111+0000 7f3296065640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 msgr2=0x7f327c004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.111+0000 7f3296065640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f327c004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.111+0000 7f3296065640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c003a50 msgr2=0x7f327c003ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.111+0000 7f3296065640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c003a50 0x7f327c003ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.111+0000 7f3296065640 1 -- 172.21.15.1:0/3883070838 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f327c003520 con 0x7f327c0099b0 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.112+0000 7f3296065640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c0099b0 0x7f327c00bdd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f32900c58d0 tx=0x7f327c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8dd2be31d66e514 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:41.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.112+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32880018f0 con 0x7f327c0099b0 2024-05-09T05:05:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.112+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f328802e070 con 0x7f327c0099b0 2024-05-09T05:05:41.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.112+0000 7f3294862640 1 -- 172.21.15.1:0/3883070838 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328802a900 con 0x7f327c0099b0 2024-05-09T05:05:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 -- 172.21.15.1:0/3883070838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c0099b0 msgr2=0x7f327c00bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c0099b0 0x7f327c00bdd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f32900c58d0 tx=0x7f327c005830 comp rx=0 tx=0).stop 2024-05-09T05:05:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 -- 172.21.15.1:0/3883070838 shutdown_connections 2024-05-09T05:05:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c0099b0 0x7f327c00bdd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f327c004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 --2- 172.21.15.1:0/3883070838 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c003a50 0x7f327c003ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 -- 172.21.15.1:0/3883070838 >> 172.21.15.1:0/3883070838 conn(0x7f32900747b0 msgr2=0x7f32900ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 -- 172.21.15.1:0/3883070838 shutdown_connections 2024-05-09T05:05:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.113+0000 7f3297ac8640 1 -- 172.21.15.1:0/3883070838 wait complete. 2024-05-09T05:05:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.115+0000 7f3297ac8640 1 Processor -- start 2024-05-09T05:05:41.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.116+0000 7f3297ac8640 1 -- start start 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3297ac8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c003a50 0x7f32901ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3297ac8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f3290148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3297ac8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c0099b0 0x7f329014b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3297ac8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f32900c97c0 con 0x7f327c003a50 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3297ac8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f32900c94e0 con 0x7f327c0099b0 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3297ac8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f32900ca930 con 0x7f327c004800 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3295864640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c003a50 0x7f32901ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3295063640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f3290148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3295864640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c003a50 0x7f32901ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50852/0 (socket says 172.21.15.1:50852) 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3295864640 1 -- 172.21.15.1:0/1354431660 learned_addr learned my addr 172.21.15.1:0/1354431660 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3296065640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c0099b0 0x7f329014b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3295063640 1 -- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 msgr2=0x7f3290148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3295063640 1 -- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 msgr2=0x7f3290148cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3295063640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f3290148cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.117+0000 7f3295063640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f3290148cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f3295864640 1 -- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 msgr2=0x7f3290148cc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f3295864640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f3290148cc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f3295864640 1 -- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c0099b0 msgr2=0x7f329014b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f3295864640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c0099b0 0x7f329014b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f3295864640 1 -- 172.21.15.1:0/1354431660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32901eeb30 con 0x7f327c003a50 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f3295864640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c003a50 0x7f32901ee5f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f32840119e0 tx=0x7f3284011ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f32827fc640 1 -- 172.21.15.1:0/1354431660 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3284021020 con 0x7f327c003a50 2024-05-09T05:05:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f32827fc640 1 -- 172.21.15.1:0/1354431660 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3284010d20 con 0x7f327c003a50 2024-05-09T05:05:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f329014a120 con 0x7f327c003a50 2024-05-09T05:05:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f32827fc640 1 -- 172.21.15.1:0/1354431660 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f328401d650 con 0x7f327c003a50 2024-05-09T05:05:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.118+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32901eddf0 con 0x7f327c003a50 2024-05-09T05:05:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.120+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32901edbe0 con 0x7f327c003a50 2024-05-09T05:05:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.120+0000 7f32827fc640 1 -- 172.21.15.1:0/1354431660 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3284024020 con 0x7f327c003a50 2024-05-09T05:05:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.120+0000 7f32827fc640 1 -- 172.21.15.1:0/1354431660 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f32840061b0 con 0x7f327c003a50 2024-05-09T05:05:41.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.122+0000 7f32827fc640 1 -- 172.21.15.1:0/1354431660 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f328402b030 con 0x7f327c003a50 2024-05-09T05:05:41.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.192+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f3290050870 con 0x7f327c003a50 2024-05-09T05:05:41.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.193+0000 7f32827fc640 1 -- 172.21.15.1:0/1354431660 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) ==== 124+0+0 (secure 0 0 0) 0x7f328402b030 con 0x7f327c003a50 2024-05-09T05:05:41.194 INFO:teuthology.orchestra.run.smithi001.stderr:adjusted tunables profile to default 2024-05-09T05:05:41.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c003a50 msgr2=0x7f32901ee5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c003a50 0x7f32901ee5f0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f32840119e0 tx=0x7f3284011ea0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 shutdown_connections 2024-05-09T05:05:41.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f327c0099b0 0x7f329014b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f327c004800 0x7f3290148cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 --2- 172.21.15.1:0/1354431660 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f327c003a50 0x7f32901ee5f0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 >> 172.21.15.1:0/1354431660 conn(0x7f32900747b0 msgr2=0x7f3290071f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:41.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 shutdown_connections 2024-05-09T05:05:41.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:41.195+0000 7f3297ac8640 1 -- 172.21.15.1:0/1354431660 wait complete. 2024-05-09T05:05:41.207 INFO:tasks.ceph:check_enable_crimson: False 2024-05-09T05:05:41.207 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2024-05-09T05:05:41.208 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:41.208 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2024-05-09T05:05:41.229 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:41.228+0000 7fb7c513a040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T05:05:41.235 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:41.235 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2024-05-09T05:05:41.242 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:41.242+0000 7f75b19ee040 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T05:05:41.268 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:41.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2024-05-09T05:05:41.335 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:05:41.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo dd if=/var/lib/ceph/osd/ceph-3/fsid of=/dev/stdout 2024-05-09T05:05:41.393 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:41.392+0000 7fb7c513a040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T05:05:41.401 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:41.401 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-4/fsid of=/dev/stdout 2024-05-09T05:05:41.410 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:41.410+0000 7f75b19ee040 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T05:05:41.426 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:41.427 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-5/fsid of=/dev/stdout 2024-05-09T05:05:41.449 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:41.448+0000 7fb7c513a040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T05:05:41.466 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:41.466+0000 7f75b19ee040 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T05:05:41.474 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:41.474 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-6/fsid of=/dev/stdout 2024-05-09T05:05:41.539 DEBUG:teuthology.orchestra.run.smithi196:> set -ex 2024-05-09T05:05:41.539 DEBUG:teuthology.orchestra.run.smithi196:> sudo dd if=/var/lib/ceph/osd/ceph-7/fsid of=/dev/stdout 2024-05-09T05:05:41.604 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new cfa581c7-a28f-41ce-98c2-bf8e56f9af33 0 2024-05-09T05:05:41.712 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:41.711+0000 7fb7c513a040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T05:05:41.721 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:41.721+0000 7f75b19ee040 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T05:05:41.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.754+0000 7ff089186640 1 Processor -- start 2024-05-09T05:05:41.755 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.754+0000 7ff089186640 1 -- start start 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff089186640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 0x7ff084154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff089186640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff084070df0 0x7ff08406c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff089186640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff0840704b0 con 0x7ff0840c9ff0 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff089186640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff084070790 con 0x7ff084070df0 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff089186640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff0840c7a30 con 0x7ff0840c6100 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff082d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 0x7ff084154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff082d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 0x7ff084154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:41062/0 (socket says 172.21.15.196:41062) 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff082d9d640 1 -- 172.21.15.196:0/147883269 learned_addr learned my addr 172.21.15.196:0/147883269 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff08359e640 1 --1- 172.21.15.196:0/147883269 >> v1:172.21.15.196:6789/0 conn(0x7ff0840c9ff0 0x7ff0840c9520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:56086/0 (socket says 172.21.15.196:56086) 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff08259c640 1 --2- 172.21.15.196:0/147883269 >> v2:172.21.15.1:3300/0 conn(0x7ff084070df0 0x7ff08406c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.758 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff082d9d640 1 -- 172.21.15.196:0/147883269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 msgr2=0x7ff084154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff082d9d640 1 -- 172.21.15.196:0/147883269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 msgr2=0x7ff084154bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff082d9d640 1 --2- 172.21.15.196:0/147883269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 0x7ff084154bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff082d9d640 1 --2- 172.21.15.196:0/147883269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 0x7ff084154bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff081d9b640 1 -- 172.21.15.196:0/147883269 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1869257032 0 0) 0x7ff0840704b0 con 0x7ff0840c9ff0 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff081d9b640 1 -- 172.21.15.196:0/147883269 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff06c003520 con 0x7ff0840c9ff0 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff08259c640 1 -- 172.21.15.196:0/147883269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 msgr2=0x7ff084154bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff08259c640 1 --2- 172.21.15.196:0/147883269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 0x7ff084154bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff08259c640 1 -- 172.21.15.196:0/147883269 >> v1:172.21.15.196:6789/0 conn(0x7ff0840c9ff0 legacy=0x7ff0840c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.756+0000 7ff08259c640 1 -- 172.21.15.196:0/147883269 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0840c7d10 con 0x7ff084070df0 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.757+0000 7ff08259c640 1 --2- 172.21.15.196:0/147883269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08406c7f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff0780097a0 tx=0x7ff07802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=749aab96da859e80 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.757+0000 7ff081d9b640 1 -- 172.21.15.196:0/147883269 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff078037070 con 0x7ff084070df0 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.757+0000 7ff081d9b640 1 -- 172.21.15.196:0/147883269 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff078032440 con 0x7ff084070df0 2024-05-09T05:05:41.759 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.757+0000 7ff081d9b640 1 -- 172.21.15.196:0/147883269 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0780327f0 con 0x7ff084070df0 2024-05-09T05:05:41.760 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.758+0000 7ff089186640 1 -- 172.21.15.196:0/147883269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 msgr2=0x7ff08406c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.760 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.758+0000 7ff089186640 1 --2- 172.21.15.196:0/147883269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08406c7f0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff0780097a0 tx=0x7ff07802a180 comp rx=0 tx=0).stop 2024-05-09T05:05:41.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.758+0000 7ff089186640 1 -- 172.21.15.196:0/147883269 shutdown_connections 2024-05-09T05:05:41.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.758+0000 7ff089186640 1 --2- 172.21.15.196:0/147883269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08406c7f0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.758+0000 7ff089186640 1 --2- 172.21.15.196:0/147883269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c6100 0x7ff084154bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.758+0000 7ff089186640 1 -- 172.21.15.196:0/147883269 >> 172.21.15.196:0/147883269 conn(0x7ff08406d800 msgr2=0x7ff08406dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:41.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.758+0000 7ff089186640 1 -- 172.21.15.196:0/147883269 shutdown_connections 2024-05-09T05:05:41.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.758+0000 7ff089186640 1 -- 172.21.15.196:0/147883269 wait complete. 2024-05-09T05:05:41.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.760+0000 7ff089186640 1 Processor -- start 2024-05-09T05:05:41.761 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.760+0000 7ff089186640 1 -- start start 2024-05-09T05:05:41.761 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:41.760+0000 7fb7c513a040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.761+0000 7ff089186640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08415a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.761+0000 7ff089186640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0840c6100 0x7ff08414dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.761+0000 7ff089186640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 0x7ff084152cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.761+0000 7ff089186640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff0840c9db0 con 0x7ff084070df0 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.761+0000 7ff089186640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff0840c9c30 con 0x7ff0840c6100 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.761+0000 7ff089186640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff084070070 con 0x7ff0840c9ff0 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff08259c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0840c6100 0x7ff08414dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08415a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff08359e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 0x7ff084152cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.764 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08415a230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.196:41654/0 (socket says 172.21.15.196:41654) 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff08259c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0840c6100 0x7ff08414dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:41990/0 (socket says 172.21.15.196:41990) 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 -- 172.21.15.196:0/1307364704 learned_addr learned my addr 172.21.15.196:0/1307364704 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff08359e640 1 -- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 msgr2=0x7ff084152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff08359e640 1 -- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 msgr2=0x7ff084152cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff08359e640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 0x7ff084152cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff08359e640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 0x7ff084152cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 -- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 msgr2=0x7ff084152cb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 0x7ff084152cb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 -- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0840c6100 msgr2=0x7ff08414dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0840c6100 0x7ff08414dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 -- 172.21.15.196:0/1307364704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff08415a770 con 0x7ff084070df0 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff08259c640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0840c6100 0x7ff08414dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff082d9d640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08415a230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff07400e840 tx=0x7ff07400ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:41.765 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff06b7fe640 1 -- 172.21.15.196:0/1307364704 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff074017070 con 0x7ff084070df0 2024-05-09T05:05:41.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff06b7fe640 1 -- 172.21.15.196:0/1307364704 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff074004020 con 0x7ff084070df0 2024-05-09T05:05:41.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0841f36b0 con 0x7ff084070df0 2024-05-09T05:05:41.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff06b7fe640 1 -- 172.21.15.196:0/1307364704 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff07400f040 con 0x7ff084070df0 2024-05-09T05:05:41.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.762+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff084159730 con 0x7ff084070df0 2024-05-09T05:05:41.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.763+0000 7ff06b7fe640 1 -- 172.21.15.196:0/1307364704 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7ff07400c6d0 con 0x7ff084070df0 2024-05-09T05:05:41.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.763+0000 7ff06b7fe640 1 -- 172.21.15.196:0/1307364704 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 725+0+0 (secure 0 0 0) 0x7ff074003b30 con 0x7ff084070df0 2024-05-09T05:05:41.766 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.763+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0840696d0 con 0x7ff084070df0 2024-05-09T05:05:41.767 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.766+0000 7ff06b7fe640 1 -- 172.21.15.196:0/1307364704 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7ff074003610 con 0x7ff084070df0 2024-05-09T05:05:41.770 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:41.769+0000 7f75b19ee040 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T05:05:41.837 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.836+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd new", "uuid": "cfa581c7-a28f-41ce-98c2-bf8e56f9af33", "id": 0} v 0) -- 0x7ff084050870 con 0x7ff084070df0 2024-05-09T05:05:41.838 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.838+0000 7ff06b7fe640 1 -- 172.21.15.196:0/1307364704 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "cfa581c7-a28f-41ce-98c2-bf8e56f9af33", "id": 0}]=0 v3) ==== 112+0+2 (secure 0 0 0) 0x7ff074003d30 con 0x7ff084070df0 2024-05-09T05:05:41.838 INFO:teuthology.orchestra.run.smithi196.stdout:0 2024-05-09T05:05:41.839 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 msgr2=0x7ff08415a230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08415a230 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7ff07400e840 tx=0x7ff07400ed00 comp rx=0 tx=0).stop 2024-05-09T05:05:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 shutdown_connections 2024-05-09T05:05:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0840c6100 0x7ff08414dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0840c9ff0 0x7ff084152cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 --2- 172.21.15.196:0/1307364704 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff084070df0 0x7ff08415a230 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 >> 172.21.15.196:0/1307364704 conn(0x7ff08406d800 msgr2=0x7ff084072010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 shutdown_connections 2024-05-09T05:05:41.840 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.839+0000 7ff089186640 1 -- 172.21.15.196:0/1307364704 wait complete. 2024-05-09T05:05:41.850 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 040c2166-4e3a-4f17-a385-a50c34d987ec 1 2024-05-09T05:05:41.961 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.961+0000 7f8a8b019640 1 Processor -- start 2024-05-09T05:05:41.961 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.961+0000 7f8a8b019640 1 -- start start 2024-05-09T05:05:41.964 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a8b019640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 0x7f8a84156b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.964 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a8b019640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a840c9520 0x7f8a840717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.964 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a8b019640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a840cb450 con 0x7f8a84071d30 2024-05-09T05:05:41.964 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a8b019640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8a840c9e50 con 0x7f8a840c9520 2024-05-09T05:05:41.964 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a8b019640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8a840caed0 con 0x7f8a840c8b70 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a88db5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 0x7f8a84156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a88db5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 0x7f8a84156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:41082/0 (socket says 172.21.15.196:41082) 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a88db5640 1 -- 172.21.15.196:0/3538981279 learned_addr learned my addr 172.21.15.196:0/3538981279 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a83fff640 1 --2- 172.21.15.196:0/3538981279 >> v2:172.21.15.1:3300/0 conn(0x7f8a840c9520 0x7f8a840717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a88db5640 1 -- 172.21.15.196:0/3538981279 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 msgr2=0x7f8a84156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a88db5640 1 -- 172.21.15.196:0/3538981279 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 msgr2=0x7f8a84156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a88db5640 1 --2- 172.21.15.196:0/3538981279 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 0x7f8a84156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.962+0000 7f8a88db5640 1 --2- 172.21.15.196:0/3538981279 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 0x7f8a84156b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.963+0000 7f8a837fe640 1 -- 172.21.15.196:0/3538981279 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2845041517 0 0) 0x7f8a840cb450 con 0x7f8a84071d30 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.963+0000 7f8a837fe640 1 -- 172.21.15.196:0/3538981279 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a74003520 con 0x7f8a84071d30 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.963+0000 7f8a83fff640 1 -- 172.21.15.196:0/3538981279 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 msgr2=0x7f8a84156b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:41.965 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.963+0000 7f8a83fff640 1 --2- 172.21.15.196:0/3538981279 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 0x7f8a84156b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.963+0000 7f8a83fff640 1 -- 172.21.15.196:0/3538981279 >> v1:172.21.15.1:6789/0 conn(0x7f8a84071d30 legacy=0x7f8a84159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.963+0000 7f8a83fff640 1 -- 172.21.15.196:0/3538981279 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a840cb170 con 0x7f8a840c9520 2024-05-09T05:05:41.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a83fff640 1 --2- 172.21.15.196:0/3538981279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 0x7f8a840717f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8a700097a0 tx=0x7f8a7002a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd88003e6172dd2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:41.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a837fe640 1 -- 172.21.15.196:0/3538981279 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70037070 con 0x7f8a840c9520 2024-05-09T05:05:41.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a837fe640 1 -- 172.21.15.196:0/3538981279 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a70032440 con 0x7f8a840c9520 2024-05-09T05:05:41.966 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a837fe640 1 -- 172.21.15.196:0/3538981279 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70032810 con 0x7f8a840c9520 2024-05-09T05:05:41.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a8b019640 1 -- 172.21.15.196:0/3538981279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 msgr2=0x7f8a840717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a8b019640 1 --2- 172.21.15.196:0/3538981279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 0x7f8a840717f0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8a700097a0 tx=0x7f8a7002a180 comp rx=0 tx=0).stop 2024-05-09T05:05:41.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a8b019640 1 -- 172.21.15.196:0/3538981279 shutdown_connections 2024-05-09T05:05:41.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a8b019640 1 --2- 172.21.15.196:0/3538981279 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 0x7f8a840717f0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.964+0000 7f8a8b019640 1 --2- 172.21.15.196:0/3538981279 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a840c8b70 0x7f8a84156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.967 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.965+0000 7f8a8b019640 1 -- 172.21.15.196:0/3538981279 >> 172.21.15.196:0/3538981279 conn(0x7f8a8406d800 msgr2=0x7f8a8406dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:41.968 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.965+0000 7f8a8b019640 1 -- 172.21.15.196:0/3538981279 shutdown_connections 2024-05-09T05:05:41.968 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.965+0000 7f8a8b019640 1 -- 172.21.15.196:0/3538981279 wait complete. 2024-05-09T05:05:41.968 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.966+0000 7f8a8b019640 1 Processor -- start 2024-05-09T05:05:41.968 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.966+0000 7f8a8b019640 1 -- start start 2024-05-09T05:05:41.970 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a8b019640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 0x7f8a841ee3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.970 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a8b019640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a840c8b70 0x7f8a841472a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.970 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a8b019640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 0x7f8a841e51b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:41.970 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a8b019640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8a84071210 con 0x7f8a840c9520 2024-05-09T05:05:41.970 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a8b019640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8a840c3f30 con 0x7f8a840c8b70 2024-05-09T05:05:41.970 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a8b019640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8a841570d0 con 0x7f8a84071d30 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a88db5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 0x7f8a841ee3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a88db5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 0x7f8a841ee3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:41086/0 (socket says 172.21.15.196:41086) 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a88db5640 1 -- 172.21.15.196:0/4257410529 learned_addr learned my addr 172.21.15.196:0/4257410529 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a83fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a840c8b70 0x7f8a841472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a895b6640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 0x7f8a841e51b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a88db5640 1 -- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 msgr2=0x7f8a841ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a88db5640 1 -- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 msgr2=0x7f8a841ee3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a88db5640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 0x7f8a841ee3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:41.971 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a88db5640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 0x7f8a841ee3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a83fff640 1 -- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 msgr2=0x7f8a841ee3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a83fff640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 0x7f8a841ee3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a83fff640 1 -- 172.21.15.196:0/4257410529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 msgr2=0x7f8a841e51b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a83fff640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 0x7f8a841e51b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a83fff640 1 -- 172.21.15.196:0/4257410529 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a840c6f80 con 0x7f8a840c8b70 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a83fff640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a840c8b70 0x7f8a841472a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8a70031000 tx=0x7f8a700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a817fa640 1 -- 172.21.15.196:0/4257410529 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70041030 con 0x7f8a840c8b70 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a817fa640 1 -- 172.21.15.196:0/4257410529 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a70030500 con 0x7f8a840c8b70 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a841f27e0 con 0x7f8a840c8b70 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a817fa640 1 -- 172.21.15.196:0/4257410529 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a70037040 con 0x7f8a840c8b70 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.968+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a841f1a20 con 0x7f8a840c8b70 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.969+0000 7f8a817fa640 1 -- 172.21.15.196:0/4257410529 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f8a70044070 con 0x7f8a840c8b70 2024-05-09T05:05:41.972 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.969+0000 7f8a817fa640 1 -- 172.21.15.196:0/4257410529 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 840+0+0 (secure 0 0 0) 0x7f8a7003d780 con 0x7f8a840c8b70 2024-05-09T05:05:41.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.969+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a840696d0 con 0x7f8a840c8b70 2024-05-09T05:05:41.973 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:41.972+0000 7f8a817fa640 1 -- 172.21.15.196:0/4257410529 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f8a7003d9d0 con 0x7f8a840c8b70 2024-05-09T05:05:42.044 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.043+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "040c2166-4e3a-4f17-a385-a50c34d987ec", "id": 1} v 0) -- 0x7f8a84050870 con 0x7f8a840c8b70 2024-05-09T05:05:42.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.046+0000 7f8a817fa640 1 -- 172.21.15.196:0/4257410529 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "040c2166-4e3a-4f17-a385-a50c34d987ec", "id": 1}]=0 v4) ==== 112+0+2 (secure 0 0 0) 0x7f8a7003dbb0 con 0x7f8a840c8b70 2024-05-09T05:05:42.047 INFO:teuthology.orchestra.run.smithi196.stdout:1 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a840c8b70 msgr2=0x7f8a841472a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a840c8b70 0x7f8a841472a0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f8a70031000 tx=0x7f8a700046c0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 shutdown_connections 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a840c8b70 0x7f8a841472a0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a840c9520 0x7f8a841e51b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 --2- 172.21.15.196:0/4257410529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a84071d30 0x7f8a841ee3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 >> 172.21.15.196:0/4257410529 conn(0x7f8a8406d800 msgr2=0x7f8a841510c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 shutdown_connections 2024-05-09T05:05:42.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.047+0000 7f8a8b019640 1 -- 172.21.15.196:0/4257410529 wait complete. 2024-05-09T05:05:42.059 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 4977c108-0fff-47ec-802e-67b4e9464153 2 2024-05-09T05:05:42.172 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.172+0000 7fa53a1a8640 1 Processor -- start 2024-05-09T05:05:42.173 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.172+0000 7fa53a1a8640 1 -- start start 2024-05-09T05:05:42.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa53a1a8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa5340c8b70 0x7fa534156b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa53a1a8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa534159700 0x7fa534159b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa53a1a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5340cb450 con 0x7fa5340c9520 2024-05-09T05:05:42.175 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa53a1a8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa5340c9e50 con 0x7fa534159700 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa53a1a8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fa5340caed0 con 0x7fa5340c8b70 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa5337fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa5340c8b70 0x7fa534156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa5337fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa5340c8b70 0x7fa534156b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42006/0 (socket says 172.21.15.196:42006) 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa532ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fa5340c9520 0x7fa5340717f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:56096/0 (socket says 172.21.15.196:56096) 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa5337fe640 1 -- 172.21.15.196:0/3534115785 learned_addr learned my addr 172.21.15.196:0/3534115785 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.173+0000 7fa533fff640 1 --2- 172.21.15.196:0/3534115785 >> v2:172.21.15.1:3300/0 conn(0x7fa534159700 0x7fa534159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.174+0000 7fa5337fe640 1 -- 172.21.15.196:0/3534115785 >> v2:172.21.15.1:3300/0 conn(0x7fa534159700 msgr2=0x7fa534159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.174+0000 7fa5337fe640 1 --2- 172.21.15.196:0/3534115785 >> v2:172.21.15.1:3300/0 conn(0x7fa534159700 0x7fa534159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.174+0000 7fa5337fe640 1 -- 172.21.15.196:0/3534115785 >> v1:172.21.15.196:6789/0 conn(0x7fa5340c9520 legacy=0x7fa5340717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.174+0000 7fa5337fe640 1 -- 172.21.15.196:0/3534115785 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5340cb170 con 0x7fa5340c8b70 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.174+0000 7fa5337fe640 1 --2- 172.21.15.196:0/3534115785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c8b70 0x7fa534156b90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa534073740 tx=0x7fa52402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9fcdaff2f280a4b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:42.176 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.174+0000 7fa5327fc640 1 -- 172.21.15.196:0/3534115785 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524037070 con 0x7fa5340c8b70 2024-05-09T05:05:42.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa5327fc640 1 -- 172.21.15.196:0/3534115785 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa524032440 con 0x7fa5340c8b70 2024-05-09T05:05:42.177 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa5327fc640 1 -- 172.21.15.196:0/3534115785 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524032810 con 0x7fa5340c8b70 2024-05-09T05:05:42.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa53a1a8640 1 -- 172.21.15.196:0/3534115785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c8b70 msgr2=0x7fa534156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa53a1a8640 1 --2- 172.21.15.196:0/3534115785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c8b70 0x7fa534156b90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa534073740 tx=0x7fa52402a270 comp rx=0 tx=0).stop 2024-05-09T05:05:42.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa53a1a8640 1 -- 172.21.15.196:0/3534115785 shutdown_connections 2024-05-09T05:05:42.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa53a1a8640 1 --2- 172.21.15.196:0/3534115785 >> v2:172.21.15.1:3300/0 conn(0x7fa534159700 0x7fa534159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa53a1a8640 1 --2- 172.21.15.196:0/3534115785 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c8b70 0x7fa534156b90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.178 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa53a1a8640 1 -- 172.21.15.196:0/3534115785 >> 172.21.15.196:0/3534115785 conn(0x7fa53406d800 msgr2=0x7fa53406dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa53a1a8640 1 -- 172.21.15.196:0/3534115785 shutdown_connections 2024-05-09T05:05:42.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.175+0000 7fa53a1a8640 1 -- 172.21.15.196:0/3534115785 wait complete. 2024-05-09T05:05:42.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.177+0000 7fa53a1a8640 1 Processor -- start 2024-05-09T05:05:42.179 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.177+0000 7fa53a1a8640 1 -- start start 2024-05-09T05:05:42.183 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa53a1a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5340c8b70 0x7fa5340c6de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.183 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa53a1a8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c9520 0x7fa5341587a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.183 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa53a1a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 0x7fa5340cc920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.183 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa53a1a8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa53415a050 con 0x7fa5340c8b70 2024-05-09T05:05:42.183 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa53a1a8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa5340c3f30 con 0x7fa5340c9520 2024-05-09T05:05:42.183 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa53a1a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa534071210 con 0x7fa534159700 2024-05-09T05:05:42.183 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa532ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c9520 0x7fa5341587a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa532ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c9520 0x7fa5341587a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42014/0 (socket says 172.21.15.196:42014) 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa532ffd640 1 -- 172.21.15.196:0/809722655 learned_addr learned my addr 172.21.15.196:0/809722655 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa533fff640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 0x7fa5340cc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa5337fe640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5340c8b70 0x7fa5340c6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa533fff640 1 -- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 msgr2=0x7fa5340cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa533fff640 1 -- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 msgr2=0x7fa5340cc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa533fff640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 0x7fa5340cc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa533fff640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 0x7fa5340cc920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa5337fe640 1 -- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 msgr2=0x7fa5340cc920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa5337fe640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 0x7fa5340cc920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa5337fe640 1 -- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c9520 msgr2=0x7fa5341587a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa5337fe640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c9520 0x7fa5341587a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa5337fe640 1 -- 172.21.15.196:0/809722655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5340c7320 con 0x7fa5340c8b70 2024-05-09T05:05:42.184 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa532ffd640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c9520 0x7fa5341587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa5337fe640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5340c8b70 0x7fa5340c6de0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa5340cb050 tx=0x7fa5240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa52bfff640 1 -- 172.21.15.196:0/809722655 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524041030 con 0x7fa5340c8b70 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa52bfff640 1 -- 172.21.15.196:0/809722655 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa524030440 con 0x7fa5340c8b70 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.179+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa534156500 con 0x7fa5340c8b70 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.180+0000 7fa52bfff640 1 -- 172.21.15.196:0/809722655 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa524037040 con 0x7fa5340c8b70 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.180+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5340c5f00 con 0x7fa5340c8b70 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.180+0000 7fa52bfff640 1 -- 172.21.15.196:0/809722655 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa524044070 con 0x7fa5340c8b70 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.180+0000 7fa52bfff640 1 -- 172.21.15.196:0/809722655 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 955+0+0 (secure 0 0 0) 0x7fa52403da50 con 0x7fa5340c8b70 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.180+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5340696d0 con 0x7fa5340c8b70 2024-05-09T05:05:42.185 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.183+0000 7fa52bfff640 1 -- 172.21.15.196:0/809722655 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fa52403dca0 con 0x7fa5340c8b70 2024-05-09T05:05:42.254 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.254+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4977c108-0fff-47ec-802e-67b4e9464153", "id": 2} v 0) -- 0x7fa534050870 con 0x7fa5340c8b70 2024-05-09T05:05:42.256 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.256+0000 7fa52bfff640 1 -- 172.21.15.196:0/809722655 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4977c108-0fff-47ec-802e-67b4e9464153", "id": 2}]=0 v5) ==== 112+0+2 (secure 0 0 0) 0x7fa52403d630 con 0x7fa5340c8b70 2024-05-09T05:05:42.256 INFO:teuthology.orchestra.run.smithi196.stdout:2 2024-05-09T05:05:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.256+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5340c8b70 msgr2=0x7fa5340c6de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.256+0000 7fa53a1a8640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5340c8b70 0x7fa5340c6de0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fa5340cb050 tx=0x7fa5240046c0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.257+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 shutdown_connections 2024-05-09T05:05:42.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.257+0000 7fa53a1a8640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5340c9520 0x7fa5341587a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.257+0000 7fa53a1a8640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa534159700 0x7fa5340cc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.257+0000 7fa53a1a8640 1 --2- 172.21.15.196:0/809722655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5340c8b70 0x7fa5340c6de0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.257+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 >> 172.21.15.196:0/809722655 conn(0x7fa53406d800 msgr2=0x7fa534071780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.257+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 shutdown_connections 2024-05-09T05:05:42.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.257+0000 7fa53a1a8640 1 -- 172.21.15.196:0/809722655 wait complete. 2024-05-09T05:05:42.268 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 4f41003a-2ede-4fa9-a0d3-66c790841802 3 2024-05-09T05:05:42.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.379+0000 7f1aad36c640 1 Processor -- start 2024-05-09T05:05:42.379 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.379+0000 7f1aad36c640 1 -- start start 2024-05-09T05:05:42.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aad36c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa815ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aad36c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1aa815cf60 0x7f1aa81529d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aad36c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1aa8073010 con 0x7f1aa80caa00 2024-05-09T05:05:42.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aad36c640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f1aa80c7b40 con 0x7f1aa815cf60 2024-05-09T05:05:42.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aad36c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa80732d0 con 0x7f1aa80749f0 2024-05-09T05:05:42.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa67fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa815ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa77fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1aa815cf60 0x7f1aa81529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.382 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa67fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa815ca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:41104/0 (socket says 172.21.15.196:41104) 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa77fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1aa815cf60 0x7f1aa81529d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42026/0 (socket says 172.21.15.196:42026) 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa6ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1aa80caa00 0x7f1aa80c7430 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:56110/0 (socket says 172.21.15.196:56110) 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa67fc640 1 -- 172.21.15.196:0/2586045330 learned_addr learned my addr 172.21.15.196:0/2586045330 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa67fc640 1 -- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 msgr2=0x7f1aa815ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa67fc640 1 -- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 msgr2=0x7f1aa815ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa67fc640 1 --2- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa815ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.380+0000 7f1aa67fc640 1 --2- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa815ca20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.381+0000 7f1aa77fe640 1 -- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 msgr2=0x7f1aa815ca20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.381+0000 7f1aa77fe640 1 --2- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa815ca20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.381+0000 7f1aa77fe640 1 -- 172.21.15.196:0/2586045330 >> v1:172.21.15.196:6789/0 conn(0x7f1aa80caa00 legacy=0x7f1aa80c7430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.381+0000 7f1aa77fe640 1 -- 172.21.15.196:0/2586045330 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa8073680 con 0x7f1aa815cf60 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.381+0000 7f1aa77fe640 1 --2- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa815cf60 0x7f1aa81529d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0009d90 tx=0x7f1aa002a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2d13027ac1c41871 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.381+0000 7f1aa5ffb640 1 -- 172.21.15.196:0/2586045330 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0031600 con 0x7f1aa815cf60 2024-05-09T05:05:42.383 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.381+0000 7f1aa5ffb640 1 -- 172.21.15.196:0/2586045330 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa00317a0 con 0x7f1aa815cf60 2024-05-09T05:05:42.384 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.381+0000 7f1aa5ffb640 1 -- 172.21.15.196:0/2586045330 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa0031b70 con 0x7f1aa815cf60 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.382+0000 7f1aad36c640 1 -- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa815cf60 msgr2=0x7f1aa81529d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.382+0000 7f1aad36c640 1 --2- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa815cf60 0x7f1aa81529d0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1aa0009d90 tx=0x7f1aa002a5d0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.382+0000 7f1aad36c640 1 -- 172.21.15.196:0/2586045330 shutdown_connections 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.382+0000 7f1aad36c640 1 --2- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa815cf60 0x7f1aa81529d0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.382+0000 7f1aad36c640 1 --2- 172.21.15.196:0/2586045330 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa815ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.382+0000 7f1aad36c640 1 -- 172.21.15.196:0/2586045330 >> 172.21.15.196:0/2586045330 conn(0x7f1aa8070f40 msgr2=0x7f1aa806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.382+0000 7f1aad36c640 1 -- 172.21.15.196:0/2586045330 shutdown_connections 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.382+0000 7f1aad36c640 1 -- 172.21.15.196:0/2586045330 wait complete. 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.384+0000 7f1aad36c640 1 Processor -- start 2024-05-09T05:05:42.385 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.384+0000 7f1aad36c640 1 -- start start 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.385+0000 7f1aad36c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa81ee360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.385+0000 7f1aad36c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa80caa00 0x7f1aa815b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.385+0000 7f1aad36c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa815cf60 0x7f1aa8144280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.385+0000 7f1aad36c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa8073c40 con 0x7f1aa815cf60 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.385+0000 7f1aad36c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1aa815c590 con 0x7f1aa80caa00 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.385+0000 7f1aad36c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1aa815c220 con 0x7f1aa80749f0 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa80caa00 0x7f1aa815b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa6ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa81ee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa80caa00 0x7f1aa815b230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42032/0 (socket says 172.21.15.196:42032) 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 -- 172.21.15.196:0/1065954570 learned_addr learned my addr 172.21.15.196:0/1065954570 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa77fe640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa815cf60 0x7f1aa8144280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.388 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa6ffd640 1 -- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 msgr2=0x7f1aa81ee360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa6ffd640 1 -- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 msgr2=0x7f1aa81ee360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa6ffd640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa81ee360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa6ffd640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa81ee360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 -- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 msgr2=0x7f1aa81ee360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa81ee360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 -- 172.21.15.196:0/1065954570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa815cf60 msgr2=0x7f1aa8144280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa815cf60 0x7f1aa8144280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 -- 172.21.15.196:0/1065954570 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1aa81ee8a0 con 0x7f1aa80caa00 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aa67fc640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa80caa00 0x7f1aa815b230 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00da70 tx=0x7f1a9c00df30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1a8f7fe640 1 -- 172.21.15.196:0/1065954570 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c00bb40 con 0x7f1aa80caa00 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1a8f7fe640 1 -- 172.21.15.196:0/1065954570 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a9c00bd00 con 0x7f1aa80caa00 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1aa8150250 con 0x7f1aa80caa00 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1a8f7fe640 1 -- 172.21.15.196:0/1065954570 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a9c016630 con 0x7f1aa80caa00 2024-05-09T05:05:42.389 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1aa8150810 con 0x7f1aa80caa00 2024-05-09T05:05:42.390 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1a8f7fe640 1 -- 172.21.15.196:0/1065954570 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f1a9c0167d0 con 0x7f1aa80caa00 2024-05-09T05:05:42.390 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.386+0000 7f1a8f7fe640 1 -- 172.21.15.196:0/1065954570 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(5..5 src has 1..5) ==== 1070+0+0 (secure 0 0 0) 0x7f1a9c015390 con 0x7f1aa80caa00 2024-05-09T05:05:42.390 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.387+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1aa8050870 con 0x7f1aa80caa00 2024-05-09T05:05:42.390 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.389+0000 7f1a8f7fe640 1 -- 172.21.15.196:0/1065954570 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f1a9c016c00 con 0x7f1aa80caa00 2024-05-09T05:05:42.462 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.461+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "4f41003a-2ede-4fa9-a0d3-66c790841802", "id": 3} v 0) -- 0x7f1aa8149e30 con 0x7f1aa80caa00 2024-05-09T05:05:42.465 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.465+0000 7f1a8f7fe640 1 -- 172.21.15.196:0/1065954570 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "4f41003a-2ede-4fa9-a0d3-66c790841802", "id": 3}]=0 v6) ==== 112+0+2 (secure 0 0 0) 0x7f1a9c01f400 con 0x7f1aa80caa00 2024-05-09T05:05:42.465 INFO:teuthology.orchestra.run.smithi196.stdout:3 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa80caa00 msgr2=0x7f1aa815b230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa80caa00 0x7f1aa815b230 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c00da70 tx=0x7f1a9c00df30 comp rx=0 tx=0).stop 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 shutdown_connections 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1aa80caa00 0x7f1aa815b230 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1aa815cf60 0x7f1aa8144280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 --2- 172.21.15.196:0/1065954570 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1aa80749f0 0x7f1aa81ee360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 >> 172.21.15.196:0/1065954570 conn(0x7f1aa8070f40 msgr2=0x7f1aa80c7ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 shutdown_connections 2024-05-09T05:05:42.467 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.466+0000 7f1aad36c640 1 -- 172.21.15.196:0/1065954570 wait complete. 2024-05-09T05:05:42.477 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 62cd371e-e40a-4361-8778-94e828501072 4 2024-05-09T05:05:42.543 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:42.542+0000 7f75b19ee040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T05:05:42.566 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:42.564+0000 7fb7c513a040 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T05:05:42.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.589+0000 7f3a20b14640 1 Processor -- start 2024-05-09T05:05:42.590 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.589+0000 7f3a20b14640 1 -- start start 2024-05-09T05:05:42.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a20b14640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 0x7f3a1c154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a20b14640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3a1c0cab00 0x7f3a1c0cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a20b14640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3a1c0ca4a0 0x7f3a1c0cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a20b14640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3a1c070c60 con 0x7f3a1c0cab00 2024-05-09T05:05:42.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a20b14640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f3a1c154a50 con 0x7f3a1c0ca4a0 2024-05-09T05:05:42.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a20b14640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3a1c15a800 con 0x7f3a1c0b2850 2024-05-09T05:05:42.596 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1a59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 0x7f3a1c154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1ad9d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3a1c0ca4a0 0x7f3a1c0cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1a59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 0x7f3a1c154510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:41124/0 (socket says 172.21.15.196:41124) 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1a59c640 1 -- 172.21.15.196:0/2475086947 learned_addr learned my addr 172.21.15.196:0/2475086947 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a19d9b640 1 --2- 172.21.15.196:0/2475086947 >> v2:172.21.15.1:3300/0 conn(0x7f3a1c0cab00 0x7f3a1c0cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1a59c640 1 -- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 msgr2=0x7f3a1c154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1a59c640 1 -- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 msgr2=0x7f3a1c154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1a59c640 1 --2- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 0x7f3a1c154510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1a59c640 1 --2- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 0x7f3a1c154510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1ad9d640 1 -- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 msgr2=0x7f3a1c154510 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:42.597 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1ad9d640 1 --2- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 0x7f3a1c154510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1ad9d640 1 -- 172.21.15.196:0/2475086947 >> v2:172.21.15.1:3300/0 conn(0x7f3a1c0cab00 msgr2=0x7f3a1c0cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1ad9d640 1 --2- 172.21.15.196:0/2475086947 >> v2:172.21.15.1:3300/0 conn(0x7f3a1c0cab00 0x7f3a1c0cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.591+0000 7f3a1ad9d640 1 -- 172.21.15.196:0/2475086947 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c15aae0 con 0x7f3a1c0ca4a0 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.592+0000 7f3a1ad9d640 1 --2- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0ca4a0 0x7f3a1c0cb760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0b31b0 tx=0x7f3a1402e960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48982bf2da7b1ca8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.592+0000 7f3a1959a640 1 -- 172.21.15.196:0/2475086947 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14038020 con 0x7f3a1c0ca4a0 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.592+0000 7f3a1959a640 1 -- 172.21.15.196:0/2475086947 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a14030040 con 0x7f3a1c0ca4a0 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.592+0000 7f3a1959a640 1 -- 172.21.15.196:0/2475086947 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a14034a80 con 0x7f3a1c0ca4a0 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 -- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0ca4a0 msgr2=0x7f3a1c0cb760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 --2- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0ca4a0 0x7f3a1c0cb760 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c0b31b0 tx=0x7f3a1402e960 comp rx=0 tx=0).stop 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 -- 172.21.15.196:0/2475086947 shutdown_connections 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 --2- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0ca4a0 0x7f3a1c0cb760 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.598 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 --2- 172.21.15.196:0/2475086947 >> v2:172.21.15.1:3300/0 conn(0x7f3a1c0cab00 0x7f3a1c0cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 --2- 172.21.15.196:0/2475086947 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0b2850 0x7f3a1c154510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 -- 172.21.15.196:0/2475086947 >> 172.21.15.196:0/2475086947 conn(0x7f3a1c072c30 msgr2=0x7f3a1c073040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 -- 172.21.15.196:0/2475086947 shutdown_connections 2024-05-09T05:05:42.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.593+0000 7f3a20b14640 1 -- 172.21.15.196:0/2475086947 wait complete. 2024-05-09T05:05:42.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.595+0000 7f3a20b14640 1 Processor -- start 2024-05-09T05:05:42.599 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.595+0000 7f3a20b14640 1 -- start start 2024-05-09T05:05:42.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a20b14640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0b2850 0x7f3a1c150690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a20b14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a1c0ca4a0 0x7f3a1c0c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a20b14640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 0x7f3a1c0cf550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a20b14640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c15b380 con 0x7f3a1c0ca4a0 2024-05-09T05:05:42.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a20b14640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3a1c15b0a0 con 0x7f3a1c0b2850 2024-05-09T05:05:42.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a20b14640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3a1c0710f0 con 0x7f3a1c0cab00 2024-05-09T05:05:42.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a1a59c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0b2850 0x7f3a1c150690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.602 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a1a59c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0b2850 0x7f3a1c150690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42050/0 (socket says 172.21.15.196:42050) 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a1a59c640 1 -- 172.21.15.196:0/56338891 learned_addr learned my addr 172.21.15.196:0/56338891 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a1ad9d640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 0x7f3a1c0cf550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a19d9b640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a1c0ca4a0 0x7f3a1c0c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a1ad9d640 1 -- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 msgr2=0x7f3a1c0cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a1ad9d640 1 -- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 msgr2=0x7f3a1c0cf550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.596+0000 7f3a1ad9d640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 0x7f3a1c0cf550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a1ad9d640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 0x7f3a1c0cf550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a1a59c640 1 -- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 msgr2=0x7f3a1c0cf550 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a1a59c640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 0x7f3a1c0cf550 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a1a59c640 1 -- 172.21.15.196:0/56338891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a1c0ca4a0 msgr2=0x7f3a1c0c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a1a59c640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a1c0ca4a0 0x7f3a1c0c93a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a1a59c640 1 -- 172.21.15.196:0/56338891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a1c06e780 con 0x7f3a1c0b2850 2024-05-09T05:05:42.603 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a19d9b640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a1c0ca4a0 0x7f3a1c0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a1a59c640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0b2850 0x7f3a1c150690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00e840 tx=0x7f3a0c00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a12ffd640 1 -- 172.21.15.196:0/56338891 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c017070 con 0x7f3a1c0b2850 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a12ffd640 1 -- 172.21.15.196:0/56338891 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3a0c004020 con 0x7f3a1c0b2850 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a1c0cdbd0 con 0x7f3a1c0b2850 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a12ffd640 1 -- 172.21.15.196:0/56338891 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c00f040 con 0x7f3a1c0b2850 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a1c0cf0a0 con 0x7f3a1c0b2850 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a12ffd640 1 -- 172.21.15.196:0/56338891 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3a0c00c670 con 0x7f3a1c0b2850 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.597+0000 7f3a12ffd640 1 -- 172.21.15.196:0/56338891 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(6..6 src has 1..6) ==== 1185+0+0 (secure 0 0 0) 0x7f3a0c013410 con 0x7f3a1c0b2850 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.598+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a1c06a3c0 con 0x7f3a1c0b2850 2024-05-09T05:05:42.604 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.600+0000 7f3a12ffd640 1 -- 172.21.15.196:0/56338891 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f3a1c06a3c0 con 0x7f3a1c0b2850 2024-05-09T05:05:42.634 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:42.634+0000 7f75b19ee040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T05:05:42.657 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:42.656+0000 7fb7c513a040 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T05:05:42.673 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.672+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "62cd371e-e40a-4361-8778-94e828501072", "id": 4} v 0) -- 0x7f3a1c050870 con 0x7f3a1c0b2850 2024-05-09T05:05:42.677 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.677+0000 7f3a12ffd640 1 -- 172.21.15.196:0/56338891 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "62cd371e-e40a-4361-8778-94e828501072", "id": 4}]=0 v7) ==== 112+0+2 (secure 0 0 0) 0x7f3a0c003610 con 0x7f3a1c0b2850 2024-05-09T05:05:42.677 INFO:teuthology.orchestra.run.smithi196.stdout:4 2024-05-09T05:05:42.678 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0b2850 msgr2=0x7f3a1c150690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0b2850 0x7f3a1c150690 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c00e840 tx=0x7f3a0c00ed00 comp rx=0 tx=0).stop 2024-05-09T05:05:42.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 shutdown_connections 2024-05-09T05:05:42.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3a1c0cab00 0x7f3a1c0cf550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3a1c0ca4a0 0x7f3a1c0c93a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 --2- 172.21.15.196:0/56338891 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3a1c0b2850 0x7f3a1c150690 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 >> 172.21.15.196:0/56338891 conn(0x7f3a1c072c30 msgr2=0x7f3a1c159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 shutdown_connections 2024-05-09T05:05:42.679 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.678+0000 7f3a20b14640 1 -- 172.21.15.196:0/56338891 wait complete. 2024-05-09T05:05:42.680 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:42.680+0000 7f75b19ee040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T05:05:42.690 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 33179512-6a79-4c65-a6cc-f26eb69974d5 5 2024-05-09T05:05:42.702 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:42.701+0000 7fb7c513a040 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T05:05:42.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.799+0000 7f3703413640 1 Processor -- start 2024-05-09T05:05:42.800 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.799+0000 7f3703413640 1 -- start start 2024-05-09T05:05:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f3703413640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f36fc156a10 0x7f36fc15d020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f3703413640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f36fc15d560 0x7f36fc0cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f3703413640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f36fc070c60 con 0x7f36fc0cb760 2024-05-09T05:05:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f3703413640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f36fc15c840 con 0x7f36fc156a10 2024-05-09T05:05:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f3703413640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f36fc0cc010 con 0x7f36fc15d560 2024-05-09T05:05:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f37009ae640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f36fc15d560 0x7f36fc0cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.802 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f37009ae640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f36fc15d560 0x7f36fc0cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42066/0 (socket says 172.21.15.196:42066) 2024-05-09T05:05:42.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f37009ae640 1 -- 172.21.15.196:0/168888868 learned_addr learned my addr 172.21.15.196:0/168888868 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:42.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f37011af640 1 --2- 172.21.15.196:0/168888868 >> v2:172.21.15.1:3300/0 conn(0x7f36fc156a10 0x7f36fc15d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f37009ae640 1 -- 172.21.15.196:0/168888868 >> v2:172.21.15.1:3300/0 conn(0x7f36fc156a10 msgr2=0x7f36fc15d020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:42.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f37009ae640 1 --2- 172.21.15.196:0/168888868 >> v2:172.21.15.1:3300/0 conn(0x7f36fc156a10 0x7f36fc15d020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f37009ae640 1 -- 172.21.15.196:0/168888868 >> v1:172.21.15.1:6789/0 conn(0x7f36fc0cb760 legacy=0x7f36fc0ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.803 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.801+0000 7f37009ae640 1 -- 172.21.15.196:0/168888868 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36fc06e310 con 0x7f36fc15d560 2024-05-09T05:05:42.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.802+0000 7f37009ae640 1 --2- 172.21.15.196:0/168888868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc15d560 0x7f36fc0cb220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f36fc071f20 tx=0x7f36f802a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5a800be9af1fa05 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:42.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.802+0000 7f36effff640 1 -- 172.21.15.196:0/168888868 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8037070 con 0x7f36fc15d560 2024-05-09T05:05:42.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.802+0000 7f36effff640 1 -- 172.21.15.196:0/168888868 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f802ae50 con 0x7f36fc15d560 2024-05-09T05:05:42.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.802+0000 7f36effff640 1 -- 172.21.15.196:0/168888868 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f8032860 con 0x7f36fc15d560 2024-05-09T05:05:42.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.803+0000 7f3703413640 1 -- 172.21.15.196:0/168888868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc15d560 msgr2=0x7f36fc0cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.803+0000 7f3703413640 1 --2- 172.21.15.196:0/168888868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc15d560 0x7f36fc0cb220 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f36fc071f20 tx=0x7f36f802a300 comp rx=0 tx=0).stop 2024-05-09T05:05:42.804 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.803+0000 7f3703413640 1 -- 172.21.15.196:0/168888868 shutdown_connections 2024-05-09T05:05:42.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.803+0000 7f3703413640 1 --2- 172.21.15.196:0/168888868 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc15d560 0x7f36fc0cb220 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.803+0000 7f3703413640 1 --2- 172.21.15.196:0/168888868 >> v2:172.21.15.1:3300/0 conn(0x7f36fc156a10 0x7f36fc15d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.803+0000 7f3703413640 1 -- 172.21.15.196:0/168888868 >> 172.21.15.196:0/168888868 conn(0x7f36fc072c30 msgr2=0x7f36fc073040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.803+0000 7f3703413640 1 -- 172.21.15.196:0/168888868 shutdown_connections 2024-05-09T05:05:42.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.803+0000 7f3703413640 1 -- 172.21.15.196:0/168888868 wait complete. 2024-05-09T05:05:42.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.805+0000 7f3703413640 1 Processor -- start 2024-05-09T05:05:42.805 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.805+0000 7f3703413640 1 -- start start 2024-05-09T05:05:42.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f3703413640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc0cb760 0x7f36fc1f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f3703413640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 0x7f36fc151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f3703413640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36fc15d560 0x7f36fc1f2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:42.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f3703413640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc06ebb0 con 0x7f36fc15d560 2024-05-09T05:05:42.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f3703413640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f36fc06e8d0 con 0x7f36fc0cb760 2024-05-09T05:05:42.809 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f3703413640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f36fc15cb00 con 0x7f36fc156a10 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f37011af640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc0cb760 0x7f36fc1f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f37011af640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc0cb760 0x7f36fc1f3900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42080/0 (socket says 172.21.15.196:42080) 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f37011af640 1 -- 172.21.15.196:0/4170026548 learned_addr learned my addr 172.21.15.196:0/4170026548 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f37009ae640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 0x7f36fc151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.806+0000 7f37019b0640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36fc15d560 0x7f36fc1f2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37009ae640 1 -- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 msgr2=0x7f36fc151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37009ae640 1 -- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 msgr2=0x7f36fc151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37009ae640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 0x7f36fc151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37009ae640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 0x7f36fc151ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37011af640 1 -- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 msgr2=0x7f36fc151ee0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37011af640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 0x7f36fc151ee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37011af640 1 -- 172.21.15.196:0/4170026548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36fc15d560 msgr2=0x7f36fc1f2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.810 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37011af640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36fc15d560 0x7f36fc1f2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37011af640 1 -- 172.21.15.196:0/4170026548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36fc1f3e40 con 0x7f36fc0cb760 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f37011af640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc0cb760 0x7f36fc1f3900 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e870 tx=0x7f36f000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f36edffb640 1 -- 172.21.15.196:0/4170026548 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f0017070 con 0x7f36fc0cb760 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f36edffb640 1 -- 172.21.15.196:0/4170026548 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f36f000c940 con 0x7f36fc0cb760 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36fc0b0ac0 con 0x7f36fc0cb760 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f36edffb640 1 -- 172.21.15.196:0/4170026548 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36f000cd30 con 0x7f36fc0cb760 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36fc0cf1c0 con 0x7f36fc0cb760 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.807+0000 7f36edffb640 1 -- 172.21.15.196:0/4170026548 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f36f001a070 con 0x7f36fc0cb760 2024-05-09T05:05:42.811 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.808+0000 7f36edffb640 1 -- 172.21.15.196:0/4170026548 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(7..7 src has 1..7) ==== 1300+0+0 (secure 0 0 0) 0x7f36f001c9d0 con 0x7f36fc0cb760 2024-05-09T05:05:42.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.809+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f36fc050870 con 0x7f36fc0cb760 2024-05-09T05:05:42.812 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.811+0000 7f36edffb640 1 -- 172.21.15.196:0/4170026548 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f36f001c450 con 0x7f36fc0cb760 2024-05-09T05:05:42.854 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:42.853+0000 7f75b19ee040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T05:05:42.877 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:42.877+0000 7fb7c513a040 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T05:05:42.883 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.882+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "33179512-6a79-4c65-a6cc-f26eb69974d5", "id": 5} v 0) -- 0x7f36fc0cb000 con 0x7f36fc0cb760 2024-05-09T05:05:42.886 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.886+0000 7f36edffb640 1 -- 172.21.15.196:0/4170026548 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "33179512-6a79-4c65-a6cc-f26eb69974d5", "id": 5}]=0 v8) ==== 112+0+2 (secure 0 0 0) 0x7f36f001c630 con 0x7f36fc0cb760 2024-05-09T05:05:42.886 INFO:teuthology.orchestra.run.smithi196.stdout:5 2024-05-09T05:05:42.887 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc0cb760 msgr2=0x7f36fc1f3900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:42.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc0cb760 0x7f36fc1f3900 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f36f000e870 tx=0x7f36f000ed30 comp rx=0 tx=0).stop 2024-05-09T05:05:42.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 shutdown_connections 2024-05-09T05:05:42.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f36fc15d560 0x7f36fc1f2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f36fc156a10 0x7f36fc151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 --2- 172.21.15.196:0/4170026548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f36fc0cb760 0x7f36fc1f3900 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:42.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 >> 172.21.15.196:0/4170026548 conn(0x7f36fc072c30 msgr2=0x7f36fc154550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:42.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 shutdown_connections 2024-05-09T05:05:42.888 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:42.887+0000 7f3703413640 1 -- 172.21.15.196:0/4170026548 wait complete. 2024-05-09T05:05:42.898 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new fecedcf1-e751-4020-8a8f-d3e6057c6e29 6 2024-05-09T05:05:42.904 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:42.903+0000 7f75b19ee040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T05:05:42.931 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:42.931+0000 7fb7c513a040 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T05:05:42.959 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:42.958+0000 7f75b19ee040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T05:05:42.988 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:42.988+0000 7fb7c513a040 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T05:05:43.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.042+0000 7f010aeb8640 1 Processor -- start 2024-05-09T05:05:43.043 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.042+0000 7f010aeb8640 1 -- start start 2024-05-09T05:05:43.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.043+0000 7f010aeb8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f01040c8b70 0x7f0104156b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.043+0000 7f010aeb8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 0x7f01040717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.043+0000 7f010aeb8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0104071d30 0x7f0104159700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.046 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.043+0000 7f010aeb8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f01040cb450 con 0x7f01040c8b70 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.043+0000 7f010aeb8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f01040c9e50 con 0x7f0104071d30 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.043+0000 7f010aeb8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f01040caed0 con 0x7f01040c9520 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0109455640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0104071d30 0x7f0104159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0103fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 0x7f01040717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0109455640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0104071d30 0x7f0104159700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42090/0 (socket says 172.21.15.196:42090) 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0109455640 1 -- 172.21.15.196:0/3832696676 learned_addr learned my addr 172.21.15.196:0/3832696676 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0108c54640 1 --2- 172.21.15.196:0/3832696676 >> v2:172.21.15.1:3300/0 conn(0x7f01040c8b70 0x7f0104156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0103fff640 1 -- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 msgr2=0x7f01040717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0103fff640 1 -- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 msgr2=0x7f01040717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:43.047 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0103fff640 1 --2- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 0x7f01040717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0103fff640 1 --2- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 0x7f01040717f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0109455640 1 -- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 msgr2=0x7f01040717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0109455640 1 --2- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 0x7f01040717f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0109455640 1 -- 172.21.15.196:0/3832696676 >> v2:172.21.15.1:3300/0 conn(0x7f01040c8b70 msgr2=0x7f0104156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0109455640 1 --2- 172.21.15.196:0/3832696676 >> v2:172.21.15.1:3300/0 conn(0x7f01040c8b70 0x7f0104156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.044+0000 7f0109455640 1 -- 172.21.15.196:0/3832696676 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01040cb170 con 0x7f0104071d30 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f0109455640 1 --2- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0104071d30 0x7f0104159700 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0104073f00 tx=0x7f00fc02a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a49194859117f516 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f01037fe640 1 -- 172.21.15.196:0/3832696676 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc036070 con 0x7f0104071d30 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f01037fe640 1 -- 172.21.15.196:0/3832696676 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00fc031600 con 0x7f0104071d30 2024-05-09T05:05:43.048 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f01037fe640 1 -- 172.21.15.196:0/3832696676 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00fc0319d0 con 0x7f0104071d30 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f010aeb8640 1 -- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0104071d30 msgr2=0x7f0104159700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f010aeb8640 1 --2- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0104071d30 0x7f0104159700 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0104073f00 tx=0x7f00fc02a6a0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f010aeb8640 1 -- 172.21.15.196:0/3832696676 shutdown_connections 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f010aeb8640 1 --2- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0104071d30 0x7f0104159700 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f010aeb8640 1 --2- 172.21.15.196:0/3832696676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c9520 0x7f01040717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.045+0000 7f010aeb8640 1 --2- 172.21.15.196:0/3832696676 >> v2:172.21.15.1:3300/0 conn(0x7f01040c8b70 0x7f0104156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.046+0000 7f010aeb8640 1 -- 172.21.15.196:0/3832696676 >> 172.21.15.196:0/3832696676 conn(0x7f010406d800 msgr2=0x7f010406dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.046+0000 7f010aeb8640 1 -- 172.21.15.196:0/3832696676 shutdown_connections 2024-05-09T05:05:43.049 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.046+0000 7f010aeb8640 1 -- 172.21.15.196:0/3832696676 wait complete. 2024-05-09T05:05:43.052 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.047+0000 7f010aeb8640 1 Processor -- start 2024-05-09T05:05:43.052 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.047+0000 7f010aeb8640 1 -- start start 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f010aeb8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0104071d30 0x7f01040cd280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f010aeb8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 0x7f01041df880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f010aeb8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01040c9520 0x7f01041554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f010aeb8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0104071210 con 0x7f0104071d30 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f010aeb8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f01040c3f30 con 0x7f01040c9520 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f010aeb8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f01041570d0 con 0x7f01040c8b70 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0103fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 0x7f01041df880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0109455640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01040c9520 0x7f01041554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0103fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 0x7f01041df880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:41160/0 (socket says 172.21.15.196:41160) 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0109455640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01040c9520 0x7f01041554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.196:42100/0 (socket says 172.21.15.196:42100) 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0103fff640 1 -- 172.21.15.196:0/3554359712 learned_addr learned my addr 172.21.15.196:0/3554359712 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0108c54640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0104071d30 0x7f01040cd280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0103fff640 1 -- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 msgr2=0x7f01041df880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:05:43.053 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0103fff640 1 -- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 msgr2=0x7f01041df880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0103fff640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 0x7f01041df880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0103fff640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 0x7f01041df880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0108c54640 1 -- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 msgr2=0x7f01041df880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0108c54640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 0x7f01041df880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0108c54640 1 -- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01040c9520 msgr2=0x7f01041554e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0108c54640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01040c9520 0x7f01041554e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0108c54640 1 -- 172.21.15.196:0/3554359712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01040cd7c0 con 0x7f0104071d30 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0109455640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01040c9520 0x7f01041554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f0108c54640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0104071d30 0x7f01040cd280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f00f8010b40 tx=0x7f00f800e360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.049+0000 7f01017fa640 1 -- 172.21.15.196:0/3554359712 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f801e070 con 0x7f0104071d30 2024-05-09T05:05:43.054 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.050+0000 7f01017fa640 1 -- 172.21.15.196:0/3554359712 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00f8011450 con 0x7f0104071d30 2024-05-09T05:05:43.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.050+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01040ccac0 con 0x7f0104071d30 2024-05-09T05:05:43.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.050+0000 7f01017fa640 1 -- 172.21.15.196:0/3554359712 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00f8011800 con 0x7f0104071d30 2024-05-09T05:05:43.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.050+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01040c8610 con 0x7f0104071d30 2024-05-09T05:05:43.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.050+0000 7f01017fa640 1 -- 172.21.15.196:0/3554359712 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f00f8021070 con 0x7f0104071d30 2024-05-09T05:05:43.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.050+0000 7f01017fa640 1 -- 172.21.15.196:0/3554359712 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(8..8 src has 1..8) ==== 1415+0+0 (secure 0 0 0) 0x7f00f80067e0 con 0x7f0104071d30 2024-05-09T05:05:43.055 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.050+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01040696d0 con 0x7f0104071d30 2024-05-09T05:05:43.058 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.053+0000 7f01017fa640 1 -- 172.21.15.196:0/3554359712 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f00f8028070 con 0x7f0104071d30 2024-05-09T05:05:43.077 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:43.076+0000 7f75b19ee040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T05:05:43.108 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:43.107+0000 7fb7c513a040 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T05:05:43.127 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.126+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd new", "uuid": "fecedcf1-e751-4020-8a8f-d3e6057c6e29", "id": 6} v 0) -- 0x7f01040c9210 con 0x7f0104071d30 2024-05-09T05:05:43.128 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.128+0000 7f01017fa640 1 -- 172.21.15.196:0/3554359712 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "fecedcf1-e751-4020-8a8f-d3e6057c6e29", "id": 6}]=0 v9) ==== 112+0+2 (secure 0 0 0) 0x7f01040c9210 con 0x7f0104071d30 2024-05-09T05:05:43.128 INFO:teuthology.orchestra.run.smithi196.stdout:6 2024-05-09T05:05:43.129 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0104071d30 msgr2=0x7f01040cd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0104071d30 0x7f01040cd280 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f00f8010b40 tx=0x7f00f800e360 comp rx=0 tx=0).stop 2024-05-09T05:05:43.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 shutdown_connections 2024-05-09T05:05:43.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01040c9520 0x7f01041554e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01040c8b70 0x7f01041df880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 --2- 172.21.15.196:0/3554359712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0104071d30 0x7f01040cd280 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 >> 172.21.15.196:0/3554359712 conn(0x7f010406d800 msgr2=0x7f010414f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:43.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 shutdown_connections 2024-05-09T05:05:43.130 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.129+0000 7f010aeb8640 1 -- 172.21.15.196:0/3554359712 wait complete. 2024-05-09T05:05:43.141 DEBUG:teuthology.orchestra.run.smithi196:> sudo ceph --cluster ceph osd new 676efa33-462a-4779-ba5a-5ab31c6896ac 7 2024-05-09T05:05:43.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.252+0000 7f4817a13640 1 Processor -- start 2024-05-09T05:05:43.253 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.252+0000 7f4817a13640 1 -- start start 2024-05-09T05:05:43.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.253+0000 7f4817a13640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 0x7f481015d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.253+0000 7f4817a13640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f48100c5600 0x7f481006d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.253+0000 7f4817a13640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f48100cc1b0 con 0x7f481006dde0 2024-05-09T05:05:43.257 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.253+0000 7f4817a13640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f481006d710 con 0x7f48100c5600 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.253+0000 7f4817a13640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f48100cbf60 con 0x7f48100736b0 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4814fae640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 0x7f481015d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f48157af640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f481006dde0 0x7f48100c70b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.196:56114/0 (socket says 172.21.15.196:56114) 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f48157af640 1 -- 172.21.15.196:0/3084640810 learned_addr learned my addr 172.21.15.196:0/3084640810 (peer_addr_for_me v1:172.21.15.196:0/0) 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4814fae640 1 -- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 msgr2=0x7f481015d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4814fae640 1 -- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 msgr2=0x7f481015d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4814fae640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 0x7f481015d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4814fae640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 0x7f481015d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4815fb0640 1 --2- 172.21.15.196:0/3084640810 >> v2:172.21.15.1:3300/0 conn(0x7f48100c5600 0x7f481006d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367959989 0 0) 0x7f48100cc1b0 con 0x7f481006dde0 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f47fc003520 con 0x7f481006dde0 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4175711971 0 0) 0x7f47fc003520 con 0x7f481006dde0 2024-05-09T05:05:43.258 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 msgr2=0x7f481015d330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:43.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 0x7f481015d330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.259 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 >> v2:172.21.15.1:3300/0 conn(0x7f48100c5600 msgr2=0x7f481006d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:43.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 --2- 172.21.15.196:0/3084640810 >> v2:172.21.15.1:3300/0 conn(0x7f48100c5600 0x7f481006d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48100cc490 con 0x7f481006dde0 2024-05-09T05:05:43.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f48000016f0 con 0x7f481006dde0 2024-05-09T05:05:43.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 >> v1:172.21.15.196:6789/0 conn(0x7f481006dde0 legacy=0x7f48100c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.261 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47fc003a50 0x7f47fc003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 0x7f47fc004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc004d40 0x7f47fc005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f481006d710 con 0x7f47fc004d40 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f48100cbf60 con 0x7f47fc003a50 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f47fc00bb70 con 0x7f48100736b0 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f48157af640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47fc003a50 0x7f47fc003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4814fae640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 0x7f47fc004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.254+0000 7f4815fb0640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc004d40 0x7f47fc005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f48157af640 1 -- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 msgr2=0x7f47fc004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f48157af640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f48100736b0 0x7f47fc004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f48157af640 1 -- 172.21.15.196:0/3084640810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc004d40 msgr2=0x7f47fc005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f48157af640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f47fc004d40 0x7f47fc005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.262 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f48157af640 1 -- 172.21.15.196:0/3084640810 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc003520 con 0x7f47fc003a50 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f48157af640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47fc003a50 0x7f47fc003ec0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f480000c210 tx=0x7f47fc005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91eaa3bf3fa0b513 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800007d40 con 0x7f47fc003a50 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4800005ad0 con 0x7f47fc003a50 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.255+0000 7f4807fff640 1 -- 172.21.15.196:0/3084640810 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4800007070 con 0x7f47fc003a50 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.256+0000 7f4817a13640 1 -- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47fc003a50 msgr2=0x7f47fc003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.256+0000 7f4817a13640 1 --2- 172.21.15.196:0/3084640810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f47fc003a50 0x7f47fc003ec0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f480000c210 tx=0x7f47fc005830 comp rx=0 tx=0).stop 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.256+0000 7f4815fb0640 1 -- 172.21.15.196:0/3084640810 reap_dead start 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.256+0000 7f4817a13640 1 -- 172.21.15.196:0/3084640810 shutdown_connections 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.256+0000 7f4817a13640 1 -- 172.21.15.196:0/3084640810 >> 172.21.15.196:0/3084640810 conn(0x7f48100ce180 msgr2=0x7f48100ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.256+0000 7f4817a13640 1 -- 172.21.15.196:0/3084640810 shutdown_connections 2024-05-09T05:05:43.263 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.256+0000 7f4817a13640 1 -- 172.21.15.196:0/3084640810 wait complete. 2024-05-09T05:05:43.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.258+0000 7f4817a13640 1 Processor -- start 2024-05-09T05:05:43.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.258+0000 7f4817a13640 1 -- start start 2024-05-09T05:05:43.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.259+0000 7f4817a13640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 0x7f48100c90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.259+0000 7f4817a13640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48101e5ce0 0x7f48101f5ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.266 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.259+0000 7f4817a13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48101f6010 0x7f481014e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.259+0000 7f4817a13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f481006ccb0 con 0x7f48101f6010 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.259+0000 7f4817a13640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f481006c9d0 con 0x7f48101e5ce0 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.259+0000 7f4817a13640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f48100c77c0 con 0x7f47fc003a50 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.259+0000 7f48157af640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 0x7f48100c90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f48157af640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 0x7f48100c90d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.196:41188/0 (socket says 172.21.15.196:41188) 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f48157af640 1 -- 172.21.15.196:0/1732506096 learned_addr learned my addr 172.21.15.196:0/1732506096 (peer_addr_for_me v2:172.21.15.196:0/0) 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4814fae640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48101e5ce0 0x7f48101f5ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4815fb0640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48101f6010 0x7f481014e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f48157af640 1 -- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 msgr2=0x7f48100c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f48157af640 1 -- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 msgr2=0x7f48100c90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f48157af640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 0x7f48100c90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f48157af640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 0x7f48100c90d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4814fae640 1 -- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 msgr2=0x7f48100c90d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4814fae640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 0x7f48100c90d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.267 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4814fae640 1 -- 172.21.15.196:0/1732506096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48101f6010 msgr2=0x7f481014e9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4814fae640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48101f6010 0x7f481014e9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4814fae640 1 -- 172.21.15.196:0/1732506096 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48100c9610 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4814fae640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48101e5ce0 0x7f48101f5ad0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f48100720f0 tx=0x7f480c010d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4805ffb640 1 -- 172.21.15.196:0/1732506096 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c01a030 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4805ffb640 1 -- 172.21.15.196:0/1732506096 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f480c004020 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48100ca3d0 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4805ffb640 1 -- 172.21.15.196:0/1732506096 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f480c0041f0 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.260+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48101f65d0 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.261+0000 7f4805ffb640 1 -- 172.21.15.196:0/1732506096 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f480c01d070 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.261+0000 7f4805ffb640 1 -- 172.21.15.196:0/1732506096 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1530+0+0 (secure 0 0 0) 0x7f480c016480 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.262+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f481014fc70 con 0x7f48101e5ce0 2024-05-09T05:05:43.268 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.264+0000 7f4805ffb640 1 -- 172.21.15.196:0/1732506096 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f480c0043e0 con 0x7f48101e5ce0 2024-05-09T05:05:43.336 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.336+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd new", "uuid": "676efa33-462a-4779-ba5a-5ab31c6896ac", "id": 7} v 0) -- 0x7f4810050870 con 0x7f48101e5ce0 2024-05-09T05:05:43.347 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.346+0000 7f4805ffb640 1 -- 172.21.15.196:0/1732506096 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "676efa33-462a-4779-ba5a-5ab31c6896ac", "id": 7}]=0 v10) ==== 112+0+2 (secure 0 0 0) 0x7f480c00e9c0 con 0x7f48101e5ce0 2024-05-09T05:05:43.347 INFO:teuthology.orchestra.run.smithi196.stdout:7 2024-05-09T05:05:43.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48101e5ce0 msgr2=0x7f48101f5ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48101e5ce0 0x7f48101f5ad0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f48100720f0 tx=0x7f480c010d50 comp rx=0 tx=0).stop 2024-05-09T05:05:43.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 shutdown_connections 2024-05-09T05:05:43.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f48101e5ce0 0x7f48101f5ad0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.348 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f48101f6010 0x7f481014e9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 --2- 172.21.15.196:0/1732506096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f47fc003a50 0x7f48100c90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 >> 172.21.15.196:0/1732506096 conn(0x7f48100ce180 msgr2=0x7f4810152920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:43.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 shutdown_connections 2024-05-09T05:05:43.349 INFO:teuthology.orchestra.run.smithi196.stderr:2024-05-09T05:05:43.347+0000 7f4817a13640 1 -- 172.21.15.196:0/1732506096 wait complete. 2024-05-09T05:05:43.359 INFO:tasks.ceph.osd.0:Restarting daemon 2024-05-09T05:05:43.359 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2024-05-09T05:05:43.361 INFO:tasks.ceph.osd.0:Started 2024-05-09T05:05:43.362 INFO:tasks.ceph.osd.1:Restarting daemon 2024-05-09T05:05:43.362 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2024-05-09T05:05:43.364 INFO:tasks.ceph.osd.1:Started 2024-05-09T05:05:43.364 INFO:tasks.ceph.osd.2:Restarting daemon 2024-05-09T05:05:43.364 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2024-05-09T05:05:43.366 INFO:tasks.ceph.osd.2:Started 2024-05-09T05:05:43.366 INFO:tasks.ceph.osd.3:Restarting daemon 2024-05-09T05:05:43.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2024-05-09T05:05:43.369 INFO:tasks.ceph.osd.3:Started 2024-05-09T05:05:43.369 INFO:tasks.ceph.osd.4:Restarting daemon 2024-05-09T05:05:43.369 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2024-05-09T05:05:43.372 INFO:tasks.ceph.osd.4:Started 2024-05-09T05:05:43.372 INFO:tasks.ceph.osd.5:Restarting daemon 2024-05-09T05:05:43.372 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2024-05-09T05:05:43.374 INFO:tasks.ceph.osd.5:Started 2024-05-09T05:05:43.374 INFO:tasks.ceph.osd.6:Restarting daemon 2024-05-09T05:05:43.374 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2024-05-09T05:05:43.376 INFO:tasks.ceph.osd.6:Started 2024-05-09T05:05:43.376 INFO:tasks.ceph.osd.7:Restarting daemon 2024-05-09T05:05:43.377 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2024-05-09T05:05:43.379 INFO:tasks.ceph.osd.7:Started 2024-05-09T05:05:43.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:05:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.524+0000 7fe569270640 1 Processor -- start 2024-05-09T05:05:43.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.524+0000 7fe569270640 1 -- start start 2024-05-09T05:05:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.525+0000 7fe569270640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe5640c7eb0 0x7fe5640c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.525+0000 7fe569270640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe5640cb060 con 0x7fe56415e710 2024-05-09T05:05:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.525+0000 7fe569270640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe56415b6f0 con 0x7fe5640ca9b0 2024-05-09T05:05:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.525+0000 7fe569270640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe56415a820 con 0x7fe5640c7eb0 2024-05-09T05:05:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.525+0000 7fe5627fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe56415e710 0x7fe56415afe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57396/0 (socket says 172.21.15.1:57396) 2024-05-09T05:05:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.525+0000 7fe5627fc640 1 -- 172.21.15.1:0/2299066750 learned_addr learned my addr 172.21.15.1:0/2299066750 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:05:43.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.525+0000 7fe5637fe640 1 --2- 172.21.15.1:0/2299066750 >> v2:172.21.15.196:3300/0 conn(0x7fe5640c7eb0 0x7fe5640c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.526 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:43.525+0000 7f75b19ee040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T05:05:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.526+0000 7fe561ffb640 1 -- 172.21.15.1:0/2299066750 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 362696972 0 0) 0x7fe5640cb060 con 0x7fe56415e710 2024-05-09T05:05:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.526+0000 7fe561ffb640 1 -- 172.21.15.1:0/2299066750 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe550003520 con 0x7fe56415e710 2024-05-09T05:05:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.526+0000 7fe561ffb640 1 -- 172.21.15.1:0/2299066750 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3530977207 0 0) 0x7fe55c001490 con 0x7fe5640ca9b0 2024-05-09T05:05:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.526+0000 7fe561ffb640 1 -- 172.21.15.1:0/2299066750 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5640cb060 con 0x7fe5640ca9b0 2024-05-09T05:05:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.526+0000 7fe5637fe640 1 -- 172.21.15.1:0/2299066750 >> v1:172.21.15.196:6789/0 conn(0x7fe5640ca9b0 legacy=0x7fe564154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.526+0000 7fe5637fe640 1 -- 172.21.15.1:0/2299066750 >> v1:172.21.15.1:6789/0 conn(0x7fe56415e710 legacy=0x7fe56415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.526+0000 7fe5637fe640 1 -- 172.21.15.1:0/2299066750 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe5640cadc0 con 0x7fe5640c7eb0 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.526+0000 7fe5637fe640 1 --2- 172.21.15.1:0/2299066750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640c7eb0 0x7fe5640c82c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe5540097a0 tx=0x7fe55402a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb1ee9ccc89a11c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.527+0000 7fe561ffb640 1 -- 172.21.15.1:0/2299066750 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554037070 con 0x7fe5640c7eb0 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.527+0000 7fe561ffb640 1 -- 172.21.15.1:0/2299066750 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe554032440 con 0x7fe5640c7eb0 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.527+0000 7fe561ffb640 1 -- 172.21.15.1:0/2299066750 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe554032810 con 0x7fe5640c7eb0 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.527+0000 7fe569270640 1 -- 172.21.15.1:0/2299066750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640c7eb0 msgr2=0x7fe5640c82c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.527+0000 7fe569270640 1 --2- 172.21.15.1:0/2299066750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640c7eb0 0x7fe5640c82c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe5540097a0 tx=0x7fe55402a1a0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.528+0000 7fe569270640 1 -- 172.21.15.1:0/2299066750 shutdown_connections 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.528+0000 7fe569270640 1 --2- 172.21.15.1:0/2299066750 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640c7eb0 0x7fe5640c82c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.528+0000 7fe569270640 1 -- 172.21.15.1:0/2299066750 >> 172.21.15.1:0/2299066750 conn(0x7fe5640745a0 msgr2=0x7fe564071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.528+0000 7fe569270640 1 -- 172.21.15.1:0/2299066750 shutdown_connections 2024-05-09T05:05:43.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.528+0000 7fe569270640 1 -- 172.21.15.1:0/2299066750 wait complete. 2024-05-09T05:05:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.530+0000 7fe569270640 1 Processor -- start 2024-05-09T05:05:43.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.530+0000 7fe569270640 1 -- start start 2024-05-09T05:05:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe569270640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5640c7eb0 0x7fe5641f0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe569270640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640ca9b0 0x7fe56414a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe569270640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 0x7fe56414e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe569270640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe56415aac0 con 0x7fe5640c7eb0 2024-05-09T05:05:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe569270640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe5640c9810 con 0x7fe5640ca9b0 2024-05-09T05:05:43.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe562ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5640c7eb0 0x7fe5641f0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe569270640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe5640c9af0 con 0x7fe56415e710 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe562ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5640c7eb0 0x7fe5641f0920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50878/0 (socket says 172.21.15.1:50878) 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe562ffd640 1 -- 172.21.15.1:0/2345912544 learned_addr learned my addr 172.21.15.1:0/2345912544 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe5627fc640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640ca9b0 0x7fe56414a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.531+0000 7fe5637fe640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 0x7fe56414e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe5637fe640 1 -- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 msgr2=0x7fe56414e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe5637fe640 1 -- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 msgr2=0x7fe56414e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe5637fe640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 0x7fe56414e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe5637fe640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 0x7fe56414e540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe562ffd640 1 -- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 msgr2=0x7fe56414e540 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe562ffd640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 0x7fe56414e540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe562ffd640 1 -- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640ca9b0 msgr2=0x7fe56414a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe562ffd640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640ca9b0 0x7fe56414a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe562ffd640 1 -- 172.21.15.1:0/2345912544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe564158810 con 0x7fe5640c7eb0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe562ffd640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5640c7eb0 0x7fe5641f0920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00c790 tx=0x7fe55c00cc50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe55b7fe640 1 -- 172.21.15.1:0/2345912544 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c003b70 con 0x7fe5640c7eb0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe55b7fe640 1 -- 172.21.15.1:0/2345912544 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe55c003d30 con 0x7fe5640c7eb0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe55b7fe640 1 -- 172.21.15.1:0/2345912544 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe55c00aa90 con 0x7fe5640c7eb0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5641f40b0 con 0x7fe5640c7eb0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5641f4670 con 0x7fe5640c7eb0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe55b7fe640 1 -- 172.21.15.1:0/2345912544 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe55c00ac30 con 0x7fe5640c7eb0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.532+0000 7fe55b7fe640 1 -- 172.21.15.1:0/2345912544 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7fe55c012d10 con 0x7fe5640c7eb0 2024-05-09T05:05:43.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.533+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe564050870 con 0x7fe5640c7eb0 2024-05-09T05:05:43.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.535+0000 7fe55b7fe640 1 -- 172.21.15.1:0/2345912544 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7fe56415b6f0 con 0x7fe5640c7eb0 2024-05-09T05:05:43.544 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:43.544+0000 7fb7c513a040 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T05:05:43.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.597+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe564158130 con 0x7fe5640c7eb0 2024-05-09T05:05:43.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.598+0000 7fe55b7fe640 1 -- 172.21.15.1:0/2345912544 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7fe55c012600 con 0x7fe5640c7eb0 2024-05-09T05:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:43.599 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":10,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:05:43.337985+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5640c7eb0 msgr2=0x7fe5641f0920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5640c7eb0 0x7fe5641f0920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fe55c00c790 tx=0x7fe55c00cc50 comp rx=0 tx=0).stop 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 shutdown_connections 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe5640ca9b0 0x7fe56414a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe56415e710 0x7fe56414e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 --2- 172.21.15.1:0/2345912544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe5640c7eb0 0x7fe5641f0920 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 >> 172.21.15.1:0/2345912544 conn(0x7fe5640745a0 msgr2=0x7fe5640cd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 shutdown_connections 2024-05-09T05:05:43.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:43.599+0000 7fe569270640 1 -- 172.21.15.1:0/2345912544 wait complete. 2024-05-09T05:05:43.610 INFO:tasks.ceph.ceph_manager.ceph:[] 2024-05-09T05:05:43.611 INFO:tasks.ceph:Waiting for OSDs to come up 2024-05-09T05:05:43.675 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:43.674+0000 7f75b19ee040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-09T05:05:43.681 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:43.681+0000 7fb7c513a040 -1 mgr[py] Module smb has missing NOTIFY_TYPES member 2024-05-09T05:05:43.942 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:43.942+0000 7fb7c513a040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T05:05:43.947 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:43.946+0000 7f75b19ee040 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T05:05:43.982 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-05-09T05:05:43.982+0000 7f8b5f83c640 -1 Falling back to public interface 2024-05-09T05:05:43.992 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-05-09T05:05:43.991+0000 7fa837af0640 -1 Falling back to public interface 2024-05-09T05:05:43.995 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:43.995+0000 7fb7c513a040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T05:05:44.001 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.001+0000 7f75b19ee040 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T05:05:44.006 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-05-09T05:05:44.005+0000 7fd15a03f640 -1 Falling back to public interface 2024-05-09T05:05:44.013 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-05-09T05:05:44.012+0000 7f94f8218640 -1 Falling back to public interface 2024-05-09T05:05:44.013 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-05-09T05:05:44.013+0000 7f8e5f222640 -1 Falling back to public interface 2024-05-09T05:05:44.018 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-05-09T05:05:44.018+0000 7f476fc13640 -1 Falling back to public interface 2024-05-09T05:05:44.018 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-05-09T05:05:44.017+0000 7f423102c640 -1 Falling back to public interface 2024-05-09T05:05:44.020 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-05-09T05:05:44.019+0000 7f71dfc14640 -1 Falling back to public interface 2024-05-09T05:05:44.114 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:44.113+0000 7fb7c513a040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T05:05:44.128 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.127+0000 7f75b19ee040 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T05:05:44.416 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T05:05:44.416 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:44.416+0000 7fb7c513a040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T05:05:44.440 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.440+0000 7f75b19ee040 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T05:05:44.464 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:44.464+0000 7fb7c513a040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T05:05:44.494 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.492+0000 7f75b19ee040 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T05:05:44.522 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.521+0000 7f5fec861640 1 Processor -- start 2024-05-09T05:05:44.522 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.521+0000 7f5fec861640 1 -- start start 2024-05-09T05:05:44.523 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fec861640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fe8154750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:44.523 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fec861640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fe80cba40 con 0x7f5fe80cb220 2024-05-09T05:05:44.536 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fec861640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fe80c5d80 con 0x7f5fe80cab00 2024-05-09T05:05:44.536 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fec861640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5fe80cbe70 con 0x7f5fe815e5b0 2024-05-09T05:05:44.537 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5fe80cb220 0x7f5fe80cc900 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57408/0 (socket says 172.21.15.1:57408) 2024-05-09T05:05:44.537 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe659c640 1 -- 172.21.15.1:0/3879215083 learned_addr learned my addr 172.21.15.1:0/3879215083 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:05:44.537 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe6d9d640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fe8154750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:44.537 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3007773605 0 0) 0x7f5fe80cba40 con 0x7f5fe80cb220 2024-05-09T05:05:44.537 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe6d9d640 1 -- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 msgr2=0x7f5fe8154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:05:44.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe6d9d640 1 -- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 msgr2=0x7f5fe8154750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:05:44.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe6d9d640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fe8154750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:44.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fd0003520 con 0x7f5fe80cb220 2024-05-09T05:05:44.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe6d9d640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fe8154750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:44.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3679357650 0 0) 0x7f5fe80c5d80 con 0x7f5fe80cab00 2024-05-09T05:05:44.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.523+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5fe80cba40 con 0x7f5fe80cab00 2024-05-09T05:05:44.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2626231904 0 0) 0x7f5fd0003520 con 0x7f5fe80cb220 2024-05-09T05:05:44.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 msgr2=0x7f5fe8154750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fe8154750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 >> v1:172.21.15.196:6789/0 conn(0x7f5fe80cab00 legacy=0x7f5fe815dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe80cc150 con 0x7f5fe80cb220 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f5fd80016f0 con 0x7f5fe80cb220 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 >> v1:172.21.15.1:6789/0 conn(0x7f5fe80cb220 legacy=0x7f5fe80cc900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fd0003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fd0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0004e60 0x7f5fd00052b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe80cbe70 con 0x7f5fd00041a0 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd000bca0 con 0x7f5fd0004e60 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd000bea0 con 0x7f5fe815e5b0 2024-05-09T05:05:44.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe659c640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fd0003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:44.540 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe5d9b640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fd0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:44.540 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe6d9d640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0004e60 0x7f5fd00052b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:44.540 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe5d9b640 1 -- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 msgr2=0x7f5fd0003bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:44.540 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe5d9b640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fd0003bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:44.540 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe5d9b640 1 -- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0004e60 msgr2=0x7f5fd00052b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:44.540 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe5d9b640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0004e60 0x7f5fd00052b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:44.540 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe5d9b640 1 -- 172.21.15.1:0/3879215083 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd0003520 con 0x7f5fd00041a0 2024-05-09T05:05:44.540 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe6d9d640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0004e60 0x7f5fd00052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.524+0000 7f5fe659c640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe815e5b0 0x7f5fd0003bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.525+0000 7f5fe5d9b640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fd0004800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5fe81580e0 tx=0x7f5fd0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe5982e653dfa12a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.525+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc036070 con 0x7f5fd00041a0 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.525+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fdc031600 con 0x7f5fd00041a0 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.525+0000 7f5fe559a640 1 -- 172.21.15.1:0/3879215083 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fdc007b50 con 0x7f5fd00041a0 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.526+0000 7f5fec861640 1 -- 172.21.15.1:0/3879215083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 msgr2=0x7f5fd0004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.526+0000 7f5fec861640 1 --2- 172.21.15.1:0/3879215083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fd0004800 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f5fe81580e0 tx=0x7f5fd0005830 comp rx=0 tx=0).stop 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.526+0000 7f5fe6d9d640 1 -- 172.21.15.1:0/3879215083 reap_dead start 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.526+0000 7f5fec861640 1 -- 172.21.15.1:0/3879215083 shutdown_connections 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.526+0000 7f5fec861640 1 -- 172.21.15.1:0/3879215083 >> 172.21.15.1:0/3879215083 conn(0x7f5fe8072d70 msgr2=0x7f5fe8073180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.526+0000 7f5fec861640 1 -- 172.21.15.1:0/3879215083 shutdown_connections 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.526+0000 7f5fec861640 1 -- 172.21.15.1:0/3879215083 wait complete. 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.528+0000 7f5fec861640 1 Processor -- start 2024-05-09T05:05:44.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.528+0000 7f5fec861640 1 -- start start 2024-05-09T05:05:44.542 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.529+0000 7f5fec861640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fe814fd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:44.542 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.529+0000 7f5fec861640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fe814f340 0x7f5fe81f3420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fec861640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 0x7f5fe806f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fec861640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe80cd010 con 0x7f5fd00041a0 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fec861640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5fe80cb700 con 0x7f5fe814f340 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fec861640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5fe80cd190 con 0x7f5fe806ef50 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fe814fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fe814fd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50900/0 (socket says 172.21.15.1:50900) 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 -- 172.21.15.1:0/306739948 learned_addr learned my addr 172.21.15.1:0/306739948 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe5d9b640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fe814f340 0x7f5fe81f3420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe6d9d640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 0x7f5fe806f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe6d9d640 1 -- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 msgr2=0x7f5fe806f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe6d9d640 1 -- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 msgr2=0x7f5fe806f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe6d9d640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 0x7f5fe806f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe6d9d640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 0x7f5fe806f360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:05:44.543 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 -- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 msgr2=0x7f5fe806f360 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 0x7f5fe806f360 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 -- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fe814f340 msgr2=0x7f5fe81f3420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fe814f340 0x7f5fe81f3420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 -- 172.21.15.1:0/306739948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fe81502b0 con 0x7f5fd00041a0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe5d9b640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fe814f340 0x7f5fe81f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fe659c640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fe814fd70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80072c0 tx=0x7f5fd8007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fd6ffd640 1 -- 172.21.15.1:0/306739948 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd800f960 con 0x7f5fd00041a0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fd6ffd640 1 -- 172.21.15.1:0/306739948 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fd8018070 con 0x7f5fd00041a0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fd6ffd640 1 -- 172.21.15.1:0/306739948 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fd800fd00 con 0x7f5fd00041a0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.530+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fe814f890 con 0x7f5fd00041a0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.531+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fe80b3c80 con 0x7f5fd00041a0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.531+0000 7f5fd6ffd640 1 -- 172.21.15.1:0/306739948 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5fd801d460 con 0x7f5fd00041a0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.531+0000 7f5fd6ffd640 1 -- 172.21.15.1:0/306739948 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(10..10 src has 1..10) ==== 1645+0+0 (secure 0 0 0) 0x7f5fd8013de0 con 0x7f5fd00041a0 2024-05-09T05:05:44.544 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.531+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fe8150480 con 0x7f5fd00041a0 2024-05-09T05:05:44.545 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.534+0000 7f5fd6ffd640 1 -- 172.21.15.1:0/306739948 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73389 (secure 0 0 0) 0x7f5fd8013710 con 0x7f5fd00041a0 2024-05-09T05:05:44.598 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.597+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5fe80b2ea0 con 0x7f5fd00041a0 2024-05-09T05:05:44.598 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.598+0000 7f5fd6ffd640 1 -- 172.21.15.1:0/306739948 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v10) ==== 74+0+7058 (secure 0 0 0) 0x7f5fe80b2ea0 con 0x7f5fd00041a0 2024-05-09T05:05:44.599 INFO:teuthology.misc.health.smithi001.stdout: 2024-05-09T05:05:44.599 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":10,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:05:43.337985+0000","last_up_change":"0.000000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:05:44.599 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 msgr2=0x7f5fe814fd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:44.600 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fe814fd70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f5fd80072c0 tx=0x7f5fd8007780 comp rx=0 tx=0).stop 2024-05-09T05:05:44.600 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 shutdown_connections 2024-05-09T05:05:44.600 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fe814f340 0x7f5fe81f3420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:44.600 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fe806ef50 0x7f5fe806f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:44.600 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 --2- 172.21.15.1:0/306739948 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00041a0 0x7f5fe814fd70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:44.600 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 >> 172.21.15.1:0/306739948 conn(0x7f5fe8072d70 msgr2=0x7f5fe815a540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:44.600 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 shutdown_connections 2024-05-09T05:05:44.600 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:44.599+0000 7f5fec861640 1 -- 172.21.15.1:0/306739948 wait complete. 2024-05-09T05:05:44.610 DEBUG:teuthology.misc:0 of 8 OSDs are up 2024-05-09T05:05:44.624 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:44.623+0000 7fb7c513a040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T05:05:44.662 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.661+0000 7f75b19ee040 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T05:05:44.761 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:44.761+0000 7fb7c513a040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T05:05:44.806 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.806+0000 7f75b19ee040 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T05:05:44.811 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:44.811+0000 7fb7c513a040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T05:05:44.859 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.859+0000 7f75b19ee040 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T05:05:44.884 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:44.883+0000 7fb7c513a040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T05:05:44.929 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:44.928+0000 7fb7c513a040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T05:05:44.932 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.932+0000 7f75b19ee040 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T05:05:44.980 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:44.980+0000 7f75b19ee040 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T05:05:45.411 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:45.410+0000 7fb7c513a040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T05:05:45.483 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:45.483+0000 7f75b19ee040 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T05:05:45.527 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:45.526+0000 7fb7c513a040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T05:05:45.609 INFO:tasks.ceph.mgr.y.smithi001.stderr:2024-05-09T05:05:45.608+0000 7f75b19ee040 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T05:05:46.543 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:46.543+0000 7fb76538f640 -1 mgr.server handle_report got status from non-daemon mon.b 2024-05-09T05:05:46.544 INFO:tasks.ceph.mgr.x.smithi196.stderr:2024-05-09T05:05:46.543+0000 7fb76538f640 -1 mgr.server handle_report got status from non-daemon mon.c 2024-05-09T05:05:47.479 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-05-09T05:05:47.479+0000 7f8b5f83c640 -1 osd.6 0 log_to_monitors true 2024-05-09T05:05:47.517 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-05-09T05:05:47.515+0000 7fa837af0640 -1 osd.4 0 log_to_monitors true 2024-05-09T05:05:47.519 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-05-09T05:05:47.519+0000 7fd15a03f640 -1 osd.2 0 log_to_monitors true 2024-05-09T05:05:47.521 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-05-09T05:05:47.521+0000 7f94f8218640 -1 osd.3 0 log_to_monitors true 2024-05-09T05:05:47.528 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-05-09T05:05:47.527+0000 7f8e5f222640 -1 osd.1 0 log_to_monitors true 2024-05-09T05:05:47.532 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-05-09T05:05:47.532+0000 7f476fc13640 -1 osd.0 0 log_to_monitors true 2024-05-09T05:05:47.541 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-05-09T05:05:47.541+0000 7f423102c640 -1 osd.7 0 log_to_monitors true 2024-05-09T05:05:47.570 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-05-09T05:05:47.570+0000 7f71dfc14640 -1 osd.5 0 log_to_monitors true 2024-05-09T05:05:49.848 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-05-09T05:05:49.847+0000 7f8e5ab1a640 -1 osd.1 0 waiting for initial osdmap 2024-05-09T05:05:49.861 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-05-09T05:05:49.860+0000 7fd155fd5640 -1 osd.2 0 waiting for initial osdmap 2024-05-09T05:05:49.861 INFO:tasks.ceph.osd.1.smithi001.stderr:2024-05-09T05:05:49.861+0000 7f8e52915640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:05:49.871 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-05-09T05:05:49.870+0000 7f94f3b80640 -1 osd.3 0 waiting for initial osdmap 2024-05-09T05:05:49.883 INFO:tasks.ceph.osd.2.smithi001.stderr:2024-05-09T05:05:49.883+0000 7fd14ddd0640 -1 osd.2 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:05:49.887 INFO:tasks.ceph.osd.3.smithi001.stderr:2024-05-09T05:05:49.887+0000 7f94eb97b640 -1 osd.3 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:05:50.044 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-05-09T05:05:50.043+0000 7f8b5b7d2640 -1 osd.6 0 waiting for initial osdmap 2024-05-09T05:05:50.071 INFO:tasks.ceph.osd.6.smithi196.stderr:2024-05-09T05:05:50.071+0000 7f8b535cd640 -1 osd.6 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:05:50.074 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-05-09T05:05:50.074+0000 7fa833a86640 -1 osd.4 0 waiting for initial osdmap 2024-05-09T05:05:50.114 INFO:tasks.ceph.osd.4.smithi196.stderr:2024-05-09T05:05:50.114+0000 7fa82b881640 -1 osd.4 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:05:50.453 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-05-09T05:05:50.453+0000 7f476b48c640 -1 osd.0 0 waiting for initial osdmap 2024-05-09T05:05:50.469 INFO:tasks.ceph.osd.0.smithi001.stderr:2024-05-09T05:05:50.469+0000 7f4763287640 -1 osd.0 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:05:50.854 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-05-09T05:05:50.854+0000 7f422c5ae640 -1 osd.7 0 waiting for initial osdmap 2024-05-09T05:05:50.866 INFO:tasks.ceph.osd.7.smithi196.stderr:2024-05-09T05:05:50.866+0000 7f42243a9640 -1 osd.7 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:05:50.869 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-05-09T05:05:50.868+0000 7f71db659640 -1 osd.5 0 waiting for initial osdmap 2024-05-09T05:05:50.883 INFO:tasks.ceph.osd.5.smithi196.stderr:2024-05-09T05:05:50.883+0000 7f71d3454640 -1 osd.5 14 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T05:05:51.416 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T05:05:51.536 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.536+0000 7f436418d640 1 Processor -- start 2024-05-09T05:05:51.537 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.536+0000 7f436418d640 1 -- start start 2024-05-09T05:05:51.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.537+0000 7f436418d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f435c0ceb90 0x7f435c156b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:51.538 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.537+0000 7f436418d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f435c0d1160 0x7f435c15ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.537+0000 7f436418d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f435c0cd0f0 con 0x7f435c0cf910 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.537+0000 7f436418d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f435c0d05c0 con 0x7f435c0d1160 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.537+0000 7f436418d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f435c0d0860 con 0x7f435c0ceb90 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f436272a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f435c0d1160 0x7f435c15ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f4361f29640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f435c0cf910 0x7f435c0ce480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51588/0 (socket says 172.21.15.1:51588) 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f4361f29640 1 -- 172.21.15.1:0/187108674 learned_addr learned my addr 172.21.15.1:0/187108674 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f4361728640 1 --2- 172.21.15.1:0/187108674 >> v2:172.21.15.196:3300/0 conn(0x7f435c0ceb90 0x7f435c156b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f436272a640 1 -- 172.21.15.1:0/187108674 >> v2:172.21.15.196:3300/0 conn(0x7f435c0ceb90 msgr2=0x7f435c156b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f436272a640 1 --2- 172.21.15.1:0/187108674 >> v2:172.21.15.196:3300/0 conn(0x7f435c0ceb90 0x7f435c156b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f436272a640 1 -- 172.21.15.1:0/187108674 >> v1:172.21.15.1:6789/0 conn(0x7f435c0cf910 legacy=0x7f435c0ce480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f436272a640 1 -- 172.21.15.1:0/187108674 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c06f400 con 0x7f435c0d1160 2024-05-09T05:05:51.539 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.538+0000 7f4360f27640 1 -- 172.21.15.1:0/187108674 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 485495727 0 0) 0x7f435c0cd0f0 con 0x7f435c0cf910 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.539+0000 7f436272a640 1 --2- 172.21.15.1:0/187108674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c15ca90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4354009d90 tx=0x7f435402a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5e96162e24e4558 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.539+0000 7f4360f27640 1 -- 172.21.15.1:0/187108674 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354007740 con 0x7f435c0d1160 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.539+0000 7f4360f27640 1 -- 172.21.15.1:0/187108674 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f43540078e0 con 0x7f435c0d1160 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.539+0000 7f4360f27640 1 -- 172.21.15.1:0/187108674 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4354007cb0 con 0x7f435c0d1160 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.540+0000 7f436418d640 1 -- 172.21.15.1:0/187108674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 msgr2=0x7f435c15ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.540+0000 7f436418d640 1 --2- 172.21.15.1:0/187108674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c15ca90 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4354009d90 tx=0x7f435402a6a0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.540+0000 7f436418d640 1 -- 172.21.15.1:0/187108674 shutdown_connections 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.540+0000 7f436418d640 1 --2- 172.21.15.1:0/187108674 >> v2:172.21.15.196:3300/0 conn(0x7f435c0ceb90 0x7f435c156b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.540+0000 7f436418d640 1 --2- 172.21.15.1:0/187108674 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c15ca90 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.540+0000 7f436418d640 1 -- 172.21.15.1:0/187108674 >> 172.21.15.1:0/187108674 conn(0x7f435c0b3620 msgr2=0x7f435c0b3a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.540+0000 7f436418d640 1 -- 172.21.15.1:0/187108674 shutdown_connections 2024-05-09T05:05:51.541 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.540+0000 7f436418d640 1 -- 172.21.15.1:0/187108674 wait complete. 2024-05-09T05:05:51.542 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.541+0000 7f436418d640 1 Processor -- start 2024-05-09T05:05:51.542 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.542+0000 7f436418d640 1 -- start start 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f436418d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f435c0cf910 0x7f435c0b70b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f436418d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c1f27c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f436418d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f435c1e93e0 0x7f435c0b4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f436418d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f435c0d0c10 con 0x7f435c0d1160 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f436418d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f435c0cfe00 con 0x7f435c0cf910 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f436418d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f435c0d0d90 con 0x7f435c1e93e0 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c1f27c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c1f27c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38026/0 (socket says 172.21.15.1:38026) 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 -- 172.21.15.1:0/793760431 learned_addr learned my addr 172.21.15.1:0/793760431 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:05:51.546 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361f29640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f435c0cf910 0x7f435c0b70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f436272a640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f435c1e93e0 0x7f435c0b4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 -- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f435c1e93e0 msgr2=0x7f435c0b4580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f435c1e93e0 0x7f435c0b4580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 -- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f435c0cf910 msgr2=0x7f435c0b70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f435c0cf910 0x7f435c0b70b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 -- 172.21.15.1:0/793760431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f435c0ca740 con 0x7f435c0d1160 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.543+0000 7f4361728640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c1f27c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f434c00d620 tx=0x7f434c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.544+0000 7f43527fc640 1 -- 172.21.15.1:0/793760431 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c00b620 con 0x7f435c0d1160 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.544+0000 7f43527fc640 1 -- 172.21.15.1:0/793760431 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f434c019070 con 0x7f435c0d1160 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.544+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f435c1f5ed0 con 0x7f435c0d1160 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.544+0000 7f43527fc640 1 -- 172.21.15.1:0/793760431 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f434c00ba00 con 0x7f435c0d1160 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.544+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f435c1f6490 con 0x7f435c0d1160 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.545+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f435c050870 con 0x7f435c0d1160 2024-05-09T05:05:51.547 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.546+0000 7f43527fc640 1 -- 172.21.15.1:0/793760431 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 4) ==== 98145+0+0 (secure 0 0 0) 0x7f434c00bba0 con 0x7f435c0d1160 2024-05-09T05:05:51.548 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.546+0000 7f43527fc640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4340076240 0x7f4340078750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:51.548 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.546+0000 7f43527fc640 1 -- 172.21.15.1:0/793760431 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 3911+0+0 (secure 0 0 0) 0x7f434c097130 con 0x7f435c0d1160 2024-05-09T05:05:51.548 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.546+0000 7f4361f29640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4340076240 0x7f4340078750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:51.548 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.547+0000 7f4361f29640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4340076240 0x7f4340078750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4358007fd0 tx=0x7f4358007de0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:51.552 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.551+0000 7f43527fc640 1 -- 172.21.15.1:0/793760431 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f434c060b90 con 0x7f435c0d1160 2024-05-09T05:05:51.747 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.747+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f435c0b5010 con 0x7f435c0d1160 2024-05-09T05:05:51.748 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.748+0000 7f43527fc640 1 -- 172.21.15.1:0/793760431 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v14) ==== 74+0+9889 (secure 0 0 0) 0x7f434c064e90 con 0x7f435c0d1160 2024-05-09T05:05:51.748 INFO:teuthology.misc.health.smithi001.stdout: 2024-05-09T05:05:51.748 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":14,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:05:50.557797+0000","last_up_change":"2024-05-09T05:05:50.557797+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:05:51.750 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.749+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4340076240 msgr2=0x7f4340078750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:51.750 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.749+0000 7f436418d640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4340076240 0x7f4340078750 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f4358007fd0 tx=0x7f4358007de0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.750 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.749+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 msgr2=0x7f435c1f27c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.749+0000 7f436418d640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c1f27c0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f434c00d620 tx=0x7f434c00dae0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.750+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 shutdown_connections 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.750+0000 7f436418d640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4340076240 0x7f4340078750 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.750+0000 7f436418d640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f435c1e93e0 0x7f435c0b4580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.750+0000 7f436418d640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f435c0d1160 0x7f435c1f27c0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.750+0000 7f436418d640 1 --2- 172.21.15.1:0/793760431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f435c0cf910 0x7f435c0b70b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.750+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 >> 172.21.15.1:0/793760431 conn(0x7f435c0b3620 msgr2=0x7f435c0c9150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.750+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 shutdown_connections 2024-05-09T05:05:51.751 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:51.750+0000 7f436418d640 1 -- 172.21.15.1:0/793760431 wait complete. 2024-05-09T05:05:51.760 DEBUG:teuthology.misc:6 of 8 OSDs are up 2024-05-09T05:05:58.568 DEBUG:teuthology.orchestra.run.smithi001:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2024-05-09T05:05:58.689 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.688+0000 7f0eea2bd640 1 Processor -- start 2024-05-09T05:05:58.689 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.688+0000 7f0eea2bd640 1 -- start start 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.689+0000 7f0eea2bd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.689+0000 7f0eea2bd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ee40c7d30 0x7f0ee40c8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.689+0000 7f0eea2bd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ee415d890 con 0x7f0ee4156ad0 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.689+0000 7f0eea2bd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f0ee40c9310 con 0x7f0ee40c7d30 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.689+0000 7f0eea2bd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee40c9560 con 0x7f0ee40c5e10 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee885a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ee40c7d30 0x7f0ee40c8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0ee4156ad0 0x7f0ee40c5220 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51596/0 (socket says 172.21.15.1:51596) 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee885a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0ee40c7d30 0x7f0ee40c8140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38036/0 (socket says 172.21.15.1:38036) 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee37fe640 1 -- 172.21.15.1:0/757351453 learned_addr learned my addr 172.21.15.1:0/757351453 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:05:58.691 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee3fff640 1 --2- 172.21.15.1:0/757351453 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:58.692 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee885a640 1 -- 172.21.15.1:0/757351453 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 msgr2=0x7f0ee415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:58.692 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee885a640 1 --2- 172.21.15.1:0/757351453 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.692 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee885a640 1 -- 172.21.15.1:0/757351453 >> v1:172.21.15.1:6789/0 conn(0x7f0ee4156ad0 legacy=0x7f0ee40c5220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:58.692 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee885a640 1 -- 172.21.15.1:0/757351453 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee40c9840 con 0x7f0ee40c7d30 2024-05-09T05:05:58.692 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.690+0000 7f0ee2ffd640 1 -- 172.21.15.1:0/757351453 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 229156371 0 0) 0x7f0ee415d890 con 0x7f0ee4156ad0 2024-05-09T05:05:58.693 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.691+0000 7f0ee885a640 1 --2- 172.21.15.1:0/757351453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee40c7d30 0x7f0ee40c8140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009d90 tx=0x7f0edc02a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7f17e8e1f94ff19 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:58.693 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.691+0000 7f0ee2ffd640 1 -- 172.21.15.1:0/757351453 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc007740 con 0x7f0ee40c7d30 2024-05-09T05:05:58.693 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.691+0000 7f0ee2ffd640 1 -- 172.21.15.1:0/757351453 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0edc0078e0 con 0x7f0ee40c7d30 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.691+0000 7f0ee2ffd640 1 -- 172.21.15.1:0/757351453 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc007cb0 con 0x7f0ee40c7d30 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.692+0000 7f0eea2bd640 1 -- 172.21.15.1:0/757351453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee40c7d30 msgr2=0x7f0ee40c8140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.692+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/757351453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee40c7d30 0x7f0ee40c8140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009d90 tx=0x7f0edc02a6f0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.692+0000 7f0eea2bd640 1 -- 172.21.15.1:0/757351453 shutdown_connections 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.692+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/757351453 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee40c7d30 0x7f0ee40c8140 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.692+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/757351453 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.692+0000 7f0eea2bd640 1 -- 172.21.15.1:0/757351453 >> 172.21.15.1:0/757351453 conn(0x7f0ee4074630 msgr2=0x7f0ee4071b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.692+0000 7f0eea2bd640 1 -- 172.21.15.1:0/757351453 shutdown_connections 2024-05-09T05:05:58.694 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.692+0000 7f0eea2bd640 1 -- 172.21.15.1:0/757351453 wait complete. 2024-05-09T05:05:58.695 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.694+0000 7f0eea2bd640 1 Processor -- start 2024-05-09T05:05:58.695 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.694+0000 7f0eea2bd640 1 -- start start 2024-05-09T05:05:58.697 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0eea2bd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:58.697 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0eea2bd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ee40c7d30 0x7f0ee40cee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:58.697 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0eea2bd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee4156ad0 0x7f0ee414e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0eea2bd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee40ca0e0 con 0x7f0ee4156ad0 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0eea2bd640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0ee40c9e00 con 0x7f0ee40c7d30 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0eea2bd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0ee40c8680 con 0x7f0ee40c5e10 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0ee885a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee4156ad0 0x7f0ee414e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0ee885a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee4156ad0 0x7f0ee414e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38040/0 (socket says 172.21.15.1:38040) 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0ee885a640 1 -- 172.21.15.1:0/2989566970 learned_addr learned my addr 172.21.15.1:0/2989566970 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0ee3fff640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0ee37fe640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ee40c7d30 0x7f0ee40cee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.695+0000 7f0ee885a640 1 -- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 msgr2=0x7f0ee415f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee885a640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee885a640 1 -- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ee40c7d30 msgr2=0x7f0ee40cee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee885a640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ee40c7d30 0x7f0ee40cee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee885a640 1 -- 172.21.15.1:0/2989566970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ee415fdb0 con 0x7f0ee4156ad0 2024-05-09T05:05:58.698 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee3fff640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:05:58.699 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee37fe640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ee40c7d30 0x7f0ee40cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:05:58.699 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee885a640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee4156ad0 0x7f0ee414e6e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009eb0 tx=0x7f0edc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:58.699 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee0ff9640 1 -- 172.21.15.1:0/2989566970 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc03f070 con 0x7f0ee4156ad0 2024-05-09T05:05:58.699 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee0ff9640 1 -- 172.21.15.1:0/2989566970 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0edc031bc0 con 0x7f0ee4156ad0 2024-05-09T05:05:58.699 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0ee0ff9640 1 -- 172.21.15.1:0/2989566970 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0edc044460 con 0x7f0ee4156ad0 2024-05-09T05:05:58.699 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ee41f2d30 con 0x7f0ee4156ad0 2024-05-09T05:05:58.699 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.696+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ee41f32f0 con 0x7f0ee4156ad0 2024-05-09T05:05:58.699 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.697+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ee4050870 con 0x7f0ee4156ad0 2024-05-09T05:05:58.700 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.698+0000 7f0ee0ff9640 1 -- 172.21.15.1:0/2989566970 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0edc007790 con 0x7f0ee4156ad0 2024-05-09T05:05:58.700 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.698+0000 7f0ee0ff9640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ed0076300 0x7f0ed0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:58.700 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.699+0000 7f0ee0ff9640 1 -- 172.21.15.1:0/2989566970 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f0edc0b8b50 con 0x7f0ee4156ad0 2024-05-09T05:05:58.700 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.699+0000 7f0ee3fff640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ed0076300 0x7f0ed0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:58.700 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.699+0000 7f0ee3fff640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ed0076300 0x7f0ed0078810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4006fd0 tx=0x7f0ed4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:58.704 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.704+0000 7f0ee0ff9640 1 -- 172.21.15.1:0/2989566970 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0edc0821b0 con 0x7f0ee4156ad0 2024-05-09T05:05:58.897 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.896+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f0ee40cb860 con 0x7f0ee4156ad0 2024-05-09T05:05:58.898 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.897+0000 7f0ee0ff9640 1 -- 172.21.15.1:0/2989566970 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) ==== 74+0+12824 (secure 0 0 0) 0x7f0edc0864b0 con 0x7f0ee4156ad0 2024-05-09T05:05:58.898 INFO:teuthology.misc.health.smithi001.stdout: 2024-05-09T05:05:58.898 INFO:teuthology.misc.health.smithi001.stdout:{"epoch":17,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:05:53.875555+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":15,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:05:58.900 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ed0076300 msgr2=0x7f0ed0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:58.900 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ed0076300 0x7f0ed0078810 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f0ed4006fd0 tx=0x7f0ed4008040 comp rx=0 tx=0).stop 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee4156ad0 msgr2=0x7f0ee414e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee4156ad0 0x7f0ee414e6e0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f0edc009eb0 tx=0x7f0edc004a40 comp rx=0 tx=0).stop 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 shutdown_connections 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ed0076300 0x7f0ed0078810 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ee40c7d30 0x7f0ee40cee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ee4156ad0 0x7f0ee414e6e0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 --2- 172.21.15.1:0/2989566970 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ee40c5e10 0x7f0ee415f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 >> 172.21.15.1:0/2989566970 conn(0x7f0ee4074630 msgr2=0x7f0ee415ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 shutdown_connections 2024-05-09T05:05:58.901 INFO:teuthology.misc.health.smithi001.stderr:2024-05-09T05:05:58.900+0000 7f0eea2bd640 1 -- 172.21.15.1:0/2989566970 wait complete. 2024-05-09T05:05:58.911 DEBUG:teuthology.misc:8 of 8 OSDs are up 2024-05-09T05:05:58.911 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2024-05-09T05:05:58.911 INFO:tasks.ceph.mds.a:Restarting daemon 2024-05-09T05:05:58.912 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2024-05-09T05:05:58.914 INFO:tasks.ceph.mds.a:Started 2024-05-09T05:05:58.915 INFO:tasks.ceph.mds.c:Restarting daemon 2024-05-09T05:05:58.915 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i c 2024-05-09T05:05:58.916 INFO:tasks.ceph.mds.c:Started 2024-05-09T05:05:58.917 INFO:tasks.ceph.mds.e:Restarting daemon 2024-05-09T05:05:58.917 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i e 2024-05-09T05:05:58.918 INFO:tasks.ceph.mds.e:Started 2024-05-09T05:05:58.919 INFO:tasks.ceph.mds.b:Restarting daemon 2024-05-09T05:05:58.919 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2024-05-09T05:05:58.921 INFO:tasks.ceph.mds.b:Started 2024-05-09T05:05:58.921 INFO:tasks.ceph.mds.d:Restarting daemon 2024-05-09T05:05:58.922 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i d 2024-05-09T05:05:58.924 INFO:tasks.ceph.mds.d:Started 2024-05-09T05:05:58.924 INFO:tasks.ceph.mds.f:Restarting daemon 2024-05-09T05:05:58.924 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i f 2024-05-09T05:05:58.926 INFO:tasks.ceph.mds.f:Started 2024-05-09T05:05:58.926 INFO:tasks.ceph:Setting up CephFS filesystem(s)... 2024-05-09T05:05:58.927 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:05:59.011 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-05-09T05:05:59.011 INFO:tasks.ceph.mds.a.smithi001.stdout:starting mds.a at 2024-05-09T05:05:59.015 INFO:tasks.ceph.mds.e.smithi001.stdout:starting mds.e at 2024-05-09T05:05:59.024 INFO:tasks.ceph.mds.b.smithi196.stdout:starting mds.b at 2024-05-09T05:05:59.025 INFO:tasks.ceph.mds.f.smithi196.stdout:starting mds.f at 2024-05-09T05:05:59.025 INFO:tasks.ceph.mds.d.smithi196.stdout:starting mds.d at 2024-05-09T05:05:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.053+0000 7f7d4c76c640 1 Processor -- start 2024-05-09T05:05:59.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.053+0000 7f7d4c76c640 1 -- start start 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d4c76c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7d44070530 0x7f7d44070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d4c76c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d440cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d4c76c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d440c4430 con 0x7f7d44159440 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d4c76c640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f7d440c3b90 con 0x7f7d44070530 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d4c76c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d440c3e70 con 0x7f7d44153830 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d49d07640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7d44070530 0x7f7d44070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d4ad09640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d440cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d49d07640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7d44070530 0x7f7d44070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:52098/0 (socket says 172.21.15.1:52098) 2024-05-09T05:05:59.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d4ad09640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d440cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:49664/0 (socket says 172.21.15.1:49664) 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.055+0000 7f7d49d07640 1 -- 172.21.15.1:0/1683710691 learned_addr learned my addr 172.21.15.1:0/1683710691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.056+0000 7f7d49d07640 1 -- 172.21.15.1:0/1683710691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 msgr2=0x7f7d440cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.056+0000 7f7d49d07640 1 --2- 172.21.15.1:0/1683710691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d440cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.056+0000 7f7d49d07640 1 -- 172.21.15.1:0/1683710691 >> v1:172.21.15.196:6789/0 conn(0x7f7d44159440 legacy=0x7f7d4415d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.056+0000 7f7d49506640 1 -- 172.21.15.1:0/1683710691 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1250323420 0 0) 0x7f7d440c4430 con 0x7f7d44159440 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.056+0000 7f7d49d07640 1 -- 172.21.15.1:0/1683710691 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d440c4150 con 0x7f7d44070530 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.056+0000 7f7d49d07640 1 --2- 172.21.15.1:0/1683710691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 0x7f7d44070ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d3800dc80 tx=0x7f7d3802e5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=161d80ae7d60d647 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d49506640 1 -- 172.21.15.1:0/1683710691 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3803a070 con 0x7f7d44070530 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d49506640 1 -- 172.21.15.1:0/1683710691 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d38035600 con 0x7f7d44070530 2024-05-09T05:05:59.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d49506640 1 -- 172.21.15.1:0/1683710691 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d380359d0 con 0x7f7d44070530 2024-05-09T05:05:59.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1683710691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 msgr2=0x7f7d44070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:59.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1683710691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 0x7f7d44070ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7d3800dc80 tx=0x7f7d3802e5d0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1683710691 shutdown_connections 2024-05-09T05:05:59.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1683710691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d440cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1683710691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 0x7f7d44070ae0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1683710691 >> 172.21.15.1:0/1683710691 conn(0x7f7d44071b40 msgr2=0x7f7d44071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:59.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1683710691 shutdown_connections 2024-05-09T05:05:59.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.057+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1683710691 wait complete. 2024-05-09T05:05:59.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.059+0000 7f7d4c76c640 1 Processor -- start 2024-05-09T05:05:59.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.059+0000 7f7d4c76c640 1 -- start start 2024-05-09T05:05:59.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.060+0000 7f7d4c76c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 0x7f7d441f26e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:59.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.060+0000 7f7d4c76c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d4415c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:59.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.060+0000 7f7d4c76c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d44159440 0x7f7d441384c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:59.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.060+0000 7f7d4c76c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d440c3870 con 0x7f7d44159440 2024-05-09T05:05:59.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.060+0000 7f7d4c76c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7d4415ceb0 con 0x7f7d44070530 2024-05-09T05:05:59.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.060+0000 7f7d4c76c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d4406fad0 con 0x7f7d44153830 2024-05-09T05:05:59.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d44159440 0x7f7d441384c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d44159440 0x7f7d441384c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38086/0 (socket says 172.21.15.1:38086) 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 -- 172.21.15.1:0/1813804401 learned_addr learned my addr 172.21.15.1:0/1813804401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4a508640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 0x7f7d441f26e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d49d07640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d4415c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 -- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 msgr2=0x7f7d4415c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d4415c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 -- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 msgr2=0x7f7d441f26e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 0x7f7d441f26e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 -- 172.21.15.1:0/1813804401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d441ee690 con 0x7f7d44159440 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4a508640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 0x7f7d441f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4ad09640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d44159440 0x7f7d441384c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009cb0 tx=0x7f7d3c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d36ffd640 1 -- 172.21.15.1:0/1813804401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c018020 con 0x7f7d44159440 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d36ffd640 1 -- 172.21.15.1:0/1813804401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d3c014480 con 0x7f7d44159440 2024-05-09T05:05:59.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.061+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d440cdd40 con 0x7f7d44159440 2024-05-09T05:05:59.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.062+0000 7f7d36ffd640 1 -- 172.21.15.1:0/1813804401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d3c014870 con 0x7f7d44159440 2024-05-09T05:05:59.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.062+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d441f1c00 con 0x7f7d44159440 2024-05-09T05:05:59.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.063+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d4415b8d0 con 0x7f7d44159440 2024-05-09T05:05:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.063+0000 7f7d36ffd640 1 -- 172.21.15.1:0/1813804401 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7d3c01b020 con 0x7f7d44159440 2024-05-09T05:05:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.064+0000 7f7d36ffd640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d280762d0 0x7f7d280787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:05:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.064+0000 7f7d36ffd640 1 -- 172.21.15.1:0/1813804401 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7f7d3c097fb0 con 0x7f7d44159440 2024-05-09T05:05:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.064+0000 7f7d4a508640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d280762d0 0x7f7d280787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:05:59.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.064+0000 7f7d4a508640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d280762d0 0x7f7d280787e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7d40000f60 tx=0x7f7d40009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:05:59.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.068+0000 7f7d36ffd640 1 -- 172.21.15.1:0/1813804401 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7d4415b8d0 con 0x7f7d44159440 2024-05-09T05:05:59.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.301+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7d440c94c0 con 0x7f7d44159440 2024-05-09T05:05:59.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.301+0000 7f7d36ffd640 1 -- 172.21.15.1:0/1813804401 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1389 (secure 0 0 0) 0x7f7d3c022020 con 0x7f7d44159440 2024-05-09T05:05:59.302 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:05:59.302 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T05:05:59.302 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 2 2024-05-09T05:05:59.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d280762d0 msgr2=0x7f7d280787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:59.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d280762d0 0x7f7d280787e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f7d40000f60 tx=0x7f7d40009040 comp rx=0 tx=0).stop 2024-05-09T05:05:59.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d44159440 msgr2=0x7f7d441384c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:05:59.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d44159440 0x7f7d441384c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c009cb0 tx=0x7f7d3c00c490 comp rx=0 tx=0).stop 2024-05-09T05:05:59.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 shutdown_connections 2024-05-09T05:05:59.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d280762d0 0x7f7d280787e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d44159440 0x7f7d441384c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d44153830 0x7f7d4415c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 --2- 172.21.15.1:0/1813804401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d44070530 0x7f7d441f26e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:05:59.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.303+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 >> 172.21.15.1:0/1813804401 conn(0x7f7d44071b40 msgr2=0x7f7d44071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:05:59.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.304+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 shutdown_connections 2024-05-09T05:05:59.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:05:59.304+0000 7f7d4c76c640 1 -- 172.21.15.1:0/1813804401 wait complete. 2024-05-09T05:06:01.316 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:06:01.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.444+0000 7ffab3f27640 1 Processor -- start 2024-05-09T05:06:01.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.445+0000 7ffab3f27640 1 -- start start 2024-05-09T05:06:01.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab3f27640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0c6a30 0x7ffaac15d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab3f27640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffaac1568b0 0x7ffaac0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab3f27640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffaac0cd2e0 con 0x7ffaac0b4850 2024-05-09T05:06:01.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab3f27640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ffaac0cd090 con 0x7ffaac1568b0 2024-05-09T05:06:01.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab3f27640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffaac0cd5c0 con 0x7ffaac0c6a30 2024-05-09T05:06:01.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab24c4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffaac1568b0 0x7ffaac0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab14c2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffaac0b4850 0x7ffaac0d1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42328/0 (socket says 172.21.15.1:42328) 2024-05-09T05:06:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab14c2640 1 -- 172.21.15.1:0/1795776942 learned_addr learned my addr 172.21.15.1:0/1795776942 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.446+0000 7ffab1cc3640 1 --2- 172.21.15.1:0/1795776942 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0c6a30 0x7ffaac15d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.447+0000 7ffab24c4640 1 -- 172.21.15.1:0/1795776942 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0c6a30 msgr2=0x7ffaac15d0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.447+0000 7ffab24c4640 1 --2- 172.21.15.1:0/1795776942 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0c6a30 0x7ffaac15d0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.447+0000 7ffab24c4640 1 -- 172.21.15.1:0/1795776942 >> v1:172.21.15.1:6789/0 conn(0x7ffaac0b4850 legacy=0x7ffaac0d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.447+0000 7ffab24c4640 1 -- 172.21.15.1:0/1795776942 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac0cdeb0 con 0x7ffaac1568b0 2024-05-09T05:06:01.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.447+0000 7ffab0cc1640 1 -- 172.21.15.1:0/1795776942 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2517475922 0 0) 0x7ffaac0cd2e0 con 0x7ffaac0b4850 2024-05-09T05:06:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.447+0000 7ffab24c4640 1 --2- 172.21.15.1:0/1795776942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac0ce990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009d90 tx=0x7ffa9c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7492755524622963 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.447+0000 7ffab0cc1640 1 -- 172.21.15.1:0/1795776942 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c007740 con 0x7ffaac1568b0 2024-05-09T05:06:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab0cc1640 1 -- 172.21.15.1:0/1795776942 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9c0078e0 con 0x7ffaac1568b0 2024-05-09T05:06:01.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab0cc1640 1 -- 172.21.15.1:0/1795776942 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c007cb0 con 0x7ffaac1568b0 2024-05-09T05:06:01.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab3f27640 1 -- 172.21.15.1:0/1795776942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 msgr2=0x7ffaac0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab3f27640 1 --2- 172.21.15.1:0/1795776942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac0ce990 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009d90 tx=0x7ffa9c02a710 comp rx=0 tx=0).stop 2024-05-09T05:06:01.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab3f27640 1 -- 172.21.15.1:0/1795776942 shutdown_connections 2024-05-09T05:06:01.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab3f27640 1 --2- 172.21.15.1:0/1795776942 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac0ce990 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab3f27640 1 --2- 172.21.15.1:0/1795776942 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0c6a30 0x7ffaac15d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab3f27640 1 -- 172.21.15.1:0/1795776942 >> 172.21.15.1:0/1795776942 conn(0x7ffaac06dc00 msgr2=0x7ffaac06e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:01.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab3f27640 1 -- 172.21.15.1:0/1795776942 shutdown_connections 2024-05-09T05:06:01.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.448+0000 7ffab3f27640 1 -- 172.21.15.1:0/1795776942 wait complete. 2024-05-09T05:06:01.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.450+0000 7ffab3f27640 1 Processor -- start 2024-05-09T05:06:01.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.450+0000 7ffab3f27640 1 -- start start 2024-05-09T05:06:01.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.451+0000 7ffab3f27640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0b4850 0x7ffaac1f5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.451+0000 7ffab3f27640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffaac0c6a30 0x7ffaac1f6030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.451+0000 7ffab3f27640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac1f6570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.451+0000 7ffab3f27640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac156cc0 con 0x7ffaac1568b0 2024-05-09T05:06:01.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.451+0000 7ffab3f27640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffaac0ce470 con 0x7ffaac0c6a30 2024-05-09T05:06:01.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.451+0000 7ffab3f27640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffaac0d22d0 con 0x7ffaac0b4850 2024-05-09T05:06:01.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac1f6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac1f6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51236/0 (socket says 172.21.15.1:51236) 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 -- 172.21.15.1:0/58953388 learned_addr learned my addr 172.21.15.1:0/58953388 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab1cc3640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0b4850 0x7ffaac1f5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab14c2640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffaac0c6a30 0x7ffaac1f6030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 -- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0b4850 msgr2=0x7ffaac1f5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0b4850 0x7ffaac1f5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 -- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffaac0c6a30 msgr2=0x7ffaac1f6030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffaac0c6a30 0x7ffaac1f6030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 -- 172.21.15.1:0/58953388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffaac14e160 con 0x7ffaac1568b0 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab24c4640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac1f6570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009eb0 tx=0x7ffa9c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffaaa7fc640 1 -- 172.21.15.1:0/58953388 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c040030 con 0x7ffaac1568b0 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffaaa7fc640 1 -- 172.21.15.1:0/58953388 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa9c009410 con 0x7ffaac1568b0 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffaaa7fc640 1 -- 172.21.15.1:0/58953388 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa9c031c00 con 0x7ffaac1568b0 2024-05-09T05:06:01.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffaac14e440 con 0x7ffaac1568b0 2024-05-09T05:06:01.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.452+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffaac14ea00 con 0x7ffaac1568b0 2024-05-09T05:06:01.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.453+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffaac050870 con 0x7ffaac1568b0 2024-05-09T05:06:01.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.454+0000 7ffaaa7fc640 1 -- 172.21.15.1:0/58953388 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa9c03b410 con 0x7ffaac1568b0 2024-05-09T05:06:01.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.455+0000 7ffaaa7fc640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa980762d0 0x7ffa980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.455+0000 7ffaaa7fc640 1 -- 172.21.15.1:0/58953388 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7ffa9c0b8110 con 0x7ffaac1568b0 2024-05-09T05:06:01.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.456+0000 7ffab1cc3640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa980762d0 0x7ffa980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.456+0000 7ffab1cc3640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa980762d0 0x7ffa980787e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009a20 tx=0x7ffaa4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:01.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.460+0000 7ffaaa7fc640 1 -- 172.21.15.1:0/58953388 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffa9c081d30 con 0x7ffaac1568b0 2024-05-09T05:06:01.483 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:06:01.486 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:06:01.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.697+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffaac0cc1d0 con 0x7ffaac1568b0 2024-05-09T05:06:01.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.697+0000 7ffaaa7fc640 1 -- 172.21.15.1:0/58953388 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+5514 (secure 0 0 0) 0x7ffa9c081630 con 0x7ffaac1568b0 2024-05-09T05:06:01.699 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:01.699 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4273,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34157,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":34160,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T05:06:01.699 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 3 2024-05-09T05:06:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa980762d0 msgr2=0x7ffa980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa980762d0 0x7ffa980787e0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffaa4009a20 tx=0x7ffaa4008040 comp rx=0 tx=0).stop 2024-05-09T05:06:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 msgr2=0x7ffaac1f6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac1f6570 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ffa9c009eb0 tx=0x7ffa9c004a40 comp rx=0 tx=0).stop 2024-05-09T05:06:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 shutdown_connections 2024-05-09T05:06:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa980762d0 0x7ffa980787e0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffaac0c6a30 0x7ffaac1f6030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffaac1568b0 0x7ffaac1f6570 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 --2- 172.21.15.1:0/58953388 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffaac0b4850 0x7ffaac1f5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 >> 172.21.15.1:0/58953388 conn(0x7ffaac06dc00 msgr2=0x7ffaac154f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:01.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 shutdown_connections 2024-05-09T05:06:01.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.700+0000 7ffab3f27640 1 -- 172.21.15.1:0/58953388 wait complete. 2024-05-09T05:06:01.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T05:06:01.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.826+0000 7fc1cbc78640 1 Processor -- start 2024-05-09T05:06:01.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.826+0000 7fc1cbc78640 1 -- start start 2024-05-09T05:06:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1cbc78640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc1c406efa0 0x7fc1c4156900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1cbc78640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1c40cc970 0x7fc1c4160500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1cbc78640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1c406de20 con 0x7fc1c40cc560 2024-05-09T05:06:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1cbc78640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc1c415c9a0 con 0x7fc1c406efa0 2024-05-09T05:06:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1cbc78640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c40b51a0 con 0x7fc1c40cc970 2024-05-09T05:06:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c9213640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc1c40cc560 0x7fc1c415fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42334/0 (socket says 172.21.15.1:42334) 2024-05-09T05:06:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c9213640 1 -- 172.21.15.1:0/3652144483 learned_addr learned my addr 172.21.15.1:0/3652144483 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:01.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c9a14640 1 --2- 172.21.15.1:0/3652144483 >> v2:172.21.15.196:3300/0 conn(0x7fc1c406efa0 0x7fc1c4156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1ca215640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1c40cc970 0x7fc1c4160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 606907928 0 0) 0x7fc1c406de20 con 0x7fc1c40cc560 2024-05-09T05:06:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc1ac003520 con 0x7fc1c40cc560 2024-05-09T05:06:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4271890723 0 0) 0x7fc1ac003520 con 0x7fc1c40cc560 2024-05-09T05:06:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1c40cc970 msgr2=0x7fc1c4160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1c40cc970 0x7fc1c4160500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 >> v2:172.21.15.196:3300/0 conn(0x7fc1c406efa0 msgr2=0x7fc1c4156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:01.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 --2- 172.21.15.1:0/3652144483 >> v2:172.21.15.196:3300/0 conn(0x7fc1c406efa0 0x7fc1c4156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c415d720 con 0x7fc1c40cc560 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fc1b80016f0 con 0x7fc1c40cc560 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c9a14640 1 --2- 172.21.15.1:0/3652144483 >> v2:172.21.15.196:3300/0 conn(0x7fc1c406efa0 0x7fc1c4156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 >> v1:172.21.15.1:6789/0 conn(0x7fc1c40cc560 legacy=0x7fc1c415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.828+0000 7fc1c8a12640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac003a00 0x7fc1ac003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c8a12640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ac004800 0x7fc1ac004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c8a12640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1ac005150 0x7fc1ac00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c415c9a0 con 0x7fc1ac004800 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c40b51a0 con 0x7fc1ac005150 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc1ac00c010 con 0x7fc1ac003a00 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1ca215640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1c40cc970 0x7fc1c4160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:01.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c9213640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ac004800 0x7fc1ac004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c9a14640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac003a00 0x7fc1ac003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1ca215640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1ac005150 0x7fc1ac00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c9213640 1 -- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac003a00 msgr2=0x7fc1ac003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c9213640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac003a00 0x7fc1ac003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c9213640 1 -- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1ac005150 msgr2=0x7fc1ac00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c9213640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1ac005150 0x7fc1ac00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.829+0000 7fc1c9213640 1 -- 172.21.15.1:0/3652144483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1ac003520 con 0x7fc1ac004800 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.830+0000 7fc1c9213640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ac004800 0x7fc1ac004c10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc1c408fd00 tx=0x7fc1ac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f5f7a6c4d4040e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.830+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8007d50 con 0x7fc1ac004800 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.830+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1b8005af0 con 0x7fc1ac004800 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.830+0000 7fc1c8a12640 1 -- 172.21.15.1:0/3652144483 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b8007070 con 0x7fc1ac004800 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.830+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3652144483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ac004800 msgr2=0x7fc1ac004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.830+0000 7fc1cbc78640 1 --2- 172.21.15.1:0/3652144483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1ac004800 0x7fc1ac004c10 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc1c408fd00 tx=0x7fc1ac005830 comp rx=0 tx=0).stop 2024-05-09T05:06:01.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.831+0000 7fc1ca215640 1 -- 172.21.15.1:0/3652144483 reap_dead start 2024-05-09T05:06:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.831+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3652144483 shutdown_connections 2024-05-09T05:06:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.831+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3652144483 >> 172.21.15.1:0/3652144483 conn(0x7fc1c4068d90 msgr2=0x7fc1c40d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.831+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3652144483 shutdown_connections 2024-05-09T05:06:01.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.831+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3652144483 wait complete. 2024-05-09T05:06:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.832+0000 7fc1cbc78640 1 Processor -- start 2024-05-09T05:06:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.832+0000 7fc1cbc78640 1 -- start start 2024-05-09T05:06:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1cbc78640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac004800 0x7fc1c4157370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1cbc78640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1c4159a70 0x7fc1c40d0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1cbc78640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c4151810 0x7fc1c4151c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1cbc78640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c415cf20 con 0x7fc1c4151810 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1cbc78640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc1c415cc40 con 0x7fc1c4159a70 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1cbc78640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc1c415d200 con 0x7fc1ac004800 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c4151810 0x7fc1c4151c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c4151810 0x7fc1c4151c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51252/0 (socket says 172.21.15.1:51252) 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 -- 172.21.15.1:0/3726282431 learned_addr learned my addr 172.21.15.1:0/3726282431 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1c9213640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1c4159a70 0x7fc1c40d0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1c9a14640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac004800 0x7fc1c4157370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 -- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac004800 msgr2=0x7fc1c4157370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac004800 0x7fc1c4157370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 -- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1c4159a70 msgr2=0x7fc1c40d0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1c4159a70 0x7fc1c40d0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 -- 172.21.15.1:0/3726282431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1c41f3330 con 0x7fc1c4151810 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1ca215640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c4151810 0x7fc1c4151c20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc1c408ef60 tx=0x7fc1b400f870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:01.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1c27fc640 1 -- 172.21.15.1:0/3726282431 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b401e070 con 0x7fc1c4151810 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1c27fc640 1 -- 172.21.15.1:0/3726282431 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc1b4003990 con 0x7fc1c4151810 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1c27fc640 1 -- 172.21.15.1:0/3726282431 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1b4003d80 con 0x7fc1c4151810 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1c41f3610 con 0x7fc1c4151810 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.834+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1c41f3bd0 con 0x7fc1c4151810 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.835+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1c40b68b0 con 0x7fc1c4151810 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.836+0000 7fc1c27fc640 1 -- 172.21.15.1:0/3726282431 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc1b4021070 con 0x7fc1c4151810 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.837+0000 7fc1c27fc640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc1b0076300 0x7fc1b0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.837+0000 7fc1c27fc640 1 -- 172.21.15.1:0/3726282431 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fc1b4097340 con 0x7fc1c4151810 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.837+0000 7fc1c9a14640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc1b0076300 0x7fc1b0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:01.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.838+0000 7fc1c9a14640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc1b0076300 0x7fc1b0078810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009c10 tx=0x7fc1bc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:01.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:01.842+0000 7fc1c27fc640 1 -- 172.21.15.1:0/3726282431 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc1c40b68b0 con 0x7fc1c4151810 2024-05-09T05:06:02.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.034+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fc1c40c8cd0 con 0x7fc1c4151810 2024-05-09T05:06:02.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.034+0000 7fc1c27fc640 1 -- 172.21.15.1:0/3726282431 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v17) ==== 84+0+65 (secure 0 0 0) 0x7fc1b4010040 con 0x7fc1c4151810 2024-05-09T05:06:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:02.035 INFO:teuthology.orchestra.run.smithi001.stdout:[ 2024-05-09T05:06:02.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "poolnum": 1, 2024-05-09T05:06:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: "poolname": ".mgr" 2024-05-09T05:06:02.036 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:02.036 INFO:teuthology.orchestra.run.smithi001.stdout:] 2024-05-09T05:06:02.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc1b0076300 msgr2=0x7fc1b0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc1b0076300 0x7fc1b0078810 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc1bc009c10 tx=0x7fc1bc008040 comp rx=0 tx=0).stop 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c4151810 msgr2=0x7fc1c4151c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c4151810 0x7fc1c4151c20 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fc1c408ef60 tx=0x7fc1b400f870 comp rx=0 tx=0).stop 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 shutdown_connections 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc1b0076300 0x7fc1b0078810 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1c4159a70 0x7fc1c40d0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc1c4151810 0x7fc1c4151c20 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 --2- 172.21.15.1:0/3726282431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1ac004800 0x7fc1c4157370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 >> 172.21.15.1:0/3726282431 conn(0x7fc1c4068d90 msgr2=0x7fc1c414a2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 shutdown_connections 2024-05-09T05:06:02.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.037+0000 7fc1cbc78640 1 -- 172.21.15.1:0/3726282431 wait complete. 2024-05-09T05:06:02.048 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T05:06:02.048 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T05:06:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.166+0000 7fd9fedee640 1 Processor -- start 2024-05-09T05:06:02.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.166+0000 7fd9fedee640 1 -- start start 2024-05-09T05:06:02.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9fedee640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9f80b4570 0x7fd9f8156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9fedee640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd9f815c9a0 0x7fd9f815cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9fedee640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd9f80ce5c0 con 0x7fd9f80cf360 2024-05-09T05:06:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9fedee640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f815d4c0 con 0x7fd9f80b4570 2024-05-09T05:06:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9fedee640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd9f815d7a0 con 0x7fd9f815c9a0 2024-05-09T05:06:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9f7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9f80b4570 0x7fd9f8156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:02.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9f7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd9f80b4570 0x7fd9f8156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51264/0 (socket says 172.21.15.1:51264) 2024-05-09T05:06:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9f7fff640 1 -- 172.21.15.1:0/1036705463 learned_addr learned my addr 172.21.15.1:0/1036705463 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9fd38b640 1 --2- 172.21.15.1:0/1036705463 >> v2:172.21.15.196:3300/0 conn(0x7fd9f815c9a0 0x7fd9f815cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9f7fff640 1 -- 172.21.15.1:0/1036705463 >> v2:172.21.15.196:3300/0 conn(0x7fd9f815c9a0 msgr2=0x7fd9f815cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9f7fff640 1 --2- 172.21.15.1:0/1036705463 >> v2:172.21.15.196:3300/0 conn(0x7fd9f815c9a0 0x7fd9f815cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9f7fff640 1 -- 172.21.15.1:0/1036705463 >> v1:172.21.15.196:6789/0 conn(0x7fd9f80cf360 legacy=0x7fd9f80b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:02.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9f7fff640 1 -- 172.21.15.1:0/1036705463 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f80ce880 con 0x7fd9f80b4570 2024-05-09T05:06:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.168+0000 7fd9fd38b640 1 --2- 172.21.15.1:0/1036705463 >> v2:172.21.15.196:3300/0 conn(0x7fd9f815c9a0 0x7fd9f815cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.169+0000 7fd9f7fff640 1 --2- 172.21.15.1:0/1036705463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f80b4570 0x7fd9f8156b50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd9f806eb70 tx=0x7fd9ec02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=604b4a63dcd61647 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.169+0000 7fd9f77fe640 1 -- 172.21.15.1:0/1036705463 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec037070 con 0x7fd9f80b4570 2024-05-09T05:06:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.169+0000 7fd9f77fe640 1 -- 172.21.15.1:0/1036705463 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ec02ae50 con 0x7fd9f80b4570 2024-05-09T05:06:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.169+0000 7fd9f77fe640 1 -- 172.21.15.1:0/1036705463 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec032860 con 0x7fd9f80b4570 2024-05-09T05:06:02.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.170+0000 7fd9fedee640 1 -- 172.21.15.1:0/1036705463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f80b4570 msgr2=0x7fd9f8156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.170+0000 7fd9fedee640 1 --2- 172.21.15.1:0/1036705463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f80b4570 0x7fd9f8156b50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd9f806eb70 tx=0x7fd9ec02a300 comp rx=0 tx=0).stop 2024-05-09T05:06:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.170+0000 7fd9fedee640 1 -- 172.21.15.1:0/1036705463 shutdown_connections 2024-05-09T05:06:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.170+0000 7fd9fedee640 1 --2- 172.21.15.1:0/1036705463 >> v2:172.21.15.196:3300/0 conn(0x7fd9f815c9a0 0x7fd9f815cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.170+0000 7fd9fedee640 1 --2- 172.21.15.1:0/1036705463 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f80b4570 0x7fd9f8156b50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.170+0000 7fd9fedee640 1 -- 172.21.15.1:0/1036705463 >> 172.21.15.1:0/1036705463 conn(0x7fd9f806de10 msgr2=0x7fd9f806e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:02.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.170+0000 7fd9fedee640 1 -- 172.21.15.1:0/1036705463 shutdown_connections 2024-05-09T05:06:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.170+0000 7fd9fedee640 1 -- 172.21.15.1:0/1036705463 wait complete. 2024-05-09T05:06:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.171+0000 7fd9fedee640 1 Processor -- start 2024-05-09T05:06:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.171+0000 7fd9fedee640 1 -- start start 2024-05-09T05:06:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9fedee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f80b4570 0x7fd9f81e4f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9fedee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f815c9a0 0x7fd9f81f6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:02.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9fedee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9f81e9220 0x7fd9f815b7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9fedee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f8157090 con 0x7fd9f815c9a0 2024-05-09T05:06:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9fedee640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd9f80cee40 con 0x7fd9f80b4570 2024-05-09T05:06:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9fedee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd9f80cd3f0 con 0x7fd9f81e9220 2024-05-09T05:06:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f815c9a0 0x7fd9f81f6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f815c9a0 0x7fd9f81f6580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51268/0 (socket says 172.21.15.1:51268) 2024-05-09T05:06:02.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 -- 172.21.15.1:0/171385423 learned_addr learned my addr 172.21.15.1:0/171385423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9fcb8a640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f80b4570 0x7fd9f81e4f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9fd38b640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9f81e9220 0x7fd9f815b7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 -- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9f81e9220 msgr2=0x7fd9f815b7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9f81e9220 0x7fd9f815b7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 -- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f80b4570 msgr2=0x7fd9f81e4f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:02.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f80b4570 0x7fd9f81e4f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 -- 172.21.15.1:0/171385423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9f81e4770 con 0x7fd9f815c9a0 2024-05-09T05:06:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.173+0000 7fd9f7fff640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f815c9a0 0x7fd9f81f6580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec000c00 tx=0x7fd9ec0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.174+0000 7fd9f57fa640 1 -- 172.21.15.1:0/171385423 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec041020 con 0x7fd9f815c9a0 2024-05-09T05:06:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.174+0000 7fd9f57fa640 1 -- 172.21.15.1:0/171385423 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd9ec03c870 con 0x7fd9f815c9a0 2024-05-09T05:06:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.174+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9f81e5440 con 0x7fd9f815c9a0 2024-05-09T05:06:02.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.174+0000 7fd9f57fa640 1 -- 172.21.15.1:0/171385423 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd9ec037040 con 0x7fd9f815c9a0 2024-05-09T05:06:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.174+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9f80b3d00 con 0x7fd9f815c9a0 2024-05-09T05:06:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.175+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd9f81e9840 con 0x7fd9f815c9a0 2024-05-09T05:06:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.175+0000 7fd9f57fa640 1 -- 172.21.15.1:0/171385423 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd9ec044020 con 0x7fd9f815c9a0 2024-05-09T05:06:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.176+0000 7fd9f57fa640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd9e40762d0 0x7fd9e40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.176+0000 7fd9f57fa640 1 -- 172.21.15.1:0/171385423 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(17..17 src has 1..17) ==== 4857+0+0 (secure 0 0 0) 0x7fd9ec0b78e0 con 0x7fd9f815c9a0 2024-05-09T05:06:02.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.176+0000 7fd9fcb8a640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd9e40762d0 0x7fd9e40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.177+0000 7fd9fcb8a640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd9e40762d0 0x7fd9e40787e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8000f60 tx=0x7fd9e8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:02.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.181+0000 7fd9f57fa640 1 -- 172.21.15.1:0/171385423 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd9ec04b070 con 0x7fd9f815c9a0 2024-05-09T05:06:02.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:02.371+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fd9f80d0450 con 0x7fd9f815c9a0 2024-05-09T05:06:03.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.033+0000 7fd9f57fa640 1 -- 172.21.15.1:0/171385423 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v18) ==== 138+0+0 (secure 0 0 0) 0x7fd9f80d0450 con 0x7fd9f815c9a0 2024-05-09T05:06:03.034 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' created 2024-05-09T05:06:03.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.036+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd9e40762d0 msgr2=0x7fd9e40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:03.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.036+0000 7fd9fedee640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd9e40762d0 0x7fd9e40787e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd9e8000f60 tx=0x7fd9e8009040 comp rx=0 tx=0).stop 2024-05-09T05:06:03.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.036+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f815c9a0 msgr2=0x7fd9f81f6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:03.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.036+0000 7fd9fedee640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f815c9a0 0x7fd9f81f6580 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd9ec000c00 tx=0x7fd9ec0046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:03.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.037+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 shutdown_connections 2024-05-09T05:06:03.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.037+0000 7fd9fedee640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd9e40762d0 0x7fd9e40787e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:03.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.037+0000 7fd9fedee640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9f81e9220 0x7fd9f815b7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:03.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.037+0000 7fd9fedee640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9f815c9a0 0x7fd9f81f6580 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:03.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.037+0000 7fd9fedee640 1 --2- 172.21.15.1:0/171385423 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9f80b4570 0x7fd9f81e4f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:03.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.037+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 >> 172.21.15.1:0/171385423 conn(0x7fd9f806de10 msgr2=0x7fd9f80d0850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:03.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.038+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 shutdown_connections 2024-05-09T05:06:03.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.038+0000 7fd9fedee640 1 -- 172.21.15.1:0/171385423 wait complete. 2024-05-09T05:06:03.056 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T05:06:03.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.183+0000 7fb1ca205640 1 Processor -- start 2024-05-09T05:06:03.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.183+0000 7fb1ca205640 1 -- start start 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1ca205640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb1c40cc9d0 0x7fb1c406ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1ca205640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c40b5290 con 0x7fb1c4156bf0 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1ca205640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1c415ff50 con 0x7fb1c40cc5c0 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1ca205640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb1c40c8020 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb1c40cc9d0 0x7fb1c406ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1c37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb1c4156bf0 0x7fb1c406d700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42336/0 (socket says 172.21.15.1:42336) 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb1c40cc9d0 0x7fb1c406ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51270/0 (socket says 172.21.15.1:51270) 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1c37fe640 1 -- 172.21.15.1:0/1819895452 learned_addr learned my addr 172.21.15.1:0/1819895452 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1c3fff640 1 -- 172.21.15.1:0/1819895452 >> v1:172.21.15.196:6789/0 conn(0x7fb1c40cc5c0 legacy=0x7fb1c406e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1c3fff640 1 -- 172.21.15.1:0/1819895452 >> v1:172.21.15.1:6789/0 conn(0x7fb1c4156bf0 legacy=0x7fb1c406d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1c3fff640 1 -- 172.21.15.1:0/1819895452 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c40c82e0 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.185+0000 7fb1c27fc640 1 -- 172.21.15.1:0/1819895452 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 940339284 0 0) 0x7fb1c40b5290 con 0x7fb1c4156bf0 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.186+0000 7fb1c3fff640 1 --2- 172.21.15.1:0/1819895452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c406ef50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009d90 tx=0x7fb1bc02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71de8172c691014a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.186+0000 7fb1c27fc640 1 -- 172.21.15.1:0/1819895452 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc007740 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.186+0000 7fb1c27fc640 1 -- 172.21.15.1:0/1819895452 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1bc0078e0 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.186+0000 7fb1c27fc640 1 -- 172.21.15.1:0/1819895452 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1bc007cb0 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.187+0000 7fb1ca205640 1 -- 172.21.15.1:0/1819895452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 msgr2=0x7fb1c406ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.187+0000 7fb1ca205640 1 --2- 172.21.15.1:0/1819895452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c406ef50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb1bc009d90 tx=0x7fb1bc02a660 comp rx=0 tx=0).stop 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.187+0000 7fb1ca205640 1 -- 172.21.15.1:0/1819895452 shutdown_connections 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.187+0000 7fb1ca205640 1 --2- 172.21.15.1:0/1819895452 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c406ef50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.187+0000 7fb1ca205640 1 -- 172.21.15.1:0/1819895452 >> 172.21.15.1:0/1819895452 conn(0x7fb1c4068db0 msgr2=0x7fb1c40d1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.187+0000 7fb1ca205640 1 -- 172.21.15.1:0/1819895452 shutdown_connections 2024-05-09T05:06:03.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.187+0000 7fb1ca205640 1 -- 172.21.15.1:0/1819895452 wait complete. 2024-05-09T05:06:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.189+0000 7fb1ca205640 1 Processor -- start 2024-05-09T05:06:03.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.189+0000 7fb1ca205640 1 -- start start 2024-05-09T05:06:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1ca205640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1c40cc5c0 0x7fb1c41f3630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1ca205640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c415baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1ca205640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1c4156bf0 0x7fb1c415a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1ca205640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41604d0 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1ca205640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb1c41601f0 con 0x7fb1c40cc5c0 2024-05-09T05:06:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1ca205640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb1c41607b0 con 0x7fb1c4156bf0 2024-05-09T05:06:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c415baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:03.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1c2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c415baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51274/0 (socket says 172.21.15.1:51274) 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1c2ffd640 1 -- 172.21.15.1:0/3179502887 learned_addr learned my addr 172.21.15.1:0/3179502887 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1c37fe640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1c40cc5c0 0x7fb1c41f3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1c3fff640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1c4156bf0 0x7fb1c415a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1c2ffd640 1 -- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1c4156bf0 msgr2=0x7fb1c415a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.190+0000 7fb1c2ffd640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1c4156bf0 0x7fb1c415a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1c2ffd640 1 -- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1c40cc5c0 msgr2=0x7fb1c41f3630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1c2ffd640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1c40cc5c0 0x7fb1c41f3630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1c2ffd640 1 -- 172.21.15.1:0/3179502887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1c415bfe0 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1c37fe640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1c40cc5c0 0x7fb1c41f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1c3fff640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1c4156bf0 0x7fb1c415a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1c2ffd640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c415baa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800b3c0 tx=0x7fb1b800b880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1abfff640 1 -- 172.21.15.1:0/3179502887 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8011830 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1abfff640 1 -- 172.21.15.1:0/3179502887 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b8016070 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1c415cbb0 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1abfff640 1 -- 172.21.15.1:0/3179502887 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b8011c50 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.191+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1c415ce70 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.192+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1c41f3b70 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.193+0000 7fb1abfff640 1 -- 172.21.15.1:0/3179502887 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb1b8004020 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.193+0000 7fb1abfff640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb1b0076300 0x7fb1b0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.194+0000 7fb1abfff640 1 -- 172.21.15.1:0/3179502887 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(18..18 src has 1..18) ==== 5227+0+0 (secure 0 0 0) 0x7fb1b8097d40 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.194+0000 7fb1c37fe640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb1b0076300 0x7fb1b0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:03.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.194+0000 7fb1c37fe640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb1b0076300 0x7fb1b0078810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4000f60 tx=0x7fb1b4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:03.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.198+0000 7fb1abfff640 1 -- 172.21.15.1:0/3179502887 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb1c41f3b70 con 0x7fb1c40cc9d0 2024-05-09T05:06:03.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:03.388+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fb1c4050870 con 0x7fb1c40cc9d0 2024-05-09T05:06:04.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.038+0000 7fb1abfff640 1 -- 172.21.15.1:0/3179502887 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v19) ==== 170+0+0 (secure 0 0 0) 0x7fb1b8061750 con 0x7fb1c40cc9d0 2024-05-09T05:06:04.039 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' created 2024-05-09T05:06:04.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb1b0076300 msgr2=0x7fb1b0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:04.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb1b0076300 0x7fb1b0078810 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb1b4000f60 tx=0x7fb1b4009040 comp rx=0 tx=0).stop 2024-05-09T05:06:04.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 msgr2=0x7fb1c415baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:04.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c415baa0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800b3c0 tx=0x7fb1b800b880 comp rx=0 tx=0).stop 2024-05-09T05:06:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 shutdown_connections 2024-05-09T05:06:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb1b0076300 0x7fb1b0078810 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1c4156bf0 0x7fb1c415a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1c40cc9d0 0x7fb1c415baa0 secure :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1b800b3c0 tx=0x7fb1b800b880 comp rx=0 tx=0).stop 2024-05-09T05:06:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 --2- 172.21.15.1:0/3179502887 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1c40cc5c0 0x7fb1c41f3630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.043+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 >> 172.21.15.1:0/3179502887 conn(0x7fb1c4068db0 msgr2=0x7fb1c406e480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.044+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 shutdown_connections 2024-05-09T05:06:04.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.044+0000 7fb1ca205640 1 -- 172.21.15.1:0/3179502887 wait complete. 2024-05-09T05:06:04.074 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T05:06:04.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.195+0000 7fc41ecd9640 1 Processor -- start 2024-05-09T05:06:04.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.195+0000 7fc41ecd9640 1 -- start start 2024-05-09T05:06:04.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc41ecd9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c49e0 0x7fc4181549c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:04.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc41ecd9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc4180c5190 0x7fc41815dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc41ecd9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4180b2860 con 0x7fc4180cbde0 2024-05-09T05:06:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc41ecd9640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc4180c6970 con 0x7fc4180c5190 2024-05-09T05:06:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc41ecd9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc4180c57b0 con 0x7fc4180c49e0 2024-05-09T05:06:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc417fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c49e0 0x7fc4181549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc41d276640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc4180c5190 0x7fc41815dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc417fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c49e0 0x7fc4181549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60396/0 (socket says 172.21.15.1:60396) 2024-05-09T05:06:04.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc417fff640 1 -- 172.21.15.1:0/4171517589 learned_addr learned my addr 172.21.15.1:0/4171517589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc417fff640 1 -- 172.21.15.1:0/4171517589 >> v2:172.21.15.196:3300/0 conn(0x7fc4180c5190 msgr2=0x7fc41815dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc417fff640 1 --2- 172.21.15.1:0/4171517589 >> v2:172.21.15.196:3300/0 conn(0x7fc4180c5190 0x7fc41815dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc417fff640 1 -- 172.21.15.1:0/4171517589 >> v1:172.21.15.196:6789/0 conn(0x7fc4180cbde0 legacy=0x7fc4180cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc417fff640 1 -- 172.21.15.1:0/4171517589 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4180c5d60 con 0x7fc4180c49e0 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.197+0000 7fc41d276640 1 --2- 172.21.15.1:0/4171517589 >> v2:172.21.15.196:3300/0 conn(0x7fc4180c5190 0x7fc41815dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.198+0000 7fc417fff640 1 --2- 172.21.15.1:0/4171517589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c49e0 0x7fc4181549c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc4180c6af0 tx=0x7fc40002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=888ae21a89db42c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.198+0000 7fc4177fe640 1 -- 172.21.15.1:0/4171517589 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc400036070 con 0x7fc4180c49e0 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.198+0000 7fc4177fe640 1 -- 172.21.15.1:0/4171517589 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc400031600 con 0x7fc4180c49e0 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.198+0000 7fc4177fe640 1 -- 172.21.15.1:0/4171517589 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4000319d0 con 0x7fc4180c49e0 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.199+0000 7fc41ecd9640 1 -- 172.21.15.1:0/4171517589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c49e0 msgr2=0x7fc4181549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.199+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/4171517589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c49e0 0x7fc4181549c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc4180c6af0 tx=0x7fc40002a710 comp rx=0 tx=0).stop 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.199+0000 7fc41ecd9640 1 -- 172.21.15.1:0/4171517589 shutdown_connections 2024-05-09T05:06:04.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.199+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/4171517589 >> v2:172.21.15.196:3300/0 conn(0x7fc4180c5190 0x7fc41815dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.199+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/4171517589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c49e0 0x7fc4181549c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.199+0000 7fc41ecd9640 1 -- 172.21.15.1:0/4171517589 >> 172.21.15.1:0/4171517589 conn(0x7fc418068830 msgr2=0x7fc418068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.199+0000 7fc41ecd9640 1 -- 172.21.15.1:0/4171517589 shutdown_connections 2024-05-09T05:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.199+0000 7fc41ecd9640 1 -- 172.21.15.1:0/4171517589 wait complete. 2024-05-09T05:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.201+0000 7fc41ecd9640 1 Processor -- start 2024-05-09T05:06:04.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.201+0000 7fc41ecd9640 1 -- start start 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ecd9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4180c49e0 0x7fc4181567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ecd9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4180c5190 0x7fc418072c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ecd9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c9ae0 0x7fc4181f41c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ecd9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc4180c6600 con 0x7fc4180c49e0 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ecd9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc4180c6320 con 0x7fc4180c5190 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ecd9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc4180cb770 con 0x7fc4180c9ae0 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ca75640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4180c49e0 0x7fc4181567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ca75640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4180c49e0 0x7fc4181567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51278/0 (socket says 172.21.15.1:51278) 2024-05-09T05:06:04.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.202+0000 7fc41ca75640 1 -- 172.21.15.1:0/3841650765 learned_addr learned my addr 172.21.15.1:0/3841650765 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc417fff640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4180c5190 0x7fc418072c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41d276640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c9ae0 0x7fc4181f41c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41ca75640 1 -- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c9ae0 msgr2=0x7fc4181f41c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41ca75640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c9ae0 0x7fc4181f41c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41ca75640 1 -- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4180c5190 msgr2=0x7fc418072c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41ca75640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4180c5190 0x7fc418072c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41ca75640 1 -- 172.21.15.1:0/3841650765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc418156d00 con 0x7fc4180c49e0 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41d276640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c9ae0 0x7fc4181f41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41ca75640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4180c49e0 0x7fc4181567c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc40800c430 tx=0x7fc40800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc4157fa640 1 -- 172.21.15.1:0/3841650765 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408017070 con 0x7fc4180c49e0 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc4157fa640 1 -- 172.21.15.1:0/3841650765 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4080052d0 con 0x7fc4180c49e0 2024-05-09T05:06:04.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4181562e0 con 0x7fc4180c49e0 2024-05-09T05:06:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc4157fa640 1 -- 172.21.15.1:0/3841650765 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc408003830 con 0x7fc4180c49e0 2024-05-09T05:06:04.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.203+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4181555c0 con 0x7fc4180c49e0 2024-05-09T05:06:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.204+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc418156ed0 con 0x7fc4180c49e0 2024-05-09T05:06:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.205+0000 7fc4157fa640 1 -- 172.21.15.1:0/3841650765 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc40801a070 con 0x7fc4180c49e0 2024-05-09T05:06:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.206+0000 7fc4157fa640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3f8076300 0x7fc3f8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.206+0000 7fc4157fa640 1 -- 172.21.15.1:0/3841650765 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(19..19 src has 1..19) ==== 5609+0+0 (secure 0 0 0) 0x7fc408096e70 con 0x7fc4180c49e0 2024-05-09T05:06:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.206+0000 7fc417fff640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3f8076300 0x7fc3f8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:04.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.206+0000 7fc417fff640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3f8076300 0x7fc3f8078810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc400033000 tx=0x7fc400009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:04.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.210+0000 7fc4157fa640 1 -- 172.21.15.1:0/3841650765 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc408021020 con 0x7fc4180c49e0 2024-05-09T05:06:04.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:04.441+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7fc418050870 con 0x7fc4180c49e0 2024-05-09T05:06:05.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.040+0000 7fc4157fa640 1 -- 172.21.15.1:0/3841650765 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v4) ==== 172+0+0 (secure 0 0 0) 0x7fc408060780 con 0x7fc4180c49e0 2024-05-09T05:06:05.041 INFO:teuthology.orchestra.run.smithi001.stderr:new fs with metadata pool 2 and data pool 3 2024-05-09T05:06:05.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.044+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3f8076300 msgr2=0x7fc3f8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.044+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3f8076300 0x7fc3f8078810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc400033000 tx=0x7fc400009000 comp rx=0 tx=0).stop 2024-05-09T05:06:05.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.044+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4180c49e0 msgr2=0x7fc4181567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.044+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4180c49e0 0x7fc4181567c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc40800c430 tx=0x7fc40800c8f0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.045+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 shutdown_connections 2024-05-09T05:06:05.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.045+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3f8076300 0x7fc3f8078810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.045+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4180c5190 0x7fc418072c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.045+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4180c9ae0 0x7fc4181f41c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.045+0000 7fc41ecd9640 1 --2- 172.21.15.1:0/3841650765 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4180c49e0 0x7fc4181567c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.045+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 >> 172.21.15.1:0/3841650765 conn(0x7fc418068830 msgr2=0x7fc4180b5620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:05.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.045+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 shutdown_connections 2024-05-09T05:06:05.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.046+0000 7fc41ecd9640 1 -- 172.21.15.1:0/3841650765 wait complete. 2024-05-09T05:06:05.076 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:06:05.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.196+0000 7fa29e049640 1 Processor -- start 2024-05-09T05:06:05.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.196+0000 7fa29e049640 1 -- start start 2024-05-09T05:06:05.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa29e049640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2980c6a30 0x7fa29815d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa29e049640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa2980b4850 0x7fa2981568b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa29e049640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa2980cd2e0 con 0x7fa2980d1bc0 2024-05-09T05:06:05.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa29e049640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa2980cd090 con 0x7fa2980c6a30 2024-05-09T05:06:05.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa29e049640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fa2980cd5c0 con 0x7fa2980b4850 2024-05-09T05:06:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa2977fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2980c6a30 0x7fa29815d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa2977fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa2980c6a30 0x7fa29815d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51282/0 (socket says 172.21.15.1:51282) 2024-05-09T05:06:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa2977fe640 1 -- 172.21.15.1:0/577155925 learned_addr learned my addr 172.21.15.1:0/577155925 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa296ffd640 1 --2- 172.21.15.1:0/577155925 >> v2:172.21.15.196:3300/0 conn(0x7fa2980b4850 0x7fa2981568b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa2977fe640 1 -- 172.21.15.1:0/577155925 >> v2:172.21.15.196:3300/0 conn(0x7fa2980b4850 msgr2=0x7fa2981568b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa2977fe640 1 --2- 172.21.15.1:0/577155925 >> v2:172.21.15.196:3300/0 conn(0x7fa2980b4850 0x7fa2981568b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa2977fe640 1 -- 172.21.15.1:0/577155925 >> v1:172.21.15.196:6789/0 conn(0x7fa2980d1bc0 legacy=0x7fa2980ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.197+0000 7fa2977fe640 1 -- 172.21.15.1:0/577155925 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2980cdeb0 con 0x7fa2980c6a30 2024-05-09T05:06:05.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.198+0000 7fa2977fe640 1 --2- 172.21.15.1:0/577155925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980c6a30 0x7fa29815d0f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa2980d2c50 tx=0x7fa28802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cb8fccc56d81481 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.198+0000 7fa2967fc640 1 -- 172.21.15.1:0/577155925 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa288004230 con 0x7fa2980c6a30 2024-05-09T05:06:05.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.198+0000 7fa2967fc640 1 -- 172.21.15.1:0/577155925 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa28802ae50 con 0x7fa2980c6a30 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.198+0000 7fa2967fc640 1 -- 172.21.15.1:0/577155925 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa288003610 con 0x7fa2980c6a30 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.199+0000 7fa29e049640 1 -- 172.21.15.1:0/577155925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980c6a30 msgr2=0x7fa29815d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.199+0000 7fa29e049640 1 --2- 172.21.15.1:0/577155925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980c6a30 0x7fa29815d0f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fa2980d2c50 tx=0x7fa28802a300 comp rx=0 tx=0).stop 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.199+0000 7fa29e049640 1 -- 172.21.15.1:0/577155925 shutdown_connections 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.199+0000 7fa29e049640 1 --2- 172.21.15.1:0/577155925 >> v2:172.21.15.196:3300/0 conn(0x7fa2980b4850 0x7fa2981568b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.199+0000 7fa29e049640 1 --2- 172.21.15.1:0/577155925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980c6a30 0x7fa29815d0f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.199+0000 7fa29e049640 1 -- 172.21.15.1:0/577155925 >> 172.21.15.1:0/577155925 conn(0x7fa29806dc00 msgr2=0x7fa29806e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.199+0000 7fa29e049640 1 -- 172.21.15.1:0/577155925 shutdown_connections 2024-05-09T05:06:05.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.199+0000 7fa29e049640 1 -- 172.21.15.1:0/577155925 wait complete. 2024-05-09T05:06:05.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.201+0000 7fa29e049640 1 Processor -- start 2024-05-09T05:06:05.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.201+0000 7fa29e049640 1 -- start start 2024-05-09T05:06:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa29e049640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2980b4850 0x7fa298160520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa29e049640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa2980c6a30 0x7fa29815e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa29e049640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980d1bc0 0x7fa2980d0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa29e049640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa2980cf0a0 con 0x7fa2980d1bc0 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa29e049640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa2980ce470 con 0x7fa2980b4850 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa29e049640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa2980cf220 con 0x7fa2980c6a30 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa297fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980d1bc0 0x7fa2980d0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa297fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980d1bc0 0x7fa2980d0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51284/0 (socket says 172.21.15.1:51284) 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.202+0000 7fa297fff640 1 -- 172.21.15.1:0/2480744748 learned_addr learned my addr 172.21.15.1:0/2480744748 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa296ffd640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa2980c6a30 0x7fa29815e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa2977fe640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2980b4850 0x7fa298160520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa297fff640 1 -- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa2980c6a30 msgr2=0x7fa29815e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa297fff640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa2980c6a30 0x7fa29815e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa297fff640 1 -- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2980b4850 msgr2=0x7fa298160520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa297fff640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2980b4850 0x7fa298160520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa297fff640 1 -- 172.21.15.1:0/2480744748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa298160a60 con 0x7fa2980d1bc0 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa2977fe640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2980b4850 0x7fa298160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:05.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa297fff640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980d1bc0 0x7fa2980d0b90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa29000c700 tx=0x7fa29000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa27ffff640 1 -- 172.21.15.1:0/2480744748 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa290003b70 con 0x7fa2980d1bc0 2024-05-09T05:06:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa27ffff640 1 -- 172.21.15.1:0/2480744748 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa290003d10 con 0x7fa2980d1bc0 2024-05-09T05:06:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa29815e900 con 0x7fa2980d1bc0 2024-05-09T05:06:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa27ffff640 1 -- 172.21.15.1:0/2480744748 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa29000aa50 con 0x7fa2980d1bc0 2024-05-09T05:06:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.203+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2981f3260 con 0x7fa2980d1bc0 2024-05-09T05:06:05.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.204+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa298050870 con 0x7fa2980d1bc0 2024-05-09T05:06:05.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.205+0000 7fa27ffff640 1 -- 172.21.15.1:0/2480744748 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa29000abf0 con 0x7fa2980d1bc0 2024-05-09T05:06:05.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.205+0000 7fa27ffff640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa284076300 0x7fa284078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.205+0000 7fa27ffff640 1 -- 172.21.15.1:0/2480744748 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fa290097460 con 0x7fa2980d1bc0 2024-05-09T05:06:05.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.205+0000 7fa2977fe640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa284076300 0x7fa284078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.206+0000 7fa2977fe640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa284076300 0x7fa284078810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa288033000 tx=0x7fa28802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.210+0000 7fa27ffff640 1 -- 172.21.15.1:0/2480744748 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa290060dc0 con 0x7fa2980d1bc0 2024-05-09T05:06:05.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.403+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa2980cdbf0 con 0x7fa2980d1bc0 2024-05-09T05:06:05.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.404+0000 7fa27ffff640 1 -- 172.21.15.1:0/2480744748 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16558 (secure 0 0 0) 0x7fa2900606c0 con 0x7fa2980d1bc0 2024-05-09T05:06:05.405 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:05.406 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":20,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:06:04.442898+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:06:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa284076300 msgr2=0x7fa284078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa284076300 0x7fa284078810 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fa288033000 tx=0x7fa28802e000 comp rx=0 tx=0).stop 2024-05-09T05:06:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980d1bc0 msgr2=0x7fa2980d0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980d1bc0 0x7fa2980d0b90 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fa29000c700 tx=0x7fa29000cbc0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 shutdown_connections 2024-05-09T05:06:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa284076300 0x7fa284078810 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa2980d1bc0 0x7fa2980d0b90 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa2980c6a30 0x7fa29815e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 --2- 172.21.15.1:0/2480744748 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa2980b4850 0x7fa298160520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 >> 172.21.15.1:0/2480744748 conn(0x7fa29806dc00 msgr2=0x7fa2980c6e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 shutdown_connections 2024-05-09T05:06:05.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.407+0000 7fa29e049640 1 -- 172.21.15.1:0/2480744748 wait complete. 2024-05-09T05:06:05.419 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:06:05.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.532+0000 7f88a25f3640 1 Processor -- start 2024-05-09T05:06:05.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.532+0000 7f88a25f3640 1 -- start start 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f88a25f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c0c7a40 0x7f889c0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f88a25f3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f889c06de20 con 0x7f889c06efa0 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f88a25f3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f889c15c9a0 con 0x7f889c156900 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f88a25f3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f889c156d10 con 0x7f889c0c7a40 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f889bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f889c06efa0 0x7f889c0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42342/0 (socket says 172.21.15.1:42342) 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f889bfff640 1 -- 172.21.15.1:0/2365229827 learned_addr learned my addr 172.21.15.1:0/2365229827 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f88a0b90640 1 --2- 172.21.15.1:0/2365229827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c0c7a40 0x7f889c0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f889b7fe640 1 -- 172.21.15.1:0/2365229827 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2362845546 0 0) 0x7f889c06de20 con 0x7f889c06efa0 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f889b7fe640 1 -- 172.21.15.1:0/2365229827 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8888003520 con 0x7f889c06efa0 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f889b7fe640 1 -- 172.21.15.1:0/2365229827 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2789947839 0 0) 0x7f889c15c9a0 con 0x7f889c156900 2024-05-09T05:06:05.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f889b7fe640 1 -- 172.21.15.1:0/2365229827 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f889c06de20 con 0x7f889c156900 2024-05-09T05:06:05.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f88a0b90640 1 -- 172.21.15.1:0/2365229827 >> v1:172.21.15.196:6789/0 conn(0x7f889c156900 legacy=0x7f889c0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f88a0b90640 1 -- 172.21.15.1:0/2365229827 >> v1:172.21.15.1:6789/0 conn(0x7f889c06efa0 legacy=0x7f889c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.534+0000 7f88a0b90640 1 -- 172.21.15.1:0/2365229827 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c0b51a0 con 0x7f889c0c7a40 2024-05-09T05:06:05.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.535+0000 7f88a0b90640 1 --2- 172.21.15.1:0/2365229827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c0c7a40 0x7f889c0c7e50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8894009dc0 tx=0x7f889402a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=378e3894d611565b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.535+0000 7f889b7fe640 1 -- 172.21.15.1:0/2365229827 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8894035020 con 0x7f889c0c7a40 2024-05-09T05:06:05.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.535+0000 7f889b7fe640 1 -- 172.21.15.1:0/2365229827 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8894007540 con 0x7f889c0c7a40 2024-05-09T05:06:05.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.535+0000 7f889b7fe640 1 -- 172.21.15.1:0/2365229827 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f888c0016f0 con 0x7f889c0c7a40 2024-05-09T05:06:05.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.536+0000 7f88a25f3640 1 -- 172.21.15.1:0/2365229827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c0c7a40 msgr2=0x7f889c0c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.536+0000 7f88a25f3640 1 --2- 172.21.15.1:0/2365229827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c0c7a40 0x7f889c0c7e50 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8894009dc0 tx=0x7f889402a6d0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.536+0000 7f88a25f3640 1 -- 172.21.15.1:0/2365229827 shutdown_connections 2024-05-09T05:06:05.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.536+0000 7f88a25f3640 1 --2- 172.21.15.1:0/2365229827 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c0c7a40 0x7f889c0c7e50 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.536+0000 7f88a25f3640 1 -- 172.21.15.1:0/2365229827 >> 172.21.15.1:0/2365229827 conn(0x7f889c068d90 msgr2=0x7f889c0d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:05.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.536+0000 7f88a25f3640 1 -- 172.21.15.1:0/2365229827 shutdown_connections 2024-05-09T05:06:05.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.536+0000 7f88a25f3640 1 -- 172.21.15.1:0/2365229827 wait complete. 2024-05-09T05:06:05.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.538+0000 7f88a25f3640 1 Processor -- start 2024-05-09T05:06:05.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.538+0000 7f88a25f3640 1 -- start start 2024-05-09T05:06:05.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f88a25f3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f889c06efa0 0x7f889c0cd020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f88a25f3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f889c0c7a40 0x7f889c1493f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f88a25f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c156900 0x7f889c149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f88a25f3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f889c15cc40 con 0x7f889c0c7a40 2024-05-09T05:06:05.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f88a25f3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f889c06db40 con 0x7f889c06efa0 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f88a25f3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f889c15cf20 con 0x7f889c156900 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f8893fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f889c0c7a40 0x7f889c1493f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f8893fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f889c0c7a40 0x7f889c1493f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51296/0 (socket says 172.21.15.1:51296) 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f8893fff640 1 -- 172.21.15.1:0/395092231 learned_addr learned my addr 172.21.15.1:0/395092231 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f889bfff640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f889c06efa0 0x7f889c0cd020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.539+0000 7f88a0b90640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c156900 0x7f889c149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f8893fff640 1 -- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c156900 msgr2=0x7f889c149930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f8893fff640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c156900 0x7f889c149930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f8893fff640 1 -- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f889c06efa0 msgr2=0x7f889c0cd020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f8893fff640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f889c06efa0 0x7f889c0cd020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f8893fff640 1 -- 172.21.15.1:0/395092231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f889c0cbdf0 con 0x7f889c0c7a40 2024-05-09T05:06:05.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f8893fff640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f889c0c7a40 0x7f889c1493f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f888400c390 tx=0x7f888400c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f88997fa640 1 -- 172.21.15.1:0/395092231 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884017070 con 0x7f889c0c7a40 2024-05-09T05:06:05.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f88997fa640 1 -- 172.21.15.1:0/395092231 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88840052d0 con 0x7f889c0c7a40 2024-05-09T05:06:05.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f889c1f68e0 con 0x7f889c0c7a40 2024-05-09T05:06:05.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f88997fa640 1 -- 172.21.15.1:0/395092231 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8884003830 con 0x7f889c0c7a40 2024-05-09T05:06:05.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.540+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f889c1f6ea0 con 0x7f889c0c7a40 2024-05-09T05:06:05.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.541+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f889c050870 con 0x7f889c0c7a40 2024-05-09T05:06:05.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.542+0000 7f88997fa640 1 -- 172.21.15.1:0/395092231 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f888401a070 con 0x7f889c0c7a40 2024-05-09T05:06:05.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.542+0000 7f88997fa640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f887c0762d0 0x7f887c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.542+0000 7f88997fa640 1 -- 172.21.15.1:0/395092231 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7f8884096930 con 0x7f889c0c7a40 2024-05-09T05:06:05.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.543+0000 7f889bfff640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f887c0762d0 0x7f887c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.543+0000 7f889bfff640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f887c0762d0 0x7f887c0787e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f888c001860 tx=0x7f888c001710 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.547+0000 7f88997fa640 1 -- 172.21.15.1:0/395092231 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8884060290 con 0x7f889c0c7a40 2024-05-09T05:06:05.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.743+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f889c0cbb80 con 0x7f889c0c7a40 2024-05-09T05:06:05.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.744+0000 7f88997fa640 1 -- 172.21.15.1:0/395092231 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v20) ==== 74+0+16558 (secure 0 0 0) 0x7f8884068280 con 0x7f889c0c7a40 2024-05-09T05:06:05.745 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:05.745 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":20,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:06:04.442898+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:06:05.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f887c0762d0 msgr2=0x7f887c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f887c0762d0 0x7f887c0787e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f888c001860 tx=0x7f888c001710 comp rx=0 tx=0).stop 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f889c0c7a40 msgr2=0x7f889c1493f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f889c0c7a40 0x7f889c1493f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f888400c390 tx=0x7f888400c850 comp rx=0 tx=0).stop 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 shutdown_connections 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f887c0762d0 0x7f887c0787e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f889c156900 0x7f889c149930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f889c0c7a40 0x7f889c1493f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 --2- 172.21.15.1:0/395092231 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f889c06efa0 0x7f889c0cd020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 >> 172.21.15.1:0/395092231 conn(0x7f889c068d90 msgr2=0x7f889c15a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 shutdown_connections 2024-05-09T05:06:05.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.747+0000 7f88a25f3640 1 -- 172.21.15.1:0/395092231 wait complete. 2024-05-09T05:06:05.759 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T05:06:05.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.878+0000 7fc5c6384640 1 Processor -- start 2024-05-09T05:06:05.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.878+0000 7fc5c6384640 1 -- start start 2024-05-09T05:06:05.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.879+0000 7fc5c6384640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc5c00cf360 0x7fc5c00b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.879+0000 7fc5c6384640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c00b4570 0x7fc5c0156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5c6384640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5c00ce5c0 con 0x7fc5c015c9a0 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5c6384640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc5c015d4c0 con 0x7fc5c00cf360 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5c6384640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c015d7a0 con 0x7fc5c00b4570 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bffff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc5c00cf360 0x7fc5c00b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bf7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c00b4570 0x7fc5c0156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bffff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc5c00cf360 0x7fc5c00b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:32864/0 (socket says 172.21.15.1:32864) 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bf7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c00b4570 0x7fc5c0156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60438/0 (socket says 172.21.15.1:60438) 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5c4921640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc5c015c9a0 0x7fc5c00cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:36256/0 (socket says 172.21.15.1:36256) 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bffff640 1 -- 172.21.15.1:0/2985187227 learned_addr learned my addr 172.21.15.1:0/2985187227 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bffff640 1 -- 172.21.15.1:0/2985187227 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c00b4570 msgr2=0x7fc5c0156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bffff640 1 --2- 172.21.15.1:0/2985187227 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c00b4570 0x7fc5c0156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bffff640 1 -- 172.21.15.1:0/2985187227 >> v1:172.21.15.196:6789/0 conn(0x7fc5c015c9a0 legacy=0x7fc5c00cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.880+0000 7fc5bffff640 1 -- 172.21.15.1:0/2985187227 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c00ce880 con 0x7fc5c00cf360 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.881+0000 7fc5bffff640 1 --2- 172.21.15.1:0/2985187227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00cf360 0x7fc5c00b4b50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009870 tx=0x7fc5b002a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=64a5aea96046a755 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.881+0000 7fc5beffd640 1 -- 172.21.15.1:0/2985187227 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0037070 con 0x7fc5c00cf360 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.881+0000 7fc5beffd640 1 -- 172.21.15.1:0/2985187227 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b002ae50 con 0x7fc5c00cf360 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.881+0000 7fc5beffd640 1 -- 172.21.15.1:0/2985187227 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b0032860 con 0x7fc5c00cf360 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.882+0000 7fc5c6384640 1 -- 172.21.15.1:0/2985187227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00cf360 msgr2=0x7fc5c00b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.882+0000 7fc5c6384640 1 --2- 172.21.15.1:0/2985187227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00cf360 0x7fc5c00b4b50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0009870 tx=0x7fc5b002a300 comp rx=0 tx=0).stop 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.882+0000 7fc5c6384640 1 -- 172.21.15.1:0/2985187227 shutdown_connections 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.882+0000 7fc5c6384640 1 --2- 172.21.15.1:0/2985187227 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c00b4570 0x7fc5c0156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.882+0000 7fc5c6384640 1 --2- 172.21.15.1:0/2985187227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00cf360 0x7fc5c00b4b50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.882+0000 7fc5c6384640 1 -- 172.21.15.1:0/2985187227 >> 172.21.15.1:0/2985187227 conn(0x7fc5c006de10 msgr2=0x7fc5c006e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.882+0000 7fc5c6384640 1 -- 172.21.15.1:0/2985187227 shutdown_connections 2024-05-09T05:06:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.882+0000 7fc5c6384640 1 -- 172.21.15.1:0/2985187227 wait complete. 2024-05-09T05:06:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.884+0000 7fc5c6384640 1 Processor -- start 2024-05-09T05:06:05.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.884+0000 7fc5c6384640 1 -- start start 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c6384640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00b4570 0x7fc5c0157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c6384640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c015c9a0 0x7fc5c015b500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c6384640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0158ee0 0x7fc5c01528d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c6384640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c00b5090 con 0x7fc5c0158ee0 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c6384640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc5c00cee40 con 0x7fc5c00b4570 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c6384640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc5c0157090 con 0x7fc5c015c9a0 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c4921640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0158ee0 0x7fc5c01528d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c4921640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0158ee0 0x7fc5c01528d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51306/0 (socket says 172.21.15.1:51306) 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5c4921640 1 -- 172.21.15.1:0/1914177349 learned_addr learned my addr 172.21.15.1:0/1914177349 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5bffff640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00b4570 0x7fc5c0157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.885+0000 7fc5bf7fe640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c015c9a0 0x7fc5c015b500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5c4921640 1 -- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c015c9a0 msgr2=0x7fc5c015b500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5c4921640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c015c9a0 0x7fc5c015b500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5c4921640 1 -- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00b4570 msgr2=0x7fc5c0157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:05.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5c4921640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00b4570 0x7fc5c0157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5c4921640 1 -- 172.21.15.1:0/1914177349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5c0157a90 con 0x7fc5c0158ee0 2024-05-09T05:06:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5c4921640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0158ee0 0x7fc5c01528d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c760 tx=0x7fc5b800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5bcff9640 1 -- 172.21.15.1:0/1914177349 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b8017070 con 0x7fc5c0158ee0 2024-05-09T05:06:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5bcff9640 1 -- 172.21.15.1:0/1914177349 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc5b800a5e0 con 0x7fc5c0158ee0 2024-05-09T05:06:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5c015a160 con 0x7fc5c0158ee0 2024-05-09T05:06:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5bcff9640 1 -- 172.21.15.1:0/1914177349 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc5b800a990 con 0x7fc5c0158ee0 2024-05-09T05:06:05.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.886+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5c0158c70 con 0x7fc5c0158ee0 2024-05-09T05:06:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.887+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc5c0050870 con 0x7fc5c0158ee0 2024-05-09T05:06:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.888+0000 7fc5bcff9640 1 -- 172.21.15.1:0/1914177349 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc5b801a070 con 0x7fc5c0158ee0 2024-05-09T05:06:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.888+0000 7fc5bcff9640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5ac0762d0 0x7fc5ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.889+0000 7fc5bcff9640 1 -- 172.21.15.1:0/1914177349 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 5709+0+0 (secure 0 0 0) 0x7fc5b8096bd0 con 0x7fc5c0158ee0 2024-05-09T05:06:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.889+0000 7fc5bffff640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5ac0762d0 0x7fc5ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:05.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.889+0000 7fc5bffff640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5ac0762d0 0x7fc5ac0787e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0034000 tx=0x7fc5b002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:05.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:05.894+0000 7fc5bcff9640 1 -- 172.21.15.1:0/1914177349 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc5b805fd30 con 0x7fc5c0158ee0 2024-05-09T05:06:06.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:06.126+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fc5c0153970 con 0x7fc5c0158ee0 2024-05-09T05:06:07.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.057+0000 7fc5bcff9640 1 -- 172.21.15.1:0/1914177349 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v5) ==== 118+0+0 (secure 0 0 0) 0x7fc5b8068400 con 0x7fc5c0158ee0 2024-05-09T05:06:07.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5ac0762d0 msgr2=0x7fc5ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:07.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5ac0762d0 0x7fc5ac0787e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc5b0034000 tx=0x7fc5b002e000 comp rx=0 tx=0).stop 2024-05-09T05:06:07.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0158ee0 msgr2=0x7fc5c01528d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:07.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0158ee0 0x7fc5c01528d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fc5b800c760 tx=0x7fc5b800cc20 comp rx=0 tx=0).stop 2024-05-09T05:06:07.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 shutdown_connections 2024-05-09T05:06:07.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5ac0762d0 0x7fc5ac0787e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc5c0158ee0 0x7fc5c01528d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5c015c9a0 0x7fc5c015b500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 --2- 172.21.15.1:0/1914177349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc5c00b4570 0x7fc5c0157550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 >> 172.21.15.1:0/1914177349 conn(0x7fc5c006de10 msgr2=0x7fc5c006e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:07.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.060+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 shutdown_connections 2024-05-09T05:06:07.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.061+0000 7fc5c6384640 1 -- 172.21.15.1:0/1914177349 wait complete. 2024-05-09T05:06:07.075 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300} 2024-05-09T05:06:07.076 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-05-09T05:06:07.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.199+0000 7f4e00146640 1 Processor -- start 2024-05-09T05:06:07.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.200+0000 7f4e00146640 1 -- start start 2024-05-09T05:06:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4e00146640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4df80cf360 0x7f4df80b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4e00146640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f4df80b4570 0x7f4df8156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4e00146640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4df80ce5c0 con 0x7f4df815c9a0 2024-05-09T05:06:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4e00146640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4df815d4c0 con 0x7f4df80cf360 2024-05-09T05:06:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4e00146640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f4df815d7a0 con 0x7f4df80b4570 2024-05-09T05:06:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfdee2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4df80cf360 0x7f4df80b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:07.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfdee2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4df80cf360 0x7f4df80b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51322/0 (socket says 172.21.15.1:51322) 2024-05-09T05:06:07.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfdee2640 1 -- 172.21.15.1:0/3419597657 learned_addr learned my addr 172.21.15.1:0/3419597657 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:07.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfe6e3640 1 --1- 172.21.15.1:0/3419597657 >> v1:172.21.15.1:6789/0 conn(0x7f4df815c9a0 0x7f4df80cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42354/0 (socket says 172.21.15.1:42354) 2024-05-09T05:06:07.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfd6e1640 1 --2- 172.21.15.1:0/3419597657 >> v2:172.21.15.196:3300/0 conn(0x7f4df80b4570 0x7f4df8156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:07.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfdee2640 1 -- 172.21.15.1:0/3419597657 >> v2:172.21.15.196:3300/0 conn(0x7f4df80b4570 msgr2=0x7f4df8156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:07.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfdee2640 1 --2- 172.21.15.1:0/3419597657 >> v2:172.21.15.196:3300/0 conn(0x7f4df80b4570 0x7f4df8156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfdee2640 1 -- 172.21.15.1:0/3419597657 >> v1:172.21.15.1:6789/0 conn(0x7f4df815c9a0 legacy=0x7f4df80cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:07.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfcee0640 1 -- 172.21.15.1:0/3419597657 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 943591298 0 0) 0x7f4df80ce5c0 con 0x7f4df815c9a0 2024-05-09T05:06:07.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.201+0000 7f4dfdee2640 1 -- 172.21.15.1:0/3419597657 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df80ce880 con 0x7f4df80cf360 2024-05-09T05:06:07.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.202+0000 7f4dfdee2640 1 --2- 172.21.15.1:0/3419597657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df80cf360 0x7f4df80b4b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4df80cf8c0 tx=0x7f4df002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=130b1ea031e30f6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:07.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.202+0000 7f4dfcee0640 1 -- 172.21.15.1:0/3419597657 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df0037070 con 0x7f4df80cf360 2024-05-09T05:06:07.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.202+0000 7f4dfcee0640 1 -- 172.21.15.1:0/3419597657 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4df002ae50 con 0x7f4df80cf360 2024-05-09T05:06:07.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.202+0000 7f4dfcee0640 1 -- 172.21.15.1:0/3419597657 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4df0032860 con 0x7f4df80cf360 2024-05-09T05:06:07.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.203+0000 7f4e00146640 1 -- 172.21.15.1:0/3419597657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df80cf360 msgr2=0x7f4df80b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.203+0000 7f4e00146640 1 --2- 172.21.15.1:0/3419597657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df80cf360 0x7f4df80b4b50 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f4df80cf8c0 tx=0x7f4df002a300 comp rx=0 tx=0).stop 2024-05-09T05:06:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.203+0000 7f4e00146640 1 -- 172.21.15.1:0/3419597657 shutdown_connections 2024-05-09T05:06:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.203+0000 7f4e00146640 1 --2- 172.21.15.1:0/3419597657 >> v2:172.21.15.196:3300/0 conn(0x7f4df80b4570 0x7f4df8156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.203+0000 7f4e00146640 1 --2- 172.21.15.1:0/3419597657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df80cf360 0x7f4df80b4b50 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.203+0000 7f4e00146640 1 -- 172.21.15.1:0/3419597657 >> 172.21.15.1:0/3419597657 conn(0x7f4df806de10 msgr2=0x7f4df806e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.203+0000 7f4e00146640 1 -- 172.21.15.1:0/3419597657 shutdown_connections 2024-05-09T05:06:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.203+0000 7f4e00146640 1 -- 172.21.15.1:0/3419597657 wait complete. 2024-05-09T05:06:07.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.205+0000 7f4e00146640 1 Processor -- start 2024-05-09T05:06:07.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.205+0000 7f4e00146640 1 -- start start 2024-05-09T05:06:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4e00146640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4df80b4570 0x7f4df81e6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4e00146640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df815c9a0 0x7f4df81f2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4e00146640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4df81e9360 0x7f4df8156000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:07.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4e00146640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4df80b5090 con 0x7f4df815c9a0 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4e00146640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4df80cee40 con 0x7f4df81e9360 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4e00146640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4df8157090 con 0x7f4df80b4570 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4dfd6e1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df815c9a0 0x7f4df81f2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4dfd6e1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df815c9a0 0x7f4df81f2260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51324/0 (socket says 172.21.15.1:51324) 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.206+0000 7f4dfd6e1640 1 -- 172.21.15.1:0/130536709 learned_addr learned my addr 172.21.15.1:0/130536709 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfdee2640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4df80b4570 0x7f4df81e6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfe6e3640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4df81e9360 0x7f4df8156000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfd6e1640 1 -- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4df80b4570 msgr2=0x7f4df81e6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfd6e1640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4df80b4570 0x7f4df81e6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfd6e1640 1 -- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4df81e9360 msgr2=0x7f4df8156000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfd6e1640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4df81e9360 0x7f4df8156000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfd6e1640 1 -- 172.21.15.1:0/130536709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4df81f5a70 con 0x7f4df815c9a0 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfdee2640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4df80b4570 0x7f4df81e6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:07.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4dfd6e1640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df815c9a0 0x7f4df81f2260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e8a0 tx=0x7f4dec00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4df67fc640 1 -- 172.21.15.1:0/130536709 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec017070 con 0x7f4df815c9a0 2024-05-09T05:06:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4df67fc640 1 -- 172.21.15.1:0/130536709 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4dec00c940 con 0x7f4df815c9a0 2024-05-09T05:06:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4df81f5d50 con 0x7f4df815c9a0 2024-05-09T05:06:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4df67fc640 1 -- 172.21.15.1:0/130536709 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4dec00cd30 con 0x7f4df815c9a0 2024-05-09T05:06:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.207+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4df81f6310 con 0x7f4df815c9a0 2024-05-09T05:06:07.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.208+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4df8050870 con 0x7f4df815c9a0 2024-05-09T05:06:07.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.209+0000 7f4df67fc640 1 -- 172.21.15.1:0/130536709 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4dec01a070 con 0x7f4df815c9a0 2024-05-09T05:06:07.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.209+0000 7f4df67fc640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4de40762d0 0x7f4de40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:07.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.209+0000 7f4df67fc640 1 -- 172.21.15.1:0/130536709 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4dec0978e0 con 0x7f4df815c9a0 2024-05-09T05:06:07.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.209+0000 7f4dfdee2640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4de40762d0 0x7f4de40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:07.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.210+0000 7f4dfdee2640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4de40762d0 0x7f4de40787e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4df0034000 tx=0x7f4df002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:07.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.214+0000 7f4df67fc640 1 -- 172.21.15.1:0/130536709 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4dec0611b0 con 0x7f4df815c9a0 2024-05-09T05:06:07.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:07.448+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) -- 0x7f4df8069ca0 con 0x7f4df815c9a0 2024-05-09T05:06:08.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.066+0000 7f4df67fc640 1 -- 172.21.15.1:0/130536709 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v6) ==== 105+0+0 (secure 0 0 0) 0x7f4dec060ab0 con 0x7f4df815c9a0 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4de40762d0 msgr2=0x7f4de40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4de40762d0 0x7f4de40787e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f4df0034000 tx=0x7f4df002e000 comp rx=0 tx=0).stop 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df815c9a0 msgr2=0x7f4df81f2260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df815c9a0 0x7f4df81f2260 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f4dec00e8a0 tx=0x7f4dec00ed60 comp rx=0 tx=0).stop 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 shutdown_connections 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4de40762d0 0x7f4de40787e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4df81e9360 0x7f4df8156000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4df815c9a0 0x7f4df81f2260 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 --2- 172.21.15.1:0/130536709 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4df80b4570 0x7f4df81e6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 >> 172.21.15.1:0/130536709 conn(0x7f4df806de10 msgr2=0x7f4df80cfd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:08.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 shutdown_connections 2024-05-09T05:06:08.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.070+0000 7f4e00146640 1 -- 172.21.15.1:0/130536709 wait complete. 2024-05-09T05:06:08.084 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay false 2024-05-09T05:06:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.207+0000 7ff861e3b640 1 Processor -- start 2024-05-09T05:06:08.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.207+0000 7ff861e3b640 1 -- start start 2024-05-09T05:06:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff861e3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c0b4ec0 0x7ff85c0cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff861e3b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff85c160460 0x7ff85c160870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff861e3b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff85c15c7b0 con 0x7ff85c0ce010 2024-05-09T05:06:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff861e3b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff85c15d0c0 con 0x7ff85c160460 2024-05-09T05:06:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff861e3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff85c0cca50 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff85bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff85c160460 0x7ff85c160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff85bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff85c160460 0x7ff85c160870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51338/0 (socket says 172.21.15.1:51338) 2024-05-09T05:06:08.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff85bfff640 1 -- 172.21.15.1:0/1399846431 learned_addr learned my addr 172.21.15.1:0/1399846431 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.208+0000 7ff85b7fe640 1 --2- 172.21.15.1:0/1399846431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c0b4ec0 0x7ff85c0cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.209+0000 7ff85bfff640 1 -- 172.21.15.1:0/1399846431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c0b4ec0 msgr2=0x7ff85c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.209+0000 7ff85bfff640 1 --2- 172.21.15.1:0/1399846431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c0b4ec0 0x7ff85c0cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.209+0000 7ff85bfff640 1 -- 172.21.15.1:0/1399846431 >> v1:172.21.15.196:6789/0 conn(0x7ff85c0ce010 legacy=0x7ff85c15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.209+0000 7ff85bfff640 1 -- 172.21.15.1:0/1399846431 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85c0ccd30 con 0x7ff85c160460 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.209+0000 7ff85bfff640 1 --2- 172.21.15.1:0/1399846431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c160460 0x7ff85c160870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff85c15d240 tx=0x7ff85402a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673b27da1c7fc526 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.209+0000 7ff85a7fc640 1 -- 172.21.15.1:0/1399846431 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff854036070 con 0x7ff85c160460 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.209+0000 7ff85a7fc640 1 -- 172.21.15.1:0/1399846431 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff854031600 con 0x7ff85c160460 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.209+0000 7ff85a7fc640 1 -- 172.21.15.1:0/1399846431 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8540319d0 con 0x7ff85c160460 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.210+0000 7ff861e3b640 1 -- 172.21.15.1:0/1399846431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c160460 msgr2=0x7ff85c160870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.210+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1399846431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c160460 0x7ff85c160870 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7ff85c15d240 tx=0x7ff85402a7a0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.210+0000 7ff861e3b640 1 -- 172.21.15.1:0/1399846431 shutdown_connections 2024-05-09T05:06:08.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.210+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1399846431 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c160460 0x7ff85c160870 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.210+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1399846431 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c0b4ec0 0x7ff85c0cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.210+0000 7ff861e3b640 1 -- 172.21.15.1:0/1399846431 >> 172.21.15.1:0/1399846431 conn(0x7ff85c0d2360 msgr2=0x7ff85c0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:08.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.210+0000 7ff861e3b640 1 -- 172.21.15.1:0/1399846431 shutdown_connections 2024-05-09T05:06:08.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.210+0000 7ff861e3b640 1 -- 172.21.15.1:0/1399846431 wait complete. 2024-05-09T05:06:08.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.212+0000 7ff861e3b640 1 Processor -- start 2024-05-09T05:06:08.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.212+0000 7ff861e3b640 1 -- start start 2024-05-09T05:06:08.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.213+0000 7ff861e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c0b4ec0 0x7ff85c1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.213+0000 7ff861e3b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff85c0ce010 0x7ff85c15ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.213+0000 7ff861e3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c160460 0x7ff85c152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.213+0000 7ff861e3b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c0cc750 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.213+0000 7ff861e3b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff85c15cd50 con 0x7ff85c0ce010 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.213+0000 7ff861e3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff85c0cff10 con 0x7ff85c160460 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.213+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c0b4ec0 0x7ff85c1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c0b4ec0 0x7ff85c1509a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51352/0 (socket says 172.21.15.1:51352) 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85b7fe640 1 -- 172.21.15.1:0/1640069833 learned_addr learned my addr 172.21.15.1:0/1640069833 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85affd640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff85c0ce010 0x7ff85c15ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85bfff640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c160460 0x7ff85c152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85b7fe640 1 -- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c160460 msgr2=0x7ff85c152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85b7fe640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c160460 0x7ff85c152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85b7fe640 1 -- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff85c0ce010 msgr2=0x7ff85c15ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85b7fe640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff85c0ce010 0x7ff85c15ebb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85b7fe640 1 -- 172.21.15.1:0/1640069833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff85c150ee0 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff85b7fe640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c0b4ec0 0x7ff85c1509a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00d600 tx=0x7ff84c00dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff843fff640 1 -- 172.21.15.1:0/1640069833 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c0125f0 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff843fff640 1 -- 172.21.15.1:0/1640069833 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff84c019070 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff85c1f2d50 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff843fff640 1 -- 172.21.15.1:0/1640069833 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff84c0129f0 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.214+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff85c1f3310 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.215+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff85c050870 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.216+0000 7ff843fff640 1 -- 172.21.15.1:0/1640069833 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff84c012b90 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.217+0000 7ff843fff640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff848076300 0x7ff848078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.217+0000 7ff843fff640 1 -- 172.21.15.1:0/1640069833 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff84c0979d0 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.217+0000 7ff85affd640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff848076300 0x7ff848078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:08.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.218+0000 7ff85affd640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff848076300 0x7ff848078810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff85c1ed480 tx=0x7ff850009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:08.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.222+0000 7ff843fff640 1 -- 172.21.15.1:0/1640069833 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff84c060bb0 con 0x7ff85c0b4ec0 2024-05-09T05:06:08.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:08.449+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"} v 0) -- 0x7ff85c0cd2b0 con 0x7ff85c0b4ec0 2024-05-09T05:06:09.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.068+0000 7ff843fff640 1 -- 172.21.15.1:0/1640069833 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "false"}]=0 v7) ==== 122+0+0 (secure 0 0 0) 0x7ff84c064eb0 con 0x7ff85c0b4ec0 2024-05-09T05:06:09.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.070+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff848076300 msgr2=0x7ff848078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff848076300 0x7ff848078810 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ff85c1ed480 tx=0x7ff850009040 comp rx=0 tx=0).stop 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c0b4ec0 msgr2=0x7ff85c1509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c0b4ec0 0x7ff85c1509a0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff84c00d600 tx=0x7ff84c00dac0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 shutdown_connections 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff848076300 0x7ff848078810 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff85c0ce010 0x7ff85c15ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff85c160460 0x7ff85c152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 --2- 172.21.15.1:0/1640069833 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff85c0b4ec0 0x7ff85c1509a0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 >> 172.21.15.1:0/1640069833 conn(0x7ff85c0d2360 msgr2=0x7ff85c0b5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 shutdown_connections 2024-05-09T05:06:09.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.071+0000 7ff861e3b640 1 -- 172.21.15.1:0/1640069833 wait complete. 2024-05-09T05:06:09.086 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-09T05:06:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.211+0000 7fe66b42b640 1 Processor -- start 2024-05-09T05:06:09.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.211+0000 7fe66b42b640 1 -- start start 2024-05-09T05:06:09.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe66b42b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe6640ce010 0x7fe6640c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:09.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe66b42b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6640c77c0 0x7fe6640c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:09.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe66b42b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe66415c7b0 con 0x7fe6640b4ec0 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe66b42b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe66415d0c0 con 0x7fe6640c77c0 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe66b42b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe6640cca50 con 0x7fe6640ce010 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe6699c8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6640c77c0 0x7fe6640c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe6699c8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe6640c77c0 0x7fe6640c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51364/0 (socket says 172.21.15.1:51364) 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe6699c8640 1 -- 172.21.15.1:0/3328311043 learned_addr learned my addr 172.21.15.1:0/3328311043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.212+0000 7fe6691c7640 1 --1- 172.21.15.1:0/3328311043 >> v1:172.21.15.1:6789/0 conn(0x7fe6640b4ec0 0x7fe6640cf9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42366/0 (socket says 172.21.15.1:42366) 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.213+0000 7fe6689c6640 1 --2- 172.21.15.1:0/3328311043 >> v2:172.21.15.196:3300/0 conn(0x7fe6640ce010 0x7fe6640c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.213+0000 7fe6699c8640 1 -- 172.21.15.1:0/3328311043 >> v2:172.21.15.196:3300/0 conn(0x7fe6640ce010 msgr2=0x7fe6640c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.213+0000 7fe6699c8640 1 --2- 172.21.15.1:0/3328311043 >> v2:172.21.15.196:3300/0 conn(0x7fe6640ce010 0x7fe6640c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.213+0000 7fe6699c8640 1 -- 172.21.15.1:0/3328311043 >> v1:172.21.15.1:6789/0 conn(0x7fe6640b4ec0 legacy=0x7fe6640cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.213+0000 7fe65bfff640 1 -- 172.21.15.1:0/3328311043 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1420699544 0 0) 0x7fe66415c7b0 con 0x7fe6640b4ec0 2024-05-09T05:06:09.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.213+0000 7fe6699c8640 1 -- 172.21.15.1:0/3328311043 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6640ccd30 con 0x7fe6640c77c0 2024-05-09T05:06:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe6699c8640 1 --2- 172.21.15.1:0/3328311043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640c77c0 0x7fe6640c7bd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe6640c8b40 tx=0x7fe65c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed420fcd63327f68 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe65bfff640 1 -- 172.21.15.1:0/3328311043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c037070 con 0x7fe6640c77c0 2024-05-09T05:06:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe65bfff640 1 -- 172.21.15.1:0/3328311043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe65c02ae50 con 0x7fe6640c77c0 2024-05-09T05:06:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe65bfff640 1 -- 172.21.15.1:0/3328311043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c032860 con 0x7fe6640c77c0 2024-05-09T05:06:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe66b42b640 1 -- 172.21.15.1:0/3328311043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640c77c0 msgr2=0x7fe6640c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3328311043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640c77c0 0x7fe6640c7bd0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe6640c8b40 tx=0x7fe65c02a300 comp rx=0 tx=0).stop 2024-05-09T05:06:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe66b42b640 1 -- 172.21.15.1:0/3328311043 shutdown_connections 2024-05-09T05:06:09.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3328311043 >> v2:172.21.15.196:3300/0 conn(0x7fe6640ce010 0x7fe6640c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3328311043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640c77c0 0x7fe6640c7bd0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.214+0000 7fe66b42b640 1 -- 172.21.15.1:0/3328311043 >> 172.21.15.1:0/3328311043 conn(0x7fe6640d2360 msgr2=0x7fe6640ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.215+0000 7fe66b42b640 1 -- 172.21.15.1:0/3328311043 shutdown_connections 2024-05-09T05:06:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.215+0000 7fe66b42b640 1 -- 172.21.15.1:0/3328311043 wait complete. 2024-05-09T05:06:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.216+0000 7fe66b42b640 1 Processor -- start 2024-05-09T05:06:09.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.216+0000 7fe66b42b640 1 -- start start 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.217+0000 7fe66b42b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6640b4ec0 0x7fe6640c9be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.217+0000 7fe66b42b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6640c77c0 0x7fe6640b5710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.217+0000 7fe66b42b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640ce010 0x7fe6641595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe66b42b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe6640cc750 con 0x7fe6640ce010 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe66b42b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe66415cd50 con 0x7fe6640b4ec0 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe66b42b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe6640c8110 con 0x7fe6640c77c0 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640ce010 0x7fe6641595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640ce010 0x7fe6641595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51368/0 (socket says 172.21.15.1:51368) 2024-05-09T05:06:09.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 -- 172.21.15.1:0/3076973797 learned_addr learned my addr 172.21.15.1:0/3076973797 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6691c7640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6640b4ec0 0x7fe6640c9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6689c6640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6640c77c0 0x7fe6640b5710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 -- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6640c77c0 msgr2=0x7fe6640b5710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6640c77c0 0x7fe6640b5710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 -- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6640b4ec0 msgr2=0x7fe6640c9be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6640b4ec0 0x7fe6640c9be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 -- 172.21.15.1:0/3076973797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6640ca120 con 0x7fe6640ce010 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe6699c8640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640ce010 0x7fe6641595c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0090b0 tx=0x7fe65c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe659ffb640 1 -- 172.21.15.1:0/3076973797 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c041070 con 0x7fe6640ce010 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe659ffb640 1 -- 172.21.15.1:0/3076973797 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe65c03c680 con 0x7fe6640ce010 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe66415b9f0 con 0x7fe6640ce010 2024-05-09T05:06:09.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.218+0000 7fe659ffb640 1 -- 172.21.15.1:0/3076973797 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe65c037040 con 0x7fe6640ce010 2024-05-09T05:06:09.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.219+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe66415c4e0 con 0x7fe6640ce010 2024-05-09T05:06:09.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.219+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe664050870 con 0x7fe6640ce010 2024-05-09T05:06:09.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.220+0000 7fe659ffb640 1 -- 172.21.15.1:0/3076973797 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe65c044070 con 0x7fe6640ce010 2024-05-09T05:06:09.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.221+0000 7fe659ffb640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe6480762d0 0x7fe6480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:09.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.221+0000 7fe659ffb640 1 -- 172.21.15.1:0/3076973797 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe65c0b8650 con 0x7fe6640ce010 2024-05-09T05:06:09.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.221+0000 7fe6691c7640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe6480762d0 0x7fe6480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:09.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.222+0000 7fe6691c7640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe6480762d0 0x7fe6480787e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe660007ed0 tx=0x7fe660007e60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:09.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.225+0000 7fe659ffb640 1 -- 172.21.15.1:0/3076973797 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe65c081f90 con 0x7fe6640ce010 2024-05-09T05:06:09.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:09.455+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fe6640cc000 con 0x7fe6640ce010 2024-05-09T05:06:10.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.078+0000 7fe659ffb640 1 -- 172.21.15.1:0/3076973797 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v8) ==== 115+0+0 (secure 0 0 0) 0x7fe65c081890 con 0x7fe6640ce010 2024-05-09T05:06:10.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe6480762d0 msgr2=0x7fe6480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe6480762d0 0x7fe6480787e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe660007ed0 tx=0x7fe660007e60 comp rx=0 tx=0).stop 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640ce010 msgr2=0x7fe6641595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640ce010 0x7fe6641595c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe65c0090b0 tx=0x7fe65c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 shutdown_connections 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe6480762d0 0x7fe6480787e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6640ce010 0x7fe6641595c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6640c77c0 0x7fe6640b5710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 --2- 172.21.15.1:0/3076973797 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6640b4ec0 0x7fe6640c9be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 >> 172.21.15.1:0/3076973797 conn(0x7fe6640d2360 msgr2=0x7fe664154a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 shutdown_connections 2024-05-09T05:06:10.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.081+0000 7fe66b42b640 1 -- 172.21.15.1:0/3076973797 wait complete. 2024-05-09T05:06:10.096 DEBUG:tasks.cephfs.filesystem:Not Creating any subvols for filesystem 'cephfs' 2024-05-09T05:06:10.096 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:06:10.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.218+0000 7f0a136e2640 1 Processor -- start 2024-05-09T05:06:10.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.218+0000 7f0a136e2640 1 -- start start 2024-05-09T05:06:10.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a136e2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a136e2640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0a0c153830 0x7f0a0c0cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a136e2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a0c0c4430 con 0x7f0a0c070530 2024-05-09T05:06:10.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a136e2640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0a0c0c3b90 con 0x7f0a0c153830 2024-05-09T05:06:10.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a136e2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0a0c0c3e70 con 0x7f0a0c159440 2024-05-09T05:06:10.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a1147e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a11c7f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0a0c153830 0x7f0a0c0cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a11c7f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0a0c153830 0x7f0a0c0cb800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43710/0 (socket says 172.21.15.1:43710) 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a1147e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59428/0 (socket says 172.21.15.1:59428) 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a11c7f640 1 -- 172.21.15.1:0/3871270824 learned_addr learned my addr 172.21.15.1:0/3871270824 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.219+0000 7f0a10c7d640 1 --1- 172.21.15.1:0/3871270824 >> v1:172.21.15.196:6789/0 conn(0x7f0a0c070530 0x7f0a0c15d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:57578/0 (socket says 172.21.15.1:57578) 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.220+0000 7f0a1147e640 1 -- 172.21.15.1:0/3871270824 >> v2:172.21.15.196:3300/0 conn(0x7f0a0c153830 msgr2=0x7f0a0c0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.220+0000 7f0a1147e640 1 --2- 172.21.15.1:0/3871270824 >> v2:172.21.15.196:3300/0 conn(0x7f0a0c153830 0x7f0a0c0cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.220+0000 7f0a1147e640 1 -- 172.21.15.1:0/3871270824 >> v1:172.21.15.196:6789/0 conn(0x7f0a0c070530 legacy=0x7f0a0c15d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.220+0000 7f0a1147e640 1 -- 172.21.15.1:0/3871270824 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0c0c4150 con 0x7f0a0c159440 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.220+0000 7f0a11c7f640 1 --2- 172.21.15.1:0/3871270824 >> v2:172.21.15.196:3300/0 conn(0x7f0a0c153830 0x7f0a0c0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a1147e640 1 --2- 172.21.15.1:0/3871270824 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c070ae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0a040097a0 tx=0x7f0a0402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=988c9b618c2a36b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a03fff640 1 -- 172.21.15.1:0/3871270824 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04037070 con 0x7f0a0c159440 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a03fff640 1 -- 172.21.15.1:0/3871270824 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a04032440 con 0x7f0a0c159440 2024-05-09T05:06:10.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a03fff640 1 -- 172.21.15.1:0/3871270824 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04032810 con 0x7f0a0c159440 2024-05-09T05:06:10.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a136e2640 1 -- 172.21.15.1:0/3871270824 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 msgr2=0x7f0a0c070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a136e2640 1 --2- 172.21.15.1:0/3871270824 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c070ae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0a040097a0 tx=0x7f0a0402a210 comp rx=0 tx=0).stop 2024-05-09T05:06:10.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a136e2640 1 -- 172.21.15.1:0/3871270824 shutdown_connections 2024-05-09T05:06:10.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a136e2640 1 --2- 172.21.15.1:0/3871270824 >> v2:172.21.15.196:3300/0 conn(0x7f0a0c153830 0x7f0a0c0cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a136e2640 1 --2- 172.21.15.1:0/3871270824 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c070ae0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.221+0000 7f0a136e2640 1 -- 172.21.15.1:0/3871270824 >> 172.21.15.1:0/3871270824 conn(0x7f0a0c071b40 msgr2=0x7f0a0c071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:10.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.222+0000 7f0a136e2640 1 -- 172.21.15.1:0/3871270824 shutdown_connections 2024-05-09T05:06:10.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.222+0000 7f0a136e2640 1 -- 172.21.15.1:0/3871270824 wait complete. 2024-05-09T05:06:10.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.223+0000 7f0a136e2640 1 Processor -- start 2024-05-09T05:06:10.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.223+0000 7f0a136e2640 1 -- start start 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.224+0000 7f0a136e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0c070530 0x7f0a0c1ec000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.224+0000 7f0a136e2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a0c153830 0x7f0a0c13c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a136e2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c13c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a136e2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c0c3870 con 0x7f0a0c070530 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a136e2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0a0c15ceb0 con 0x7f0a0c153830 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a136e2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0a0c06fad0 con 0x7f0a0c159440 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0c070530 0x7f0a0c1ec000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0c070530 0x7f0a0c1ec000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49068/0 (socket says 172.21.15.1:49068) 2024-05-09T05:06:10.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 -- 172.21.15.1:0/2540616215 learned_addr learned my addr 172.21.15.1:0/2540616215 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a10c7d640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a0c153830 0x7f0a0c13c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a11c7f640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c13c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 -- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 msgr2=0x7f0a0c13c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c13c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 -- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a0c153830 msgr2=0x7f0a0c13c0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a0c153830 0x7f0a0c13c0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 -- 172.21.15.1:0/2540616215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a0c13cb30 con 0x7f0a0c070530 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a1147e640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0c070530 0x7f0a0c1ec000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a04035040 tx=0x7f0a040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a01ffb640 1 -- 172.21.15.1:0/2540616215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a04037040 con 0x7f0a0c070530 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a0c1ee0a0 con 0x7f0a0c070530 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a01ffb640 1 -- 172.21.15.1:0/2540616215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a04030500 con 0x7f0a0c070530 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.225+0000 7f0a01ffb640 1 -- 172.21.15.1:0/2540616215 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a040308d0 con 0x7f0a0c070530 2024-05-09T05:06:10.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.226+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a0c1ee660 con 0x7f0a0c070530 2024-05-09T05:06:10.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.226+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a0c050870 con 0x7f0a0c070530 2024-05-09T05:06:10.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.227+0000 7f0a01ffb640 1 -- 172.21.15.1:0/2540616215 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0a04044020 con 0x7f0a0c070530 2024-05-09T05:06:10.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.228+0000 7f0a01ffb640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09f00762d0 0x7f09f00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.228+0000 7f0a01ffb640 1 -- 172.21.15.1:0/2540616215 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0a040b7840 con 0x7f0a0c070530 2024-05-09T05:06:10.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.228+0000 7f0a10c7d640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09f00762d0 0x7f09f00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.229+0000 7f0a10c7d640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09f00762d0 0x7f09f00787e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c15b1a0 tx=0x7f09fc007cc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.233+0000 7f0a01ffb640 1 -- 172.21.15.1:0/2540616215 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0a0404b020 con 0x7f0a0c070530 2024-05-09T05:06:10.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.463+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0a0c0c94c0 con 0x7f0a0c070530 2024-05-09T05:06:10.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.463+0000 7f0a01ffb640 1 -- 172.21.15.1:0/2540616215 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+6848 (secure 0 0 0) 0x7f0a04081120 con 0x7f0a0c070530 2024-05-09T05:06:10.464 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:10.464 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":9,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4273,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":4287,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":4311,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":9,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:10.082334+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1],"up":{"mds_0":34160,"mds_1":34157},"failed":[],"damaged":[],"stopped":[],"info":{"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:creating","state_seq":1,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[34160]},"id":1}]} 2024-05-09T05:06:10.464 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 9 2024-05-09T05:06:10.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09f00762d0 msgr2=0x7f09f00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09f00762d0 0x7f09f00787e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0a0c15b1a0 tx=0x7f09fc007cc0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0c070530 msgr2=0x7f0a0c1ec000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0c070530 0x7f0a0c1ec000 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0a04035040 tx=0x7f0a040046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 shutdown_connections 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09f00762d0 0x7f09f00787e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a0c153830 0x7f0a0c13c0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a0c159440 0x7f0a0c13c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 --2- 172.21.15.1:0/2540616215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a0c070530 0x7f0a0c1ec000 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 >> 172.21.15.1:0/2540616215 conn(0x7f0a0c071b40 msgr2=0x7f0a0c06d050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 shutdown_connections 2024-05-09T05:06:10.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.466+0000 7f0a136e2640 1 -- 172.21.15.1:0/2540616215 wait complete. 2024-05-09T05:06:10.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:06:10.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.591+0000 7fb3039e3640 1 Processor -- start 2024-05-09T05:06:10.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.591+0000 7fb3039e3640 1 -- start start 2024-05-09T05:06:10.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb3039e3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2fc06efa0 0x7fb2fc156900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb3039e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 0x7fb2fc0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb3039e3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb2fc15fdf0 0x7fb2fc160200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb3039e3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb2fc06de20 con 0x7fb2fc06efa0 2024-05-09T05:06:10.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb3039e3640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb2fc15c9a0 con 0x7fb2fc15fdf0 2024-05-09T05:06:10.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb3039e3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb2fc0b51a0 con 0x7fb2fc0cc560 2024-05-09T05:06:10.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb30177f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2fc06efa0 0x7fb2fc156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb30177f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2fc06efa0 0x7fb2fc156900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49084/0 (socket says 172.21.15.1:49084) 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb30177f640 1 -- 172.21.15.1:0/3812547062 learned_addr learned my addr 172.21.15.1:0/3812547062 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb301f80640 1 --2- 172.21.15.1:0/3812547062 >> v2:172.21.15.196:3300/0 conn(0x7fb2fc15fdf0 0x7fb2fc160200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb300f7e640 1 --2- 172.21.15.1:0/3812547062 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 0x7fb2fc0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb30177f640 1 -- 172.21.15.1:0/3812547062 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 msgr2=0x7fb2fc0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb30177f640 1 --2- 172.21.15.1:0/3812547062 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 0x7fb2fc0cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb30177f640 1 -- 172.21.15.1:0/3812547062 >> v2:172.21.15.196:3300/0 conn(0x7fb2fc15fdf0 msgr2=0x7fb2fc160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb30177f640 1 --2- 172.21.15.1:0/3812547062 >> v2:172.21.15.196:3300/0 conn(0x7fb2fc15fdf0 0x7fb2fc160200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.593+0000 7fb30177f640 1 -- 172.21.15.1:0/3812547062 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2fc15d720 con 0x7fb2fc06efa0 2024-05-09T05:06:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.594+0000 7fb30177f640 1 --2- 172.21.15.1:0/3812547062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc156900 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc06d670 tx=0x7fb2ec02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7b625d8b041a98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.594+0000 7fb2fbfff640 1 -- 172.21.15.1:0/3812547062 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec036070 con 0x7fb2fc06efa0 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.594+0000 7fb2fbfff640 1 -- 172.21.15.1:0/3812547062 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2ec031600 con 0x7fb2fc06efa0 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.594+0000 7fb2fbfff640 1 -- 172.21.15.1:0/3812547062 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec0319d0 con 0x7fb2fc06efa0 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 -- 172.21.15.1:0/3812547062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 msgr2=0x7fb2fc156900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 --2- 172.21.15.1:0/3812547062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc156900 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc06d670 tx=0x7fb2ec02a730 comp rx=0 tx=0).stop 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 -- 172.21.15.1:0/3812547062 shutdown_connections 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 --2- 172.21.15.1:0/3812547062 >> v2:172.21.15.196:3300/0 conn(0x7fb2fc15fdf0 0x7fb2fc160200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 --2- 172.21.15.1:0/3812547062 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 0x7fb2fc0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 --2- 172.21.15.1:0/3812547062 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc156900 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 -- 172.21.15.1:0/3812547062 >> 172.21.15.1:0/3812547062 conn(0x7fb2fc068d90 msgr2=0x7fb2fc0d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 -- 172.21.15.1:0/3812547062 shutdown_connections 2024-05-09T05:06:10.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.595+0000 7fb3039e3640 1 -- 172.21.15.1:0/3812547062 wait complete. 2024-05-09T05:06:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.597+0000 7fb3039e3640 1 Processor -- start 2024-05-09T05:06:10.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.597+0000 7fb3039e3640 1 -- start start 2024-05-09T05:06:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb3039e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc0caa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb3039e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 0x7fb2fc153a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb3039e3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2fc15fdf0 0x7fb2fc0cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb3039e3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc15cf20 con 0x7fb2fc06efa0 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb3039e3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb2fc15cc40 con 0x7fb2fc15fdf0 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb3039e3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb2fc15d200 con 0x7fb2fc0cc560 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb30177f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc0caa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb30177f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc0caa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49096/0 (socket says 172.21.15.1:49096) 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb30177f640 1 -- 172.21.15.1:0/212543882 learned_addr learned my addr 172.21.15.1:0/212543882 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb300f7e640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 0x7fb2fc153a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb30177f640 1 -- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 msgr2=0x7fb2fc153a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb301f80640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2fc15fdf0 0x7fb2fc0cb490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb30177f640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 0x7fb2fc153a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb30177f640 1 -- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2fc15fdf0 msgr2=0x7fb2fc0cb490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb30177f640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2fc15fdf0 0x7fb2fc0cb490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.598+0000 7fb30177f640 1 -- 172.21.15.1:0/212543882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2fc153f70 con 0x7fb2fc06efa0 2024-05-09T05:06:10.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.599+0000 7fb30177f640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc0caa60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec000f80 tx=0x7fb2ec004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.599+0000 7fb2f9ffb640 1 -- 172.21.15.1:0/212543882 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec036050 con 0x7fb2fc06efa0 2024-05-09T05:06:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.599+0000 7fb2f9ffb640 1 -- 172.21.15.1:0/212543882 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2ec042070 con 0x7fb2fc06efa0 2024-05-09T05:06:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.599+0000 7fb2f9ffb640 1 -- 172.21.15.1:0/212543882 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2ec009510 con 0x7fb2fc06efa0 2024-05-09T05:06:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.599+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2fc1e89f0 con 0x7fb2fc06efa0 2024-05-09T05:06:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.599+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2fc0c9f80 con 0x7fb2fc06efa0 2024-05-09T05:06:10.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.600+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2fc0cafa0 con 0x7fb2fc06efa0 2024-05-09T05:06:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.601+0000 7fb2f9ffb640 1 -- 172.21.15.1:0/212543882 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb2ec007c70 con 0x7fb2fc06efa0 2024-05-09T05:06:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.601+0000 7fb2f9ffb640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2e80762d0 0x7fb2e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.601+0000 7fb2f9ffb640 1 -- 172.21.15.1:0/212543882 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb2ec0b8370 con 0x7fb2fc06efa0 2024-05-09T05:06:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.602+0000 7fb300f7e640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2e80762d0 0x7fb2e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.602+0000 7fb300f7e640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2e80762d0 0x7fb2e80787e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc156f50 tx=0x7fb2f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.606+0000 7fb2f9ffb640 1 -- 172.21.15.1:0/212543882 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb2ec081c30 con 0x7fb2fc06efa0 2024-05-09T05:06:10.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.798+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb2fc050870 con 0x7fb2fc06efa0 2024-05-09T05:06:10.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.799+0000 7fb2f9ffb640 1 -- 172.21.15.1:0/212543882 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7fb2ec081530 con 0x7fb2fc06efa0 2024-05-09T05:06:10.800 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:10.800 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:06:06.049536+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:06:10.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2e80762d0 msgr2=0x7fb2e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2e80762d0 0x7fb2e80787e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb2fc156f50 tx=0x7fb2f0008040 comp rx=0 tx=0).stop 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 msgr2=0x7fb2fc0caa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc0caa60 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec000f80 tx=0x7fb2ec004a40 comp rx=0 tx=0).stop 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 shutdown_connections 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2e80762d0 0x7fb2e80787e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2fc15fdf0 0x7fb2fc0cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2fc0cc560 0x7fb2fc153a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 --2- 172.21.15.1:0/212543882 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2fc06efa0 0x7fb2fc0caa60 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 >> 172.21.15.1:0/212543882 conn(0x7fb2fc068d90 msgr2=0x7fb2fc1558e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 shutdown_connections 2024-05-09T05:06:10.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.802+0000 7fb3039e3640 1 -- 172.21.15.1:0/212543882 wait complete. 2024-05-09T05:06:10.814 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T05:06:10.814 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:06:10.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.927+0000 7f16440ee640 1 Processor -- start 2024-05-09T05:06:10.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.928+0000 7f16440ee640 1 -- start start 2024-05-09T05:06:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f16440ee640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f163c156a80 0x7f163c06df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f16440ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c06e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f16440ee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f163c0c86a0 con 0x7f163c0c5ca0 2024-05-09T05:06:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f16440ee640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f163c0cb910 con 0x7f163c156a80 2024-05-09T05:06:10.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f16440ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f163c0c8980 con 0x7f163c156e90 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f1641e8a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f163c0c5ca0 0x7f163c0ca4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35488/0 (socket says 172.21.15.1:35488) 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f1641e8a640 1 -- 172.21.15.1:0/1726406289 learned_addr learned my addr 172.21.15.1:0/1726406289 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f164268b640 1 --2- 172.21.15.1:0/1726406289 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c06e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f1641689640 1 --2- 172.21.15.1:0/1726406289 >> v2:172.21.15.196:3300/0 conn(0x7f163c156a80 0x7f163c06df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f1640e88640 1 -- 172.21.15.1:0/1726406289 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3314074653 0 0) 0x7f163c0c86a0 con 0x7f163c0c5ca0 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.929+0000 7f1640e88640 1 -- 172.21.15.1:0/1726406289 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1628003520 con 0x7f163c0c5ca0 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.930+0000 7f164268b640 1 -- 172.21.15.1:0/1726406289 >> v2:172.21.15.196:3300/0 conn(0x7f163c156a80 msgr2=0x7f163c06df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.930+0000 7f1640e88640 1 -- 172.21.15.1:0/1726406289 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 695148058 0 0) 0x7f1628003520 con 0x7f163c0c5ca0 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.930+0000 7f164268b640 1 --2- 172.21.15.1:0/1726406289 >> v2:172.21.15.196:3300/0 conn(0x7f163c156a80 0x7f163c06df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.930+0000 7f164268b640 1 -- 172.21.15.1:0/1726406289 >> v1:172.21.15.1:6789/0 conn(0x7f163c0c5ca0 legacy=0x7f163c0ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.930+0000 7f164268b640 1 -- 172.21.15.1:0/1726406289 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c0c8c60 con 0x7f163c156e90 2024-05-09T05:06:10.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.930+0000 7f1641689640 1 --2- 172.21.15.1:0/1726406289 >> v2:172.21.15.196:3300/0 conn(0x7f163c156a80 0x7f163c06df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f164268b640 1 --2- 172.21.15.1:0/1726406289 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c06e460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1634009d90 tx=0x7f163402a680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ec98b9370af073b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f1640e88640 1 -- 172.21.15.1:0/1726406289 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16380016f0 con 0x7f163c156e90 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f1640e88640 1 -- 172.21.15.1:0/1726406289 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1634007540 con 0x7f163c156e90 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f1640e88640 1 -- 172.21.15.1:0/1726406289 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1634007910 con 0x7f163c156e90 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f16440ee640 1 -- 172.21.15.1:0/1726406289 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 msgr2=0x7f163c06e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f16440ee640 1 --2- 172.21.15.1:0/1726406289 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c06e460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f1634009d90 tx=0x7f163402a680 comp rx=0 tx=0).stop 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f16440ee640 1 -- 172.21.15.1:0/1726406289 shutdown_connections 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f16440ee640 1 --2- 172.21.15.1:0/1726406289 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c06e460 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f16440ee640 1 --2- 172.21.15.1:0/1726406289 >> v2:172.21.15.196:3300/0 conn(0x7f163c156a80 0x7f163c06df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.931+0000 7f16440ee640 1 -- 172.21.15.1:0/1726406289 >> 172.21.15.1:0/1726406289 conn(0x7f163c0745a0 msgr2=0x7f163c071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.932+0000 7f16440ee640 1 -- 172.21.15.1:0/1726406289 shutdown_connections 2024-05-09T05:06:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.932+0000 7f16440ee640 1 -- 172.21.15.1:0/1726406289 wait complete. 2024-05-09T05:06:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.933+0000 7f16440ee640 1 Processor -- start 2024-05-09T05:06:10.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.933+0000 7f16440ee640 1 -- start start 2024-05-09T05:06:10.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.934+0000 7f16440ee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c5ca0 0x7f163c15a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.934+0000 7f16440ee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f163c156a80 0x7f163c1e4f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.934+0000 7f16440ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c145580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.934+0000 7f16440ee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f163c0cad60 con 0x7f163c0c5ca0 2024-05-09T05:06:10.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f16440ee640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f163c0cb5a0 con 0x7f163c156a80 2024-05-09T05:06:10.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f16440ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f163c0c7750 con 0x7f163c156e90 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c5ca0 0x7f163c15a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c5ca0 0x7f163c15a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49102/0 (socket says 172.21.15.1:49102) 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 -- 172.21.15.1:0/4236041810 learned_addr learned my addr 172.21.15.1:0/4236041810 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641689640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f163c156a80 0x7f163c1e4f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f164268b640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c145580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 -- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 msgr2=0x7f163c145580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c145580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 -- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f163c156a80 msgr2=0x7f163c1e4f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f163c156a80 0x7f163c1e4f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 -- 172.21.15.1:0/4236041810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f163c15ad20 con 0x7f163c0c5ca0 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641689640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f163c156a80 0x7f163c1e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f1641e8a640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c5ca0 0x7f163c15a7e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f163800c4c0 tx=0x7f163800c980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f162e7fc640 1 -- 172.21.15.1:0/4236041810 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1638005ce0 con 0x7f163c0c5ca0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f162e7fc640 1 -- 172.21.15.1:0/4236041810 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1638007070 con 0x7f163c0c5ca0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f163c15cb90 con 0x7f163c0c5ca0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f162e7fc640 1 -- 172.21.15.1:0/4236041810 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1638007460 con 0x7f163c0c5ca0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.935+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f163c15ce30 con 0x7f163c0c5ca0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.936+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f163c0c9f50 con 0x7f163c0c5ca0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.937+0000 7f162e7fc640 1 -- 172.21.15.1:0/4236041810 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1638007600 con 0x7f163c0c5ca0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.937+0000 7f162e7fc640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16200762d0 0x7f16200787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.938+0000 7f162e7fc640 1 -- 172.21.15.1:0/4236041810 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f16380972b0 con 0x7f163c0c5ca0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.938+0000 7f1641689640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16200762d0 0x7f16200787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:10.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.938+0000 7f1641689640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16200762d0 0x7f16200787e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f163c1e6690 tx=0x7f1630008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:10.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:10.942+0000 7f162e7fc640 1 -- 172.21.15.1:0/4236041810 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f163c0c9f50 con 0x7f163c0c5ca0 2024-05-09T05:06:11.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.130+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f163c050870 con 0x7f16200762d0 2024-05-09T05:06:11.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.137+0000 7f162e7fc640 1 -- 172.21.15.1:0/4236041810 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251801 (secure 0 0 0) 0x7f163c050870 con 0x7f16200762d0 2024-05-09T05:06:11.142 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:11.144 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.140+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16200762d0 msgr2=0x7f16200787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.140+0000 7f16440ee640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16200762d0 0x7f16200787e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f163c1e6690 tx=0x7f1630008040 comp rx=0 tx=0).stop 2024-05-09T05:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c5ca0 msgr2=0x7f163c15a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c5ca0 0x7f163c15a7e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f163800c4c0 tx=0x7f163800c980 comp rx=0 tx=0).stop 2024-05-09T05:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 shutdown_connections 2024-05-09T05:06:11.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16200762d0 0x7f16200787e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f163c156a80 0x7f163c1e4f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f163c156e90 0x7f163c145580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 --2- 172.21.15.1:0/4236041810 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f163c0c5ca0 0x7f163c15a7e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 >> 172.21.15.1:0/4236041810 conn(0x7f163c0745a0 msgr2=0x7f163c06c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 shutdown_connections 2024-05-09T05:06:11.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.141+0000 7f16440ee640 1 -- 172.21.15.1:0/4236041810 wait complete. 2024-05-09T05:06:11.152 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-09T05:06:09.544442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239716,"kb_used_data":2744,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735004,"statfs":{"total":773094113280,"available":772848644096,"internally_reserved":0,"allocated":2809856,"data_stored":1743768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001371"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737187+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:06:06.737187+0000","last_peered":"2024-05-09T05:06:06.737187+0000","last_clean":"2024-05-09T05:06:06.737187+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:06:06.737187+0000","last_undegraded":"2024-05-09T05:06:06.737187+0000","last_fullsized":"2024-05-09T05:06:06.737187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058883+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:06:06.058883+0000","last_peered":"2024-05-09T05:06:06.058883+0000","last_clean":"2024-05-09T05:06:06.058883+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:06:06.058883+0000","last_undegraded":"2024-05-09T05:06:06.058883+0000","last_fullsized":"2024-05-09T05:06:06.058883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.056735+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:06:05.056735+0000","last_peered":"2024-05-09T05:06:05.056735+0000","last_clean":"2024-05-09T05:06:05.056735+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:06:05.056735+0000","last_undegraded":"2024-05-09T05:06:05.056735+0000","last_fullsized":"2024-05-09T05:06:05.056735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:02.244633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062625+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:06:06.062625+0000","last_peered":"2024-05-09T05:06:06.062625+0000","last_clean":"2024-05-09T05:06:06.062625+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:06:06.062625+0000","last_undegraded":"2024-05-09T05:06:06.062625+0000","last_fullsized":"2024-05-09T05:06:06.062625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737255+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:06:06.737255+0000","last_peered":"2024-05-09T05:06:06.737255+0000","last_clean":"2024-05-09T05:06:06.737255+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:06:06.737255+0000","last_undegraded":"2024-05-09T05:06:06.737255+0000","last_fullsized":"2024-05-09T05:06:06.737255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062081+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:06:06.062081+0000","last_peered":"2024-05-09T05:06:06.062081+0000","last_clean":"2024-05-09T05:06:06.062081+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:06:06.062081+0000","last_undegraded":"2024-05-09T05:06:06.062081+0000","last_fullsized":"2024-05-09T05:06:06.062081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736570+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:06:06.736570+0000","last_peered":"2024-05-09T05:06:06.736570+0000","last_clean":"2024-05-09T05:06:06.736570+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:06:06.736570+0000","last_undegraded":"2024-05-09T05:06:06.736570+0000","last_fullsized":"2024-05-09T05:06:06.736570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070328+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:06:05.070328+0000","last_peered":"2024-05-09T05:06:05.070328+0000","last_clean":"2024-05-09T05:06:05.070328+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:06:05.070328+0000","last_undegraded":"2024-05-09T05:06:05.070328+0000","last_fullsized":"2024-05-09T05:06:05.070328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737253+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:06:06.737253+0000","last_peered":"2024-05-09T05:06:06.737253+0000","last_clean":"2024-05-09T05:06:06.737253+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:06:06.737253+0000","last_undegraded":"2024-05-09T05:06:06.737253+0000","last_fullsized":"2024-05-09T05:06:06.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.068041+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:06:05.068041+0000","last_peered":"2024-05-09T05:06:05.068041+0000","last_clean":"2024-05-09T05:06:05.068041+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:06:05.068041+0000","last_undegraded":"2024-05-09T05:06:05.068041+0000","last_fullsized":"2024-05-09T05:06:05.068041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:13.268823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071169+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:06:05.071169+0000","last_peered":"2024-05-09T05:06:05.071169+0000","last_clean":"2024-05-09T05:06:05.071169+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:06:05.071169+0000","last_undegraded":"2024-05-09T05:06:05.071169+0000","last_fullsized":"2024-05-09T05:06:05.071169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:17:00.616212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.062528+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:06:05.062528+0000","last_peered":"2024-05-09T05:06:05.062528+0000","last_clean":"2024-05-09T05:06:05.062528+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:06:05.062528+0000","last_undegraded":"2024-05-09T05:06:05.062528+0000","last_fullsized":"2024-05-09T05:06:05.062528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:34.980447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070832+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:06:05.070832+0000","last_peered":"2024-05-09T05:06:05.070832+0000","last_clean":"2024-05-09T05:06:05.070832+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:06:05.070832+0000","last_undegraded":"2024-05-09T05:06:05.070832+0000","last_fullsized":"2024-05-09T05:06:05.070832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.069828+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:06:05.069828+0000","last_peered":"2024-05-09T05:06:05.069828+0000","last_clean":"2024-05-09T05:06:05.069828+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:06:05.069828+0000","last_undegraded":"2024-05-09T05:06:05.069828+0000","last_fullsized":"2024-05-09T05:06:05.069828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:39.984512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058421+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:06:06.058421+0000","last_peered":"2024-05-09T05:06:06.058421+0000","last_clean":"2024-05-09T05:06:06.058421+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:06:06.058421+0000","last_undegraded":"2024-05-09T05:06:06.058421+0000","last_fullsized":"2024-05-09T05:06:06.058421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062588+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:06:06.062588+0000","last_peered":"2024-05-09T05:06:06.062588+0000","last_clean":"2024-05-09T05:06:06.062588+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:06:06.062588+0000","last_undegraded":"2024-05-09T05:06:06.062588+0000","last_fullsized":"2024-05-09T05:06:06.062588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062864+0000","last_change":"2024-05-09T05:06:05.071898+0000","last_active":"2024-05-09T05:06:06.062864+0000","last_peered":"2024-05-09T05:06:06.062864+0000","last_clean":"2024-05-09T05:06:06.062864+0000","last_became_active":"2024-05-09T05:06:05.071618+0000","last_became_peered":"2024-05-09T05:06:05.071618+0000","last_unstale":"2024-05-09T05:06:06.062864+0000","last_undegraded":"2024-05-09T05:06:06.062864+0000","last_fullsized":"2024-05-09T05:06:06.062864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060053+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:06:06.060053+0000","last_peered":"2024-05-09T05:06:06.060053+0000","last_clean":"2024-05-09T05:06:06.060053+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:06:06.060053+0000","last_undegraded":"2024-05-09T05:06:06.060053+0000","last_fullsized":"2024-05-09T05:06:06.060053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737104+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:06:06.737104+0000","last_peered":"2024-05-09T05:06:06.737104+0000","last_clean":"2024-05-09T05:06:06.737104+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:06:06.737104+0000","last_undegraded":"2024-05-09T05:06:06.737104+0000","last_fullsized":"2024-05-09T05:06:06.737104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.065103+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:06:05.065103+0000","last_peered":"2024-05-09T05:06:05.065103+0000","last_clean":"2024-05-09T05:06:05.065103+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:06:05.065103+0000","last_undegraded":"2024-05-09T05:06:05.065103+0000","last_fullsized":"2024-05-09T05:06:05.065103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071326+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:06:05.071326+0000","last_peered":"2024-05-09T05:06:05.071326+0000","last_clean":"2024-05-09T05:06:05.071326+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:06:05.071326+0000","last_undegraded":"2024-05-09T05:06:05.071326+0000","last_fullsized":"2024-05-09T05:06:05.071326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:57.764402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062276+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:06:06.062276+0000","last_peered":"2024-05-09T05:06:06.062276+0000","last_clean":"2024-05-09T05:06:06.062276+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:06:06.062276+0000","last_undegraded":"2024-05-09T05:06:06.062276+0000","last_fullsized":"2024-05-09T05:06:06.062276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.065544+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:06:05.065544+0000","last_peered":"2024-05-09T05:06:05.065544+0000","last_clean":"2024-05-09T05:06:05.065544+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:06:05.065544+0000","last_undegraded":"2024-05-09T05:06:05.065544+0000","last_fullsized":"2024-05-09T05:06:05.065544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062254+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062254+0000","last_peered":"2024-05-09T05:06:06.062254+0000","last_clean":"2024-05-09T05:06:06.062254+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:06:06.062254+0000","last_undegraded":"2024-05-09T05:06:06.062254+0000","last_fullsized":"2024-05-09T05:06:06.062254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.065736+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:06:05.065736+0000","last_peered":"2024-05-09T05:06:05.065736+0000","last_clean":"2024-05-09T05:06:05.065736+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:06:05.065736+0000","last_undegraded":"2024-05-09T05:06:05.065736+0000","last_fullsized":"2024-05-09T05:06:05.065736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:30.304225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071222+0000","last_change":"2024-05-09T05:06:05.067878+0000","last_active":"2024-05-09T05:06:05.071222+0000","last_peered":"2024-05-09T05:06:05.071222+0000","last_clean":"2024-05-09T05:06:05.071222+0000","last_became_active":"2024-05-09T05:06:05.067426+0000","last_became_peered":"2024-05-09T05:06:05.067426+0000","last_unstale":"2024-05-09T05:06:05.071222+0000","last_undegraded":"2024-05-09T05:06:05.071222+0000","last_fullsized":"2024-05-09T05:06:05.071222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:12.846773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058528+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:06:06.058528+0000","last_peered":"2024-05-09T05:06:06.058528+0000","last_clean":"2024-05-09T05:06:06.058528+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:06:06.058528+0000","last_undegraded":"2024-05-09T05:06:06.058528+0000","last_fullsized":"2024-05-09T05:06:06.058528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.064836+0000","last_change":"2024-05-09T05:06:05.063359+0000","last_active":"2024-05-09T05:06:05.064836+0000","last_peered":"2024-05-09T05:06:05.064836+0000","last_clean":"2024-05-09T05:06:05.064836+0000","last_became_active":"2024-05-09T05:06:05.063026+0000","last_became_peered":"2024-05-09T05:06:05.063026+0000","last_unstale":"2024-05-09T05:06:05.064836+0000","last_undegraded":"2024-05-09T05:06:05.064836+0000","last_fullsized":"2024-05-09T05:06:05.064836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:08.985229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071350+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:06:05.071350+0000","last_peered":"2024-05-09T05:06:05.071350+0000","last_clean":"2024-05-09T05:06:05.071350+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:06:05.071350+0000","last_undegraded":"2024-05-09T05:06:05.071350+0000","last_fullsized":"2024-05-09T05:06:05.071350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:40.813676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062682+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:06:06.062682+0000","last_peered":"2024-05-09T05:06:06.062682+0000","last_clean":"2024-05-09T05:06:06.062682+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:06:06.062682+0000","last_undegraded":"2024-05-09T05:06:06.062682+0000","last_fullsized":"2024-05-09T05:06:06.062682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059766+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:06:06.059766+0000","last_peered":"2024-05-09T05:06:06.059766+0000","last_clean":"2024-05-09T05:06:06.059766+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:06:06.059766+0000","last_undegraded":"2024-05-09T05:06:06.059766+0000","last_fullsized":"2024-05-09T05:06:06.059766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.066232+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:06:05.066232+0000","last_peered":"2024-05-09T05:06:05.066232+0000","last_clean":"2024-05-09T05:06:05.066232+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:06:05.066232+0000","last_undegraded":"2024-05-09T05:06:05.066232+0000","last_fullsized":"2024-05-09T05:06:05.066232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.062551+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:06:05.062551+0000","last_peered":"2024-05-09T05:06:05.062551+0000","last_clean":"2024-05-09T05:06:05.062551+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:06:05.062551+0000","last_undegraded":"2024-05-09T05:06:05.062551+0000","last_fullsized":"2024-05-09T05:06:05.062551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:07.003282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.061970+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:06:05.061970+0000","last_peered":"2024-05-09T05:06:05.061970+0000","last_clean":"2024-05-09T05:06:05.061970+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:06:05.061970+0000","last_undegraded":"2024-05-09T05:06:05.061970+0000","last_fullsized":"2024-05-09T05:06:05.061970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070325+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:06:05.070325+0000","last_peered":"2024-05-09T05:06:05.070325+0000","last_clean":"2024-05-09T05:06:05.070325+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:06:05.070325+0000","last_undegraded":"2024-05-09T05:06:05.070325+0000","last_fullsized":"2024-05-09T05:06:05.070325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:35.116926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051588+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:06:05.051588+0000","last_peered":"2024-05-09T05:06:05.051588+0000","last_clean":"2024-05-09T05:06:05.051588+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:06:05.051588+0000","last_undegraded":"2024-05-09T05:06:05.051588+0000","last_fullsized":"2024-05-09T05:06:05.051588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.057973+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:06:05.057973+0000","last_peered":"2024-05-09T05:06:05.057973+0000","last_clean":"2024-05-09T05:06:05.057973+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:06:05.057973+0000","last_undegraded":"2024-05-09T05:06:05.057973+0000","last_fullsized":"2024-05-09T05:06:05.057973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.072122+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:06:05.072122+0000","last_peered":"2024-05-09T05:06:05.072122+0000","last_clean":"2024-05-09T05:06:05.072122+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:06:05.072122+0000","last_undegraded":"2024-05-09T05:06:05.072122+0000","last_fullsized":"2024-05-09T05:06:05.072122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:02:50.296743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062362+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:06:06.062362+0000","last_peered":"2024-05-09T05:06:06.062362+0000","last_clean":"2024-05-09T05:06:06.062362+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:06:06.062362+0000","last_undegraded":"2024-05-09T05:06:06.062362+0000","last_fullsized":"2024-05-09T05:06:06.062362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058418+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:06:06.058418+0000","last_peered":"2024-05-09T05:06:06.058418+0000","last_clean":"2024-05-09T05:06:06.058418+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:06:06.058418+0000","last_undegraded":"2024-05-09T05:06:06.058418+0000","last_fullsized":"2024-05-09T05:06:06.058418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051396+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:06:05.051396+0000","last_peered":"2024-05-09T05:06:05.051396+0000","last_clean":"2024-05-09T05:06:05.051396+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:06:05.051396+0000","last_undegraded":"2024-05-09T05:06:05.051396+0000","last_fullsized":"2024-05-09T05:06:05.051396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062383+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:06:06.062383+0000","last_peered":"2024-05-09T05:06:06.062383+0000","last_clean":"2024-05-09T05:06:06.062383+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:06:06.062383+0000","last_undegraded":"2024-05-09T05:06:06.062383+0000","last_fullsized":"2024-05-09T05:06:06.062383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062550+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:06:06.062550+0000","last_peered":"2024-05-09T05:06:06.062550+0000","last_clean":"2024-05-09T05:06:06.062550+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:06:06.062550+0000","last_undegraded":"2024-05-09T05:06:06.062550+0000","last_fullsized":"2024-05-09T05:06:06.062550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070033+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:06:05.070033+0000","last_peered":"2024-05-09T05:06:05.070033+0000","last_clean":"2024-05-09T05:06:05.070033+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:06:05.070033+0000","last_undegraded":"2024-05-09T05:06:05.070033+0000","last_fullsized":"2024-05-09T05:06:05.070033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:55.713350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058597+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:06:06.058597+0000","last_peered":"2024-05-09T05:06:06.058597+0000","last_clean":"2024-05-09T05:06:06.058597+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:06:06.058597+0000","last_undegraded":"2024-05-09T05:06:06.058597+0000","last_fullsized":"2024-05-09T05:06:06.058597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071559+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:06:05.071559+0000","last_peered":"2024-05-09T05:06:05.071559+0000","last_clean":"2024-05-09T05:06:05.071559+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:06:05.071559+0000","last_undegraded":"2024-05-09T05:06:05.071559+0000","last_fullsized":"2024-05-09T05:06:05.071559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:08.985229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051378+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:06:05.051378+0000","last_peered":"2024-05-09T05:06:05.051378+0000","last_clean":"2024-05-09T05:06:05.051378+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:06:05.051378+0000","last_undegraded":"2024-05-09T05:06:05.051378+0000","last_fullsized":"2024-05-09T05:06:05.051378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736736+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:06:06.736736+0000","last_peered":"2024-05-09T05:06:06.736736+0000","last_clean":"2024-05-09T05:06:06.736736+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:06:06.736736+0000","last_undegraded":"2024-05-09T05:06:06.736736+0000","last_fullsized":"2024-05-09T05:06:06.736736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.045970+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:06:05.045970+0000","last_peered":"2024-05-09T05:06:05.045970+0000","last_clean":"2024-05-09T05:06:05.045970+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:06:05.045970+0000","last_undegraded":"2024-05-09T05:06:05.045970+0000","last_fullsized":"2024-05-09T05:06:05.045970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071381+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:06:05.071381+0000","last_peered":"2024-05-09T05:06:05.071381+0000","last_clean":"2024-05-09T05:06:05.071381+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:06:05.071381+0000","last_undegraded":"2024-05-09T05:06:05.071381+0000","last_fullsized":"2024-05-09T05:06:05.071381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:52.904114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062613+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:06:06.062613+0000","last_peered":"2024-05-09T05:06:06.062613+0000","last_clean":"2024-05-09T05:06:06.062613+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:06:06.062613+0000","last_undegraded":"2024-05-09T05:06:06.062613+0000","last_fullsized":"2024-05-09T05:06:06.062613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737118+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:06:06.737118+0000","last_peered":"2024-05-09T05:06:06.737118+0000","last_clean":"2024-05-09T05:06:06.737118+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:06:06.737118+0000","last_undegraded":"2024-05-09T05:06:06.737118+0000","last_fullsized":"2024-05-09T05:06:06.737118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051510+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:06:05.051510+0000","last_peered":"2024-05-09T05:06:05.051510+0000","last_clean":"2024-05-09T05:06:05.051510+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:06:05.051510+0000","last_undegraded":"2024-05-09T05:06:05.051510+0000","last_fullsized":"2024-05-09T05:06:05.051510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.065322+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:06:05.065322+0000","last_peered":"2024-05-09T05:06:05.065322+0000","last_clean":"2024-05-09T05:06:05.065322+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:06:05.065322+0000","last_undegraded":"2024-05-09T05:06:05.065322+0000","last_fullsized":"2024-05-09T05:06:05.065322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059642+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:06:06.059642+0000","last_peered":"2024-05-09T05:06:06.059642+0000","last_clean":"2024-05-09T05:06:06.059642+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:06:06.059642+0000","last_undegraded":"2024-05-09T05:06:06.059642+0000","last_fullsized":"2024-05-09T05:06:06.059642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736921+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:06:06.736921+0000","last_peered":"2024-05-09T05:06:06.736921+0000","last_clean":"2024-05-09T05:06:06.736921+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:06:06.736921+0000","last_undegraded":"2024-05-09T05:06:06.736921+0000","last_fullsized":"2024-05-09T05:06:06.736921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.045606+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:06:05.045606+0000","last_peered":"2024-05-09T05:06:05.045606+0000","last_clean":"2024-05-09T05:06:05.045606+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:06:05.045606+0000","last_undegraded":"2024-05-09T05:06:05.045606+0000","last_fullsized":"2024-05-09T05:06:05.045606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737109+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:06:06.737109+0000","last_peered":"2024-05-09T05:06:06.737109+0000","last_clean":"2024-05-09T05:06:06.737109+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:06:06.737109+0000","last_undegraded":"2024-05-09T05:06:06.737109+0000","last_fullsized":"2024-05-09T05:06:06.737109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060165+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:06:06.060165+0000","last_peered":"2024-05-09T05:06:06.060165+0000","last_clean":"2024-05-09T05:06:06.060165+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:06:06.060165+0000","last_undegraded":"2024-05-09T05:06:06.060165+0000","last_fullsized":"2024-05-09T05:06:06.060165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737391+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:06:06.737391+0000","last_peered":"2024-05-09T05:06:06.737391+0000","last_clean":"2024-05-09T05:06:06.737391+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:06:06.737391+0000","last_undegraded":"2024-05-09T05:06:06.737391+0000","last_fullsized":"2024-05-09T05:06:06.737391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058644+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:06:06.058644+0000","last_peered":"2024-05-09T05:06:06.058644+0000","last_clean":"2024-05-09T05:06:06.058644+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:06:06.058644+0000","last_undegraded":"2024-05-09T05:06:06.058644+0000","last_fullsized":"2024-05-09T05:06:06.058644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:06:06.062865+0000","last_peered":"2024-05-09T05:06:06.062865+0000","last_clean":"2024-05-09T05:06:06.062865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:06:06.062865+0000","last_undegraded":"2024-05-09T05:06:06.062865+0000","last_fullsized":"2024-05-09T05:06:06.062865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060038+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:06:06.060038+0000","last_peered":"2024-05-09T05:06:06.060038+0000","last_clean":"2024-05-09T05:06:06.060038+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:06:06.060038+0000","last_undegraded":"2024-05-09T05:06:06.060038+0000","last_fullsized":"2024-05-09T05:06:06.060038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736885+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:06:06.736885+0000","last_peered":"2024-05-09T05:06:06.736885+0000","last_clean":"2024-05-09T05:06:06.736885+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:06:06.736885+0000","last_undegraded":"2024-05-09T05:06:06.736885+0000","last_fullsized":"2024-05-09T05:06:06.736885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736326+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:06:06.736326+0000","last_peered":"2024-05-09T05:06:06.736326+0000","last_clean":"2024-05-09T05:06:06.736326+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:06:06.736326+0000","last_undegraded":"2024-05-09T05:06:06.736326+0000","last_fullsized":"2024-05-09T05:06:06.736326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062158+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:06:06.062158+0000","last_peered":"2024-05-09T05:06:06.062158+0000","last_clean":"2024-05-09T05:06:06.062158+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:06:06.062158+0000","last_undegraded":"2024-05-09T05:06:06.062158+0000","last_fullsized":"2024-05-09T05:06:06.062158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070154+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:06:05.070154+0000","last_peered":"2024-05-09T05:06:05.070154+0000","last_clean":"2024-05-09T05:06:05.070154+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:06:05.070154+0000","last_undegraded":"2024-05-09T05:06:05.070154+0000","last_fullsized":"2024-05-09T05:06:05.070154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736248+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:06:06.736248+0000","last_peered":"2024-05-09T05:06:06.736248+0000","last_clean":"2024-05-09T05:06:06.736248+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:06:06.736248+0000","last_undegraded":"2024-05-09T05:06:06.736248+0000","last_fullsized":"2024-05-09T05:06:06.736248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070263+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:06:05.070263+0000","last_peered":"2024-05-09T05:06:05.070263+0000","last_clean":"2024-05-09T05:06:05.070263+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:06:05.070263+0000","last_undegraded":"2024-05-09T05:06:05.070263+0000","last_fullsized":"2024-05-09T05:06:05.070263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:11.571051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059887+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:06:06.059887+0000","last_peered":"2024-05-09T05:06:06.059887+0000","last_clean":"2024-05-09T05:06:06.059887+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:06:06.059887+0000","last_undegraded":"2024-05-09T05:06:06.059887+0000","last_fullsized":"2024-05-09T05:06:06.059887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058422+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:06:06.058422+0000","last_peered":"2024-05-09T05:06:06.058422+0000","last_clean":"2024-05-09T05:06:06.058422+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:06:06.058422+0000","last_undegraded":"2024-05-09T05:06:06.058422+0000","last_fullsized":"2024-05-09T05:06:06.058422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062842+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:06:06.062842+0000","last_peered":"2024-05-09T05:06:06.062842+0000","last_clean":"2024-05-09T05:06:06.062842+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:06:06.062842+0000","last_undegraded":"2024-05-09T05:06:06.062842+0000","last_fullsized":"2024-05-09T05:06:06.062842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.062168+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:06:05.062168+0000","last_peered":"2024-05-09T05:06:05.062168+0000","last_clean":"2024-05-09T05:06:05.062168+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:06:05.062168+0000","last_undegraded":"2024-05-09T05:06:05.062168+0000","last_fullsized":"2024-05-09T05:06:05.062168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:37:52.140346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058680+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:06:06.058680+0000","last_peered":"2024-05-09T05:06:06.058680+0000","last_clean":"2024-05-09T05:06:06.058680+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:06:06.058680+0000","last_undegraded":"2024-05-09T05:06:06.058680+0000","last_fullsized":"2024-05-09T05:06:06.058680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062852+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:06:06.062852+0000","last_peered":"2024-05-09T05:06:06.062852+0000","last_clean":"2024-05-09T05:06:06.062852+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:06:06.062852+0000","last_undegraded":"2024-05-09T05:06:06.062852+0000","last_fullsized":"2024-05-09T05:06:06.062852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737087+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:06:06.737087+0000","last_peered":"2024-05-09T05:06:06.737087+0000","last_clean":"2024-05-09T05:06:06.737087+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:06:06.737087+0000","last_undegraded":"2024-05-09T05:06:06.737087+0000","last_fullsized":"2024-05-09T05:06:06.737087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058622+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:06:06.058622+0000","last_peered":"2024-05-09T05:06:06.058622+0000","last_clean":"2024-05-09T05:06:06.058622+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:06:06.058622+0000","last_undegraded":"2024-05-09T05:06:06.058622+0000","last_fullsized":"2024-05-09T05:06:06.058622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058810+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:06:06.058810+0000","last_peered":"2024-05-09T05:06:06.058810+0000","last_clean":"2024-05-09T05:06:06.058810+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:06:06.058810+0000","last_undegraded":"2024-05-09T05:06:06.058810+0000","last_fullsized":"2024-05-09T05:06:06.058810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.072647+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:06:05.072647+0000","last_peered":"2024-05-09T05:06:05.072647+0000","last_clean":"2024-05-09T05:06:05.072647+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:06:05.072647+0000","last_undegraded":"2024-05-09T05:06:05.072647+0000","last_fullsized":"2024-05-09T05:06:05.072647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:17:00.616212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051890+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:06:05.051890+0000","last_peered":"2024-05-09T05:06:05.051890+0000","last_clean":"2024-05-09T05:06:05.051890+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:06:05.051890+0000","last_undegraded":"2024-05-09T05:06:05.051890+0000","last_fullsized":"2024-05-09T05:06:05.051890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.059446+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:06:05.059446+0000","last_peered":"2024-05-09T05:06:05.059446+0000","last_clean":"2024-05-09T05:06:05.059446+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:06:05.059446+0000","last_undegraded":"2024-05-09T05:06:05.059446+0000","last_fullsized":"2024-05-09T05:06:05.059446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:25.578206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736055+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:06:06.736055+0000","last_peered":"2024-05-09T05:06:06.736055+0000","last_clean":"2024-05-09T05:06:06.736055+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:06:06.736055+0000","last_undegraded":"2024-05-09T05:06:06.736055+0000","last_fullsized":"2024-05-09T05:06:06.736055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.062627+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:06:05.062627+0000","last_peered":"2024-05-09T05:06:05.062627+0000","last_clean":"2024-05-09T05:06:05.062627+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:06:05.062627+0000","last_undegraded":"2024-05-09T05:06:05.062627+0000","last_fullsized":"2024-05-09T05:06:05.062627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:19.319986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058525+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:06:06.058525+0000","last_peered":"2024-05-09T05:06:06.058525+0000","last_clean":"2024-05-09T05:06:06.058525+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:06:06.058525+0000","last_undegraded":"2024-05-09T05:06:06.058525+0000","last_fullsized":"2024-05-09T05:06:06.058525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070357+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:06:05.070357+0000","last_peered":"2024-05-09T05:06:05.070357+0000","last_clean":"2024-05-09T05:06:05.070357+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:06:05.070357+0000","last_undegraded":"2024-05-09T05:06:05.070357+0000","last_fullsized":"2024-05-09T05:06:05.070357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058700+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:06:06.058700+0000","last_peered":"2024-05-09T05:06:06.058700+0000","last_clean":"2024-05-09T05:06:06.058700+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:06:06.058700+0000","last_undegraded":"2024-05-09T05:06:06.058700+0000","last_fullsized":"2024-05-09T05:06:06.058700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.073102+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:06:05.073102+0000","last_peered":"2024-05-09T05:06:05.073102+0000","last_clean":"2024-05-09T05:06:05.073102+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:06:05.073102+0000","last_undegraded":"2024-05-09T05:06:05.073102+0000","last_fullsized":"2024-05-09T05:06:05.073102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:12.846773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737261+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:06:06.737261+0000","last_peered":"2024-05-09T05:06:06.737261+0000","last_clean":"2024-05-09T05:06:06.737261+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:06:06.737261+0000","last_undegraded":"2024-05-09T05:06:06.737261+0000","last_fullsized":"2024-05-09T05:06:06.737261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.067804+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:06:05.067804+0000","last_peered":"2024-05-09T05:06:05.067804+0000","last_clean":"2024-05-09T05:06:05.067804+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:06:05.067804+0000","last_undegraded":"2024-05-09T05:06:05.067804+0000","last_fullsized":"2024-05-09T05:06:05.067804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:02:50.296743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062736+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062736+0000","last_peered":"2024-05-09T05:06:06.062736+0000","last_clean":"2024-05-09T05:06:06.062736+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:06:06.062736+0000","last_undegraded":"2024-05-09T05:06:06.062736+0000","last_fullsized":"2024-05-09T05:06:06.062736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736771+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:06:06.736771+0000","last_peered":"2024-05-09T05:06:06.736771+0000","last_clean":"2024-05-09T05:06:06.736771+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:06:06.736771+0000","last_undegraded":"2024-05-09T05:06:06.736771+0000","last_fullsized":"2024-05-09T05:06:06.736771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062769+0000","last_change":"2024-05-09T05:06:05.063337+0000","last_active":"2024-05-09T05:06:06.062769+0000","last_peered":"2024-05-09T05:06:06.062769+0000","last_clean":"2024-05-09T05:06:06.062769+0000","last_became_active":"2024-05-09T05:06:05.063016+0000","last_became_peered":"2024-05-09T05:06:05.063016+0000","last_unstale":"2024-05-09T05:06:06.062769+0000","last_undegraded":"2024-05-09T05:06:06.062769+0000","last_fullsized":"2024-05-09T05:06:06.062769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736903+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:06:06.736903+0000","last_peered":"2024-05-09T05:06:06.736903+0000","last_clean":"2024-05-09T05:06:06.736903+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:06:06.736903+0000","last_undegraded":"2024-05-09T05:06:06.736903+0000","last_fullsized":"2024-05-09T05:06:06.736903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070063+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:06:05.070063+0000","last_peered":"2024-05-09T05:06:05.070063+0000","last_clean":"2024-05-09T05:06:05.070063+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:06:05.070063+0000","last_undegraded":"2024-05-09T05:06:05.070063+0000","last_fullsized":"2024-05-09T05:06:05.070063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:27.307574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.055610+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:06:05.055610+0000","last_peered":"2024-05-09T05:06:05.055610+0000","last_clean":"2024-05-09T05:06:05.055610+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:06:05.055610+0000","last_undegraded":"2024-05-09T05:06:05.055610+0000","last_fullsized":"2024-05-09T05:06:05.055610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070868+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:06:05.070868+0000","last_peered":"2024-05-09T05:06:05.070868+0000","last_clean":"2024-05-09T05:06:05.070868+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:06:05.070868+0000","last_undegraded":"2024-05-09T05:06:05.070868+0000","last_fullsized":"2024-05-09T05:06:05.070868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:13.268823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.045936+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:06:05.045936+0000","last_peered":"2024-05-09T05:06:05.045936+0000","last_clean":"2024-05-09T05:06:05.045936+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:06:05.045936+0000","last_undegraded":"2024-05-09T05:06:05.045936+0000","last_fullsized":"2024-05-09T05:06:05.045936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":663897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":73529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":67229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":67229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":196608,"data_stored":67229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":73529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":73529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36116,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335724,"statfs":{"total":96636764160,"available":96599781376,"internally_reserved":0,"allocated":790528,"data_stored":657597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:06:11.157 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:06:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.306+0000 7f5736270640 1 Processor -- start 2024-05-09T05:06:11.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.306+0000 7f5736270640 1 -- start start 2024-05-09T05:06:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.307+0000 7f5736270640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f573006efa0 0x7f5730156900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.307+0000 7f5736270640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f573006de20 con 0x7f57300cc970 2024-05-09T05:06:11.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.307+0000 7f5736270640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5730160c10 con 0x7f57300cc560 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.307+0000 7f5736270640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f573015c9a0 con 0x7f573006efa0 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f573480d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f57300cc970 0x7f5730160500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35490/0 (socket says 172.21.15.1:35490) 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f573480d640 1 -- 172.21.15.1:0/1920048616 learned_addr learned my addr 172.21.15.1:0/1920048616 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572ffff640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f573006efa0 0x7f5730156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2922968481 0 0) 0x7f573006de20 con 0x7f57300cc970 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f571c003520 con 0x7f57300cc970 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1938217924 0 0) 0x7f5730160c10 con 0x7f57300cc560 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f573006de20 con 0x7f57300cc560 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3138140187 0 0) 0x7f571c003520 con 0x7f57300cc970 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f573006efa0 msgr2=0x7f5730156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f573006efa0 0x7f5730156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 >> v1:172.21.15.196:6789/0 conn(0x7f57300cc560 legacy=0x7f573015fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57300b51a0 con 0x7f57300cc970 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f5728001a70 con 0x7f57300cc970 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 >> v1:172.21.15.1:6789/0 conn(0x7f57300cc970 legacy=0x7f5730160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c003c10 0x7f571c004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c004800 0x7f571c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f571c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f573015c9a0 con 0x7f571c005210 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f571c00c0f0 con 0x7f571c003c10 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f571c00c2f0 con 0x7f571c004800 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f572ffff640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f573006efa0 0x7f5730156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f573480d640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f571c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.308+0000 7f5727fff640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c004800 0x7f571c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.309+0000 7f572ffff640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c003c10 0x7f571c004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.309+0000 7f573480d640 1 -- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c004800 msgr2=0x7f571c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.309+0000 7f573480d640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c004800 0x7f571c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.309+0000 7f573480d640 1 -- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c003c10 msgr2=0x7f571c004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.309+0000 7f573480d640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c003c10 0x7f571c004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.309+0000 7f573480d640 1 -- 172.21.15.1:0/1920048616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571c003520 con 0x7f571c005210 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.309+0000 7f573480d640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f571c00b9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f572800a7f0 tx=0x7f571c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2cc3952785dd0522 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572800a450 con 0x7f571c005210 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f572802f070 con 0x7f571c005210 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f572f7fe640 1 -- 172.21.15.1:0/1920048616 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572802bba0 con 0x7f571c005210 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f5736270640 1 -- 172.21.15.1:0/1920048616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 msgr2=0x7f571c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f5736270640 1 --2- 172.21.15.1:0/1920048616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f571c00b9c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f572800a7f0 tx=0x7f571c005830 comp rx=0 tx=0).stop 2024-05-09T05:06:11.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f573480d640 1 -- 172.21.15.1:0/1920048616 reap_dead start 2024-05-09T05:06:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f5736270640 1 -- 172.21.15.1:0/1920048616 shutdown_connections 2024-05-09T05:06:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f5736270640 1 -- 172.21.15.1:0/1920048616 >> 172.21.15.1:0/1920048616 conn(0x7f5730068d90 msgr2=0x7f57300d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f5736270640 1 -- 172.21.15.1:0/1920048616 shutdown_connections 2024-05-09T05:06:11.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.310+0000 7f5736270640 1 -- 172.21.15.1:0/1920048616 wait complete. 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.312+0000 7f5736270640 1 Processor -- start 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.312+0000 7f5736270640 1 -- start start 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f5736270640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f57300cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f5736270640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f57301570f0 0x7f573014c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f5736270640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5730154320 0x7f57301494f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f5736270640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f573015cc40 con 0x7f571c005210 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f5736270640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f573006db40 con 0x7f5730154320 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f5736270640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f573015cf20 con 0x7f57301570f0 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f572ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f57300cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f572ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f57300cfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49108/0 (socket says 172.21.15.1:49108) 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.313+0000 7f572ffff640 1 -- 172.21.15.1:0/3460878184 learned_addr learned my addr 172.21.15.1:0/3460878184 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:11.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f573480d640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5730154320 0x7f57301494f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f5727fff640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f57301570f0 0x7f573014c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572ffff640 1 -- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f57301570f0 msgr2=0x7f573014c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572ffff640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f57301570f0 0x7f573014c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572ffff640 1 -- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5730154320 msgr2=0x7f57301494f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572ffff640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5730154320 0x7f57301494f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572ffff640 1 -- 172.21.15.1:0/3460878184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57300d0270 con 0x7f571c005210 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f573480d640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5730154320 0x7f57301494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f5727fff640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f57301570f0 0x7f573014c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572ffff640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f57300cfd30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5720011a30 tx=0x7f5720011ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572d7fa640 1 -- 172.21.15.1:0/3460878184 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f572000fc70 con 0x7f571c005210 2024-05-09T05:06:11.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572d7fa640 1 -- 172.21.15.1:0/3460878184 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f572000fe10 con 0x7f571c005210 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f572d7fa640 1 -- 172.21.15.1:0/3460878184 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5720014640 con 0x7f571c005210 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57300d0e70 con 0x7f571c005210 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.314+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57300d16f0 con 0x7f571c005210 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.315+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5730050870 con 0x7f571c005210 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.316+0000 7f572d7fa640 1 -- 172.21.15.1:0/3460878184 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5720012040 con 0x7f571c005210 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.316+0000 7f572d7fa640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5710076300 0x7f5710078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.316+0000 7f572d7fa640 1 -- 172.21.15.1:0/3460878184 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5720097d00 con 0x7f571c005210 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.317+0000 7f5727fff640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5710076300 0x7f5710078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.317+0000 7f5727fff640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5710076300 0x7f5710078810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5718000c00 tx=0x7f5718009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:11.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.321+0000 7f572d7fa640 1 -- 172.21.15.1:0/3460878184 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5720019030 con 0x7f571c005210 2024-05-09T05:06:11.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.509+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f573006ede0 con 0x7f5710076300 2024-05-09T05:06:11.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.516+0000 7f572d7fa640 1 -- 172.21.15.1:0/3460878184 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251801 (secure 0 0 0) 0x7f5718001520 con 0x7f5710076300 2024-05-09T05:06:11.519 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:11.520 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:06:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5710076300 msgr2=0x7f5710078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5710076300 0x7f5710078810 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f5718000c00 tx=0x7f5718009040 comp rx=0 tx=0).stop 2024-05-09T05:06:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 msgr2=0x7f57300cfd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f57300cfd30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f5720011a30 tx=0x7f5720011ef0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 shutdown_connections 2024-05-09T05:06:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5710076300 0x7f5710078810 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5730154320 0x7f57301494f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f57301570f0 0x7f573014c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 --2- 172.21.15.1:0/3460878184 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c005210 0x7f57300cfd30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 >> 172.21.15.1:0/3460878184 conn(0x7f5730068d90 msgr2=0x7f573015b570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:11.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 shutdown_connections 2024-05-09T05:06:11.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.519+0000 7f5736270640 1 -- 172.21.15.1:0/3460878184 wait complete. 2024-05-09T05:06:11.530 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":23,"stamp":"2024-05-09T05:06:09.544442+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":104,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239716,"kb_used_data":2744,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754735004,"statfs":{"total":773094113280,"available":772848644096,"internally_reserved":0,"allocated":2809856,"data_stored":1743768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":25,"apply_latency_ms":25,"commit_latency_ns":25000000,"apply_latency_ns":25000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001371"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737187+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:06:06.737187+0000","last_peered":"2024-05-09T05:06:06.737187+0000","last_clean":"2024-05-09T05:06:06.737187+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:06:06.737187+0000","last_undegraded":"2024-05-09T05:06:06.737187+0000","last_fullsized":"2024-05-09T05:06:06.737187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058883+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:06:06.058883+0000","last_peered":"2024-05-09T05:06:06.058883+0000","last_clean":"2024-05-09T05:06:06.058883+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:06:06.058883+0000","last_undegraded":"2024-05-09T05:06:06.058883+0000","last_fullsized":"2024-05-09T05:06:06.058883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.056735+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:06:05.056735+0000","last_peered":"2024-05-09T05:06:05.056735+0000","last_clean":"2024-05-09T05:06:05.056735+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:06:05.056735+0000","last_undegraded":"2024-05-09T05:06:05.056735+0000","last_fullsized":"2024-05-09T05:06:05.056735+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:02.244633+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062625+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:06:06.062625+0000","last_peered":"2024-05-09T05:06:06.062625+0000","last_clean":"2024-05-09T05:06:06.062625+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:06:06.062625+0000","last_undegraded":"2024-05-09T05:06:06.062625+0000","last_fullsized":"2024-05-09T05:06:06.062625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737255+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:06:06.737255+0000","last_peered":"2024-05-09T05:06:06.737255+0000","last_clean":"2024-05-09T05:06:06.737255+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:06:06.737255+0000","last_undegraded":"2024-05-09T05:06:06.737255+0000","last_fullsized":"2024-05-09T05:06:06.737255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062081+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:06:06.062081+0000","last_peered":"2024-05-09T05:06:06.062081+0000","last_clean":"2024-05-09T05:06:06.062081+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:06:06.062081+0000","last_undegraded":"2024-05-09T05:06:06.062081+0000","last_fullsized":"2024-05-09T05:06:06.062081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736570+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:06:06.736570+0000","last_peered":"2024-05-09T05:06:06.736570+0000","last_clean":"2024-05-09T05:06:06.736570+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:06:06.736570+0000","last_undegraded":"2024-05-09T05:06:06.736570+0000","last_fullsized":"2024-05-09T05:06:06.736570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070328+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:06:05.070328+0000","last_peered":"2024-05-09T05:06:05.070328+0000","last_clean":"2024-05-09T05:06:05.070328+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:06:05.070328+0000","last_undegraded":"2024-05-09T05:06:05.070328+0000","last_fullsized":"2024-05-09T05:06:05.070328+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737253+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:06:06.737253+0000","last_peered":"2024-05-09T05:06:06.737253+0000","last_clean":"2024-05-09T05:06:06.737253+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:06:06.737253+0000","last_undegraded":"2024-05-09T05:06:06.737253+0000","last_fullsized":"2024-05-09T05:06:06.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.068041+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:06:05.068041+0000","last_peered":"2024-05-09T05:06:05.068041+0000","last_clean":"2024-05-09T05:06:05.068041+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:06:05.068041+0000","last_undegraded":"2024-05-09T05:06:05.068041+0000","last_fullsized":"2024-05-09T05:06:05.068041+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:13.268823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071169+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:06:05.071169+0000","last_peered":"2024-05-09T05:06:05.071169+0000","last_clean":"2024-05-09T05:06:05.071169+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:06:05.071169+0000","last_undegraded":"2024-05-09T05:06:05.071169+0000","last_fullsized":"2024-05-09T05:06:05.071169+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:17:00.616212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.062528+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:06:05.062528+0000","last_peered":"2024-05-09T05:06:05.062528+0000","last_clean":"2024-05-09T05:06:05.062528+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:06:05.062528+0000","last_undegraded":"2024-05-09T05:06:05.062528+0000","last_fullsized":"2024-05-09T05:06:05.062528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:34.980447+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070832+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:06:05.070832+0000","last_peered":"2024-05-09T05:06:05.070832+0000","last_clean":"2024-05-09T05:06:05.070832+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:06:05.070832+0000","last_undegraded":"2024-05-09T05:06:05.070832+0000","last_fullsized":"2024-05-09T05:06:05.070832+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.069828+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:06:05.069828+0000","last_peered":"2024-05-09T05:06:05.069828+0000","last_clean":"2024-05-09T05:06:05.069828+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:06:05.069828+0000","last_undegraded":"2024-05-09T05:06:05.069828+0000","last_fullsized":"2024-05-09T05:06:05.069828+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:39.984512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058421+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:06:06.058421+0000","last_peered":"2024-05-09T05:06:06.058421+0000","last_clean":"2024-05-09T05:06:06.058421+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:06:06.058421+0000","last_undegraded":"2024-05-09T05:06:06.058421+0000","last_fullsized":"2024-05-09T05:06:06.058421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062588+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:06:06.062588+0000","last_peered":"2024-05-09T05:06:06.062588+0000","last_clean":"2024-05-09T05:06:06.062588+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:06:06.062588+0000","last_undegraded":"2024-05-09T05:06:06.062588+0000","last_fullsized":"2024-05-09T05:06:06.062588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062864+0000","last_change":"2024-05-09T05:06:05.071898+0000","last_active":"2024-05-09T05:06:06.062864+0000","last_peered":"2024-05-09T05:06:06.062864+0000","last_clean":"2024-05-09T05:06:06.062864+0000","last_became_active":"2024-05-09T05:06:05.071618+0000","last_became_peered":"2024-05-09T05:06:05.071618+0000","last_unstale":"2024-05-09T05:06:06.062864+0000","last_undegraded":"2024-05-09T05:06:06.062864+0000","last_fullsized":"2024-05-09T05:06:06.062864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060053+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:06:06.060053+0000","last_peered":"2024-05-09T05:06:06.060053+0000","last_clean":"2024-05-09T05:06:06.060053+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:06:06.060053+0000","last_undegraded":"2024-05-09T05:06:06.060053+0000","last_fullsized":"2024-05-09T05:06:06.060053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737104+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:06:06.737104+0000","last_peered":"2024-05-09T05:06:06.737104+0000","last_clean":"2024-05-09T05:06:06.737104+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:06:06.737104+0000","last_undegraded":"2024-05-09T05:06:06.737104+0000","last_fullsized":"2024-05-09T05:06:06.737104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.065103+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:06:05.065103+0000","last_peered":"2024-05-09T05:06:05.065103+0000","last_clean":"2024-05-09T05:06:05.065103+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:06:05.065103+0000","last_undegraded":"2024-05-09T05:06:05.065103+0000","last_fullsized":"2024-05-09T05:06:05.065103+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071326+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:06:05.071326+0000","last_peered":"2024-05-09T05:06:05.071326+0000","last_clean":"2024-05-09T05:06:05.071326+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:06:05.071326+0000","last_undegraded":"2024-05-09T05:06:05.071326+0000","last_fullsized":"2024-05-09T05:06:05.071326+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:57.764402+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062276+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:06:06.062276+0000","last_peered":"2024-05-09T05:06:06.062276+0000","last_clean":"2024-05-09T05:06:06.062276+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:06:06.062276+0000","last_undegraded":"2024-05-09T05:06:06.062276+0000","last_fullsized":"2024-05-09T05:06:06.062276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.065544+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:06:05.065544+0000","last_peered":"2024-05-09T05:06:05.065544+0000","last_clean":"2024-05-09T05:06:05.065544+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:06:05.065544+0000","last_undegraded":"2024-05-09T05:06:05.065544+0000","last_fullsized":"2024-05-09T05:06:05.065544+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062254+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062254+0000","last_peered":"2024-05-09T05:06:06.062254+0000","last_clean":"2024-05-09T05:06:06.062254+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:06:06.062254+0000","last_undegraded":"2024-05-09T05:06:06.062254+0000","last_fullsized":"2024-05-09T05:06:06.062254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.065736+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:06:05.065736+0000","last_peered":"2024-05-09T05:06:05.065736+0000","last_clean":"2024-05-09T05:06:05.065736+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:06:05.065736+0000","last_undegraded":"2024-05-09T05:06:05.065736+0000","last_fullsized":"2024-05-09T05:06:05.065736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:30.304225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071222+0000","last_change":"2024-05-09T05:06:05.067878+0000","last_active":"2024-05-09T05:06:05.071222+0000","last_peered":"2024-05-09T05:06:05.071222+0000","last_clean":"2024-05-09T05:06:05.071222+0000","last_became_active":"2024-05-09T05:06:05.067426+0000","last_became_peered":"2024-05-09T05:06:05.067426+0000","last_unstale":"2024-05-09T05:06:05.071222+0000","last_undegraded":"2024-05-09T05:06:05.071222+0000","last_fullsized":"2024-05-09T05:06:05.071222+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:12.846773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058528+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:06:06.058528+0000","last_peered":"2024-05-09T05:06:06.058528+0000","last_clean":"2024-05-09T05:06:06.058528+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:06:06.058528+0000","last_undegraded":"2024-05-09T05:06:06.058528+0000","last_fullsized":"2024-05-09T05:06:06.058528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.064836+0000","last_change":"2024-05-09T05:06:05.063359+0000","last_active":"2024-05-09T05:06:05.064836+0000","last_peered":"2024-05-09T05:06:05.064836+0000","last_clean":"2024-05-09T05:06:05.064836+0000","last_became_active":"2024-05-09T05:06:05.063026+0000","last_became_peered":"2024-05-09T05:06:05.063026+0000","last_unstale":"2024-05-09T05:06:05.064836+0000","last_undegraded":"2024-05-09T05:06:05.064836+0000","last_fullsized":"2024-05-09T05:06:05.064836+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:08.985229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071350+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:06:05.071350+0000","last_peered":"2024-05-09T05:06:05.071350+0000","last_clean":"2024-05-09T05:06:05.071350+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:06:05.071350+0000","last_undegraded":"2024-05-09T05:06:05.071350+0000","last_fullsized":"2024-05-09T05:06:05.071350+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:40.813676+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062682+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:06:06.062682+0000","last_peered":"2024-05-09T05:06:06.062682+0000","last_clean":"2024-05-09T05:06:06.062682+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:06:06.062682+0000","last_undegraded":"2024-05-09T05:06:06.062682+0000","last_fullsized":"2024-05-09T05:06:06.062682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059766+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:06:06.059766+0000","last_peered":"2024-05-09T05:06:06.059766+0000","last_clean":"2024-05-09T05:06:06.059766+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:06:06.059766+0000","last_undegraded":"2024-05-09T05:06:06.059766+0000","last_fullsized":"2024-05-09T05:06:06.059766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.066232+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:06:05.066232+0000","last_peered":"2024-05-09T05:06:05.066232+0000","last_clean":"2024-05-09T05:06:05.066232+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:06:05.066232+0000","last_undegraded":"2024-05-09T05:06:05.066232+0000","last_fullsized":"2024-05-09T05:06:05.066232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.062551+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:06:05.062551+0000","last_peered":"2024-05-09T05:06:05.062551+0000","last_clean":"2024-05-09T05:06:05.062551+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:06:05.062551+0000","last_undegraded":"2024-05-09T05:06:05.062551+0000","last_fullsized":"2024-05-09T05:06:05.062551+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:07.003282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.061970+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:06:05.061970+0000","last_peered":"2024-05-09T05:06:05.061970+0000","last_clean":"2024-05-09T05:06:05.061970+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:06:05.061970+0000","last_undegraded":"2024-05-09T05:06:05.061970+0000","last_fullsized":"2024-05-09T05:06:05.061970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070325+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:06:05.070325+0000","last_peered":"2024-05-09T05:06:05.070325+0000","last_clean":"2024-05-09T05:06:05.070325+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:06:05.070325+0000","last_undegraded":"2024-05-09T05:06:05.070325+0000","last_fullsized":"2024-05-09T05:06:05.070325+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:35.116926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051588+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:06:05.051588+0000","last_peered":"2024-05-09T05:06:05.051588+0000","last_clean":"2024-05-09T05:06:05.051588+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:06:05.051588+0000","last_undegraded":"2024-05-09T05:06:05.051588+0000","last_fullsized":"2024-05-09T05:06:05.051588+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.057973+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:06:05.057973+0000","last_peered":"2024-05-09T05:06:05.057973+0000","last_clean":"2024-05-09T05:06:05.057973+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:06:05.057973+0000","last_undegraded":"2024-05-09T05:06:05.057973+0000","last_fullsized":"2024-05-09T05:06:05.057973+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.072122+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:06:05.072122+0000","last_peered":"2024-05-09T05:06:05.072122+0000","last_clean":"2024-05-09T05:06:05.072122+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:06:05.072122+0000","last_undegraded":"2024-05-09T05:06:05.072122+0000","last_fullsized":"2024-05-09T05:06:05.072122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:02:50.296743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062362+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:06:06.062362+0000","last_peered":"2024-05-09T05:06:06.062362+0000","last_clean":"2024-05-09T05:06:06.062362+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:06:06.062362+0000","last_undegraded":"2024-05-09T05:06:06.062362+0000","last_fullsized":"2024-05-09T05:06:06.062362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058418+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:06:06.058418+0000","last_peered":"2024-05-09T05:06:06.058418+0000","last_clean":"2024-05-09T05:06:06.058418+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:06:06.058418+0000","last_undegraded":"2024-05-09T05:06:06.058418+0000","last_fullsized":"2024-05-09T05:06:06.058418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051396+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:06:05.051396+0000","last_peered":"2024-05-09T05:06:05.051396+0000","last_clean":"2024-05-09T05:06:05.051396+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:06:05.051396+0000","last_undegraded":"2024-05-09T05:06:05.051396+0000","last_fullsized":"2024-05-09T05:06:05.051396+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062383+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:06:06.062383+0000","last_peered":"2024-05-09T05:06:06.062383+0000","last_clean":"2024-05-09T05:06:06.062383+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:06:06.062383+0000","last_undegraded":"2024-05-09T05:06:06.062383+0000","last_fullsized":"2024-05-09T05:06:06.062383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062550+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:06:06.062550+0000","last_peered":"2024-05-09T05:06:06.062550+0000","last_clean":"2024-05-09T05:06:06.062550+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:06:06.062550+0000","last_undegraded":"2024-05-09T05:06:06.062550+0000","last_fullsized":"2024-05-09T05:06:06.062550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070033+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:06:05.070033+0000","last_peered":"2024-05-09T05:06:05.070033+0000","last_clean":"2024-05-09T05:06:05.070033+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:06:05.070033+0000","last_undegraded":"2024-05-09T05:06:05.070033+0000","last_fullsized":"2024-05-09T05:06:05.070033+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:55.713350+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058597+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:06:06.058597+0000","last_peered":"2024-05-09T05:06:06.058597+0000","last_clean":"2024-05-09T05:06:06.058597+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:06:06.058597+0000","last_undegraded":"2024-05-09T05:06:06.058597+0000","last_fullsized":"2024-05-09T05:06:06.058597+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071559+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:06:05.071559+0000","last_peered":"2024-05-09T05:06:05.071559+0000","last_clean":"2024-05-09T05:06:05.071559+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:06:05.071559+0000","last_undegraded":"2024-05-09T05:06:05.071559+0000","last_fullsized":"2024-05-09T05:06:05.071559+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:08.985229+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051378+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:06:05.051378+0000","last_peered":"2024-05-09T05:06:05.051378+0000","last_clean":"2024-05-09T05:06:05.051378+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:06:05.051378+0000","last_undegraded":"2024-05-09T05:06:05.051378+0000","last_fullsized":"2024-05-09T05:06:05.051378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736736+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:06:06.736736+0000","last_peered":"2024-05-09T05:06:06.736736+0000","last_clean":"2024-05-09T05:06:06.736736+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:06:06.736736+0000","last_undegraded":"2024-05-09T05:06:06.736736+0000","last_fullsized":"2024-05-09T05:06:06.736736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.045970+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:06:05.045970+0000","last_peered":"2024-05-09T05:06:05.045970+0000","last_clean":"2024-05-09T05:06:05.045970+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:06:05.045970+0000","last_undegraded":"2024-05-09T05:06:05.045970+0000","last_fullsized":"2024-05-09T05:06:05.045970+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.071381+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:06:05.071381+0000","last_peered":"2024-05-09T05:06:05.071381+0000","last_clean":"2024-05-09T05:06:05.071381+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:06:05.071381+0000","last_undegraded":"2024-05-09T05:06:05.071381+0000","last_fullsized":"2024-05-09T05:06:05.071381+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:52.904114+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062613+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:06:06.062613+0000","last_peered":"2024-05-09T05:06:06.062613+0000","last_clean":"2024-05-09T05:06:06.062613+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:06:06.062613+0000","last_undegraded":"2024-05-09T05:06:06.062613+0000","last_fullsized":"2024-05-09T05:06:06.062613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737118+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:06:06.737118+0000","last_peered":"2024-05-09T05:06:06.737118+0000","last_clean":"2024-05-09T05:06:06.737118+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:06:06.737118+0000","last_undegraded":"2024-05-09T05:06:06.737118+0000","last_fullsized":"2024-05-09T05:06:06.737118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051510+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:06:05.051510+0000","last_peered":"2024-05-09T05:06:05.051510+0000","last_clean":"2024-05-09T05:06:05.051510+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:06:05.051510+0000","last_undegraded":"2024-05-09T05:06:05.051510+0000","last_fullsized":"2024-05-09T05:06:05.051510+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.065322+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:06:05.065322+0000","last_peered":"2024-05-09T05:06:05.065322+0000","last_clean":"2024-05-09T05:06:05.065322+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:06:05.065322+0000","last_undegraded":"2024-05-09T05:06:05.065322+0000","last_fullsized":"2024-05-09T05:06:05.065322+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059642+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:06:06.059642+0000","last_peered":"2024-05-09T05:06:06.059642+0000","last_clean":"2024-05-09T05:06:06.059642+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:06:06.059642+0000","last_undegraded":"2024-05-09T05:06:06.059642+0000","last_fullsized":"2024-05-09T05:06:06.059642+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736921+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:06:06.736921+0000","last_peered":"2024-05-09T05:06:06.736921+0000","last_clean":"2024-05-09T05:06:06.736921+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:06:06.736921+0000","last_undegraded":"2024-05-09T05:06:06.736921+0000","last_fullsized":"2024-05-09T05:06:06.736921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.045606+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:06:05.045606+0000","last_peered":"2024-05-09T05:06:05.045606+0000","last_clean":"2024-05-09T05:06:05.045606+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:06:05.045606+0000","last_undegraded":"2024-05-09T05:06:05.045606+0000","last_fullsized":"2024-05-09T05:06:05.045606+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737109+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:06:06.737109+0000","last_peered":"2024-05-09T05:06:06.737109+0000","last_clean":"2024-05-09T05:06:06.737109+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:06:06.737109+0000","last_undegraded":"2024-05-09T05:06:06.737109+0000","last_fullsized":"2024-05-09T05:06:06.737109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060165+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:06:06.060165+0000","last_peered":"2024-05-09T05:06:06.060165+0000","last_clean":"2024-05-09T05:06:06.060165+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:06:06.060165+0000","last_undegraded":"2024-05-09T05:06:06.060165+0000","last_fullsized":"2024-05-09T05:06:06.060165+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737391+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:06:06.737391+0000","last_peered":"2024-05-09T05:06:06.737391+0000","last_clean":"2024-05-09T05:06:06.737391+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:06:06.737391+0000","last_undegraded":"2024-05-09T05:06:06.737391+0000","last_fullsized":"2024-05-09T05:06:06.737391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058644+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:06:06.058644+0000","last_peered":"2024-05-09T05:06:06.058644+0000","last_clean":"2024-05-09T05:06:06.058644+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:06:06.058644+0000","last_undegraded":"2024-05-09T05:06:06.058644+0000","last_fullsized":"2024-05-09T05:06:06.058644+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:06:06.062865+0000","last_peered":"2024-05-09T05:06:06.062865+0000","last_clean":"2024-05-09T05:06:06.062865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:06:06.062865+0000","last_undegraded":"2024-05-09T05:06:06.062865+0000","last_fullsized":"2024-05-09T05:06:06.062865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060038+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:06:06.060038+0000","last_peered":"2024-05-09T05:06:06.060038+0000","last_clean":"2024-05-09T05:06:06.060038+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:06:06.060038+0000","last_undegraded":"2024-05-09T05:06:06.060038+0000","last_fullsized":"2024-05-09T05:06:06.060038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736885+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:06:06.736885+0000","last_peered":"2024-05-09T05:06:06.736885+0000","last_clean":"2024-05-09T05:06:06.736885+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:06:06.736885+0000","last_undegraded":"2024-05-09T05:06:06.736885+0000","last_fullsized":"2024-05-09T05:06:06.736885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736326+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:06:06.736326+0000","last_peered":"2024-05-09T05:06:06.736326+0000","last_clean":"2024-05-09T05:06:06.736326+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:06:06.736326+0000","last_undegraded":"2024-05-09T05:06:06.736326+0000","last_fullsized":"2024-05-09T05:06:06.736326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062158+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:06:06.062158+0000","last_peered":"2024-05-09T05:06:06.062158+0000","last_clean":"2024-05-09T05:06:06.062158+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:06:06.062158+0000","last_undegraded":"2024-05-09T05:06:06.062158+0000","last_fullsized":"2024-05-09T05:06:06.062158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070154+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:06:05.070154+0000","last_peered":"2024-05-09T05:06:05.070154+0000","last_clean":"2024-05-09T05:06:05.070154+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:06:05.070154+0000","last_undegraded":"2024-05-09T05:06:05.070154+0000","last_fullsized":"2024-05-09T05:06:05.070154+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736248+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:06:06.736248+0000","last_peered":"2024-05-09T05:06:06.736248+0000","last_clean":"2024-05-09T05:06:06.736248+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:06:06.736248+0000","last_undegraded":"2024-05-09T05:06:06.736248+0000","last_fullsized":"2024-05-09T05:06:06.736248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070263+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:06:05.070263+0000","last_peered":"2024-05-09T05:06:05.070263+0000","last_clean":"2024-05-09T05:06:05.070263+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:06:05.070263+0000","last_undegraded":"2024-05-09T05:06:05.070263+0000","last_fullsized":"2024-05-09T05:06:05.070263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:11.571051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059887+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:06:06.059887+0000","last_peered":"2024-05-09T05:06:06.059887+0000","last_clean":"2024-05-09T05:06:06.059887+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:06:06.059887+0000","last_undegraded":"2024-05-09T05:06:06.059887+0000","last_fullsized":"2024-05-09T05:06:06.059887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058422+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:06:06.058422+0000","last_peered":"2024-05-09T05:06:06.058422+0000","last_clean":"2024-05-09T05:06:06.058422+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:06:06.058422+0000","last_undegraded":"2024-05-09T05:06:06.058422+0000","last_fullsized":"2024-05-09T05:06:06.058422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062842+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:06:06.062842+0000","last_peered":"2024-05-09T05:06:06.062842+0000","last_clean":"2024-05-09T05:06:06.062842+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:06:06.062842+0000","last_undegraded":"2024-05-09T05:06:06.062842+0000","last_fullsized":"2024-05-09T05:06:06.062842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.062168+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:06:05.062168+0000","last_peered":"2024-05-09T05:06:05.062168+0000","last_clean":"2024-05-09T05:06:05.062168+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:06:05.062168+0000","last_undegraded":"2024-05-09T05:06:05.062168+0000","last_fullsized":"2024-05-09T05:06:05.062168+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:37:52.140346+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058680+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:06:06.058680+0000","last_peered":"2024-05-09T05:06:06.058680+0000","last_clean":"2024-05-09T05:06:06.058680+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:06:06.058680+0000","last_undegraded":"2024-05-09T05:06:06.058680+0000","last_fullsized":"2024-05-09T05:06:06.058680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062852+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:06:06.062852+0000","last_peered":"2024-05-09T05:06:06.062852+0000","last_clean":"2024-05-09T05:06:06.062852+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:06:06.062852+0000","last_undegraded":"2024-05-09T05:06:06.062852+0000","last_fullsized":"2024-05-09T05:06:06.062852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737087+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:06:06.737087+0000","last_peered":"2024-05-09T05:06:06.737087+0000","last_clean":"2024-05-09T05:06:06.737087+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:06:06.737087+0000","last_undegraded":"2024-05-09T05:06:06.737087+0000","last_fullsized":"2024-05-09T05:06:06.737087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058622+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:06:06.058622+0000","last_peered":"2024-05-09T05:06:06.058622+0000","last_clean":"2024-05-09T05:06:06.058622+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:06:06.058622+0000","last_undegraded":"2024-05-09T05:06:06.058622+0000","last_fullsized":"2024-05-09T05:06:06.058622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058810+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:06:06.058810+0000","last_peered":"2024-05-09T05:06:06.058810+0000","last_clean":"2024-05-09T05:06:06.058810+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:06:06.058810+0000","last_undegraded":"2024-05-09T05:06:06.058810+0000","last_fullsized":"2024-05-09T05:06:06.058810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.072647+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:06:05.072647+0000","last_peered":"2024-05-09T05:06:05.072647+0000","last_clean":"2024-05-09T05:06:05.072647+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:06:05.072647+0000","last_undegraded":"2024-05-09T05:06:05.072647+0000","last_fullsized":"2024-05-09T05:06:05.072647+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:17:00.616212+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.051890+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:06:05.051890+0000","last_peered":"2024-05-09T05:06:05.051890+0000","last_clean":"2024-05-09T05:06:05.051890+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:06:05.051890+0000","last_undegraded":"2024-05-09T05:06:05.051890+0000","last_fullsized":"2024-05-09T05:06:05.051890+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.059446+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:06:05.059446+0000","last_peered":"2024-05-09T05:06:05.059446+0000","last_clean":"2024-05-09T05:06:05.059446+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:06:05.059446+0000","last_undegraded":"2024-05-09T05:06:05.059446+0000","last_fullsized":"2024-05-09T05:06:05.059446+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:25.578206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736055+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:06:06.736055+0000","last_peered":"2024-05-09T05:06:06.736055+0000","last_clean":"2024-05-09T05:06:06.736055+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:06:06.736055+0000","last_undegraded":"2024-05-09T05:06:06.736055+0000","last_fullsized":"2024-05-09T05:06:06.736055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.062627+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:06:05.062627+0000","last_peered":"2024-05-09T05:06:05.062627+0000","last_clean":"2024-05-09T05:06:05.062627+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:06:05.062627+0000","last_undegraded":"2024-05-09T05:06:05.062627+0000","last_fullsized":"2024-05-09T05:06:05.062627+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:19.319986+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058525+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:06:06.058525+0000","last_peered":"2024-05-09T05:06:06.058525+0000","last_clean":"2024-05-09T05:06:06.058525+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:06:06.058525+0000","last_undegraded":"2024-05-09T05:06:06.058525+0000","last_fullsized":"2024-05-09T05:06:06.058525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070357+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:06:05.070357+0000","last_peered":"2024-05-09T05:06:05.070357+0000","last_clean":"2024-05-09T05:06:05.070357+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:06:05.070357+0000","last_undegraded":"2024-05-09T05:06:05.070357+0000","last_fullsized":"2024-05-09T05:06:05.070357+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058700+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:06:06.058700+0000","last_peered":"2024-05-09T05:06:06.058700+0000","last_clean":"2024-05-09T05:06:06.058700+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:06:06.058700+0000","last_undegraded":"2024-05-09T05:06:06.058700+0000","last_fullsized":"2024-05-09T05:06:06.058700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.073102+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:06:05.073102+0000","last_peered":"2024-05-09T05:06:05.073102+0000","last_clean":"2024-05-09T05:06:05.073102+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:06:05.073102+0000","last_undegraded":"2024-05-09T05:06:05.073102+0000","last_fullsized":"2024-05-09T05:06:05.073102+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:12.846773+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737261+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:06:06.737261+0000","last_peered":"2024-05-09T05:06:06.737261+0000","last_clean":"2024-05-09T05:06:06.737261+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:06:06.737261+0000","last_undegraded":"2024-05-09T05:06:06.737261+0000","last_fullsized":"2024-05-09T05:06:06.737261+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.067804+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:06:05.067804+0000","last_peered":"2024-05-09T05:06:05.067804+0000","last_clean":"2024-05-09T05:06:05.067804+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:06:05.067804+0000","last_undegraded":"2024-05-09T05:06:05.067804+0000","last_fullsized":"2024-05-09T05:06:05.067804+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:02:50.296743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062736+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062736+0000","last_peered":"2024-05-09T05:06:06.062736+0000","last_clean":"2024-05-09T05:06:06.062736+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:06:06.062736+0000","last_undegraded":"2024-05-09T05:06:06.062736+0000","last_fullsized":"2024-05-09T05:06:06.062736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736771+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:06:06.736771+0000","last_peered":"2024-05-09T05:06:06.736771+0000","last_clean":"2024-05-09T05:06:06.736771+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:06:06.736771+0000","last_undegraded":"2024-05-09T05:06:06.736771+0000","last_fullsized":"2024-05-09T05:06:06.736771+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062769+0000","last_change":"2024-05-09T05:06:05.063337+0000","last_active":"2024-05-09T05:06:06.062769+0000","last_peered":"2024-05-09T05:06:06.062769+0000","last_clean":"2024-05-09T05:06:06.062769+0000","last_became_active":"2024-05-09T05:06:05.063016+0000","last_became_peered":"2024-05-09T05:06:05.063016+0000","last_unstale":"2024-05-09T05:06:06.062769+0000","last_undegraded":"2024-05-09T05:06:06.062769+0000","last_fullsized":"2024-05-09T05:06:06.062769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736903+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:06:06.736903+0000","last_peered":"2024-05-09T05:06:06.736903+0000","last_clean":"2024-05-09T05:06:06.736903+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:06:06.736903+0000","last_undegraded":"2024-05-09T05:06:06.736903+0000","last_fullsized":"2024-05-09T05:06:06.736903+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070063+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:06:05.070063+0000","last_peered":"2024-05-09T05:06:05.070063+0000","last_clean":"2024-05-09T05:06:05.070063+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:06:05.070063+0000","last_undegraded":"2024-05-09T05:06:05.070063+0000","last_fullsized":"2024-05-09T05:06:05.070063+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:27.307574+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.055610+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:06:05.055610+0000","last_peered":"2024-05-09T05:06:05.055610+0000","last_clean":"2024-05-09T05:06:05.055610+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:06:05.055610+0000","last_undegraded":"2024-05-09T05:06:05.055610+0000","last_fullsized":"2024-05-09T05:06:05.055610+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":9,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.070868+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:06:05.070868+0000","last_peered":"2024-05-09T05:06:05.070868+0000","last_clean":"2024-05-09T05:06:05.070868+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:06:05.070868+0000","last_undegraded":"2024-05-09T05:06:05.070868+0000","last_fullsized":"2024-05-09T05:06:05.070868+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:50:13.268823+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"0'0","reported_seq":12,"reported_epoch":20,"state":"active+clean","last_fresh":"2024-05-09T05:06:05.045936+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:06:05.045936+0000","last_peered":"2024-05-09T05:06:05.045936+0000","last_clean":"2024-05-09T05:06:05.045936+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:06:05.045936+0000","last_undegraded":"2024-05-09T05:06:05.045936+0000","last_fullsized":"2024-05-09T05:06:05.045936+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509444,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":784,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":802816,"data_stored":663897,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509445,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":208896,"data_stored":73529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542149,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":67229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542150,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27212,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344628,"statfs":{"total":96636764160,"available":96608899072,"internally_reserved":0,"allocated":196608,"data_stored":67229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542150,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31372,"kb_used_data":192,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340468,"statfs":{"total":96636764160,"available":96604639232,"internally_reserved":0,"allocated":196608,"data_stored":67229,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542151,"num_pgs":7,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":73529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542150,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":208896,"data_stored":73529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542149,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36116,"kb_used_data":772,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335724,"statfs":{"total":96636764160,"available":96599781376,"internally_reserved":0,"allocated":790528,"data_stored":657597,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:06:11.534 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T05:06:11.534 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-09T05:06:11.535 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2024-05-09T05:06:11.535 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2024-05-09T05:06:11.535 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-05-09T05:06:11.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.684+0000 7fc4d63b2640 1 Processor -- start 2024-05-09T05:06:11.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.684+0000 7fc4d63b2640 1 -- start start 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4d63b2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc4d015ca90 0x7fc4d015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4d63b2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4d006f6d0 con 0x7fc4d006f1b0 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4d63b2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4d00cc4d0 con 0x7fc4d00d16b0 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4d63b2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc4d00cee50 con 0x7fc4d015ca90 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4cf7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc4d006f1b0 0x7fc4d00ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35494/0 (socket says 172.21.15.1:35494) 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4cf7fe640 1 -- 172.21.15.1:0/35443932 learned_addr learned my addr 172.21.15.1:0/35443932 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4d494f640 1 --2- 172.21.15.1:0/35443932 >> v2:172.21.15.1:3300/0 conn(0x7fc4d015ca90 0x7fc4d015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4d494f640 1 -- 172.21.15.1:0/35443932 >> v1:172.21.15.196:6789/0 conn(0x7fc4d00d16b0 legacy=0x7fc4d00d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4d494f640 1 -- 172.21.15.1:0/35443932 >> v1:172.21.15.1:6789/0 conn(0x7fc4d006f1b0 legacy=0x7fc4d00ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.686+0000 7fc4d494f640 1 -- 172.21.15.1:0/35443932 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d00cc7b0 con 0x7fc4d015ca90 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.687+0000 7fc4d494f640 1 --2- 172.21.15.1:0/35443932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d015cea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009d90 tx=0x7fc4c802a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6afe2efeffbc5ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:11.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.687+0000 7fc4ceffd640 1 -- 172.21.15.1:0/35443932 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8037070 con 0x7fc4d015ca90 2024-05-09T05:06:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.687+0000 7fc4ceffd640 1 -- 172.21.15.1:0/35443932 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c8007740 con 0x7fc4d015ca90 2024-05-09T05:06:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.687+0000 7fc4ceffd640 1 -- 172.21.15.1:0/35443932 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8007b10 con 0x7fc4d015ca90 2024-05-09T05:06:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.688+0000 7fc4d63b2640 1 -- 172.21.15.1:0/35443932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 msgr2=0x7fc4d015cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.688+0000 7fc4d63b2640 1 --2- 172.21.15.1:0/35443932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d015cea0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009d90 tx=0x7fc4c802a780 comp rx=0 tx=0).stop 2024-05-09T05:06:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.688+0000 7fc4d63b2640 1 -- 172.21.15.1:0/35443932 shutdown_connections 2024-05-09T05:06:11.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.688+0000 7fc4d63b2640 1 --2- 172.21.15.1:0/35443932 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d015cea0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.688+0000 7fc4d63b2640 1 -- 172.21.15.1:0/35443932 >> 172.21.15.1:0/35443932 conn(0x7fc4d00b3660 msgr2=0x7fc4d00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:11.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.688+0000 7fc4d63b2640 1 -- 172.21.15.1:0/35443932 shutdown_connections 2024-05-09T05:06:11.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.688+0000 7fc4d63b2640 1 -- 172.21.15.1:0/35443932 wait complete. 2024-05-09T05:06:11.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.690+0000 7fc4d63b2640 1 Processor -- start 2024-05-09T05:06:11.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.690+0000 7fc4d63b2640 1 -- start start 2024-05-09T05:06:11.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d63b2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d006f1b0 0x7fc4d00ca810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d63b2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4d00d16b0 0x7fc4d0152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d63b2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d00c5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d63b2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d00ccd70 con 0x7fc4d015ca90 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d63b2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc4d00cd0e0 con 0x7fc4d006f1b0 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d63b2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc4d015d3e0 con 0x7fc4d00d16b0 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d494f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d00c5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d494f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d00c5300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49122/0 (socket says 172.21.15.1:49122) 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d494f640 1 -- 172.21.15.1:0/1939421539 learned_addr learned my addr 172.21.15.1:0/1939421539 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4cffff640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d006f1b0 0x7fc4d00ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4cf7fe640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4d00d16b0 0x7fc4d0152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d494f640 1 -- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4d00d16b0 msgr2=0x7fc4d0152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d494f640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4d00d16b0 0x7fc4d0152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d494f640 1 -- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d006f1b0 msgr2=0x7fc4d00ca810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d494f640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d006f1b0 0x7fc4d00ca810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4d494f640 1 -- 172.21.15.1:0/1939421539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4d00cad50 con 0x7fc4d015ca90 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.691+0000 7fc4cf7fe640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4d00d16b0 0x7fc4d0152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.692+0000 7fc4d494f640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d00c5300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009d90 tx=0x7fc4c8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.692+0000 7fc4ccff9640 1 -- 172.21.15.1:0/1939421539 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8042020 con 0x7fc4d015ca90 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.692+0000 7fc4ccff9640 1 -- 172.21.15.1:0/1939421539 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4c8032990 con 0x7fc4d015ca90 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.692+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4d00c6f70 con 0x7fc4d015ca90 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.692+0000 7fc4ccff9640 1 -- 172.21.15.1:0/1939421539 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4c8037040 con 0x7fc4d015ca90 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.692+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4d00c7530 con 0x7fc4d015ca90 2024-05-09T05:06:11.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.693+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4d01f6660 con 0x7fc4d015ca90 2024-05-09T05:06:11.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.694+0000 7fc4ccff9640 1 -- 172.21.15.1:0/1939421539 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc4c8045020 con 0x7fc4d015ca90 2024-05-09T05:06:11.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.694+0000 7fc4ccff9640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4bc0762d0 0x7fc4bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:11.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.694+0000 7fc4ccff9640 1 -- 172.21.15.1:0/1939421539 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc4c80b88e0 con 0x7fc4d015ca90 2024-05-09T05:06:11.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.695+0000 7fc4cffff640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4bc0762d0 0x7fc4bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:11.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.695+0000 7fc4cffff640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4bc0762d0 0x7fc4bc0787e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0000f60 tx=0x7fc4c0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:11.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.699+0000 7fc4ccff9640 1 -- 172.21.15.1:0/1939421539 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc4c804c030 con 0x7fc4d015ca90 2024-05-09T05:06:11.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.942+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fc4d0050870 con 0x7fc4d015ca90 2024-05-09T05:06:11.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.945+0000 7fc4ccff9640 1 -- 172.21.15.1:0/1939421539 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v5) ==== 74+0+189527 (secure 0 0 0) 0x7fc4c80821b0 con 0x7fc4d015ca90 2024-05-09T05:06:11.946 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:11.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4bc0762d0 msgr2=0x7fc4bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4bc0762d0 0x7fc4bc0787e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc4c0000f60 tx=0x7fc4c0009040 comp rx=0 tx=0).stop 2024-05-09T05:06:11.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 msgr2=0x7fc4d00c5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:11.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d00c5300 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc4c8009d90 tx=0x7fc4c8004a40 comp rx=0 tx=0).stop 2024-05-09T05:06:11.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 shutdown_connections 2024-05-09T05:06:11.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4bc0762d0 0x7fc4bc0787e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4d015ca90 0x7fc4d00c5300 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4d00d16b0 0x7fc4d0152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 --2- 172.21.15.1:0/1939421539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4d006f1b0 0x7fc4d00ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:11.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.947+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 >> 172.21.15.1:0/1939421539 conn(0x7fc4d00b3660 msgr2=0x7fc4d015b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:11.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.948+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 shutdown_connections 2024-05-09T05:06:11.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:11.948+0000 7fc4d63b2640 1 -- 172.21.15.1:0/1939421539 wait complete. 2024-05-09T05:06:11.959 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6832","nonce":1287021241},{"type":"v1","addr":"172.21.15.196:6833","nonce":1287021241}]},"active_addr":"172.21.15.196:6833/1287021241","active_change":"2024-05-09T05:05:45.529418+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":4113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"smb","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"update_orchestration":{"name":"update_orchestration","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically update orchestration when smb resources are changed","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sqlite3_killpoint":{"name":"sqlite3_killpoint","type":"int","level":"dev","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":2051542353}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":4096251208}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":461170790}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.196:0","nonce":3751848845}]}]} 2024-05-09T05:06:11.963 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2024-05-09T05:06:11.963 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2024-05-09T05:06:11.963 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:06:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.079+0000 7f89f2de0640 1 Processor -- start 2024-05-09T05:06:12.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.079+0000 7f89f2de0640 1 -- start start 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f2de0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89ec0c7a40 0x7f89ec0c7e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f2de0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89ec06de20 con 0x7f89ec06efa0 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f2de0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f89ec15c9a0 con 0x7f89ec156900 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f2de0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f89ec156d10 con 0x7f89ec0c7a40 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f137d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89ec0c7a40 0x7f89ec0c7e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f0b7c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f89ec06efa0 0x7f89ec0cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35510/0 (socket says 172.21.15.1:35510) 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f137d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f89ec0c7a40 0x7f89ec0c7e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49128/0 (socket says 172.21.15.1:49128) 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f0b7c640 1 -- 172.21.15.1:0/1120398619 learned_addr learned my addr 172.21.15.1:0/1120398619 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f137d640 1 -- 172.21.15.1:0/1120398619 >> v1:172.21.15.196:6789/0 conn(0x7f89ec156900 legacy=0x7f89ec0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f137d640 1 -- 172.21.15.1:0/1120398619 >> v1:172.21.15.1:6789/0 conn(0x7f89ec06efa0 legacy=0x7f89ec0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.082+0000 7f89f137d640 1 -- 172.21.15.1:0/1120398619 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89ec0b51a0 con 0x7f89ec0c7a40 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.083+0000 7f89f137d640 1 --2- 172.21.15.1:0/1120398619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec0c7a40 0x7f89ec0c7e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009d90 tx=0x7f89e402a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3232b5daf5b85918 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.083+0000 7f89eb7fe640 1 -- 172.21.15.1:0/1120398619 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4037070 con 0x7f89ec0c7a40 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.083+0000 7f89eb7fe640 1 -- 172.21.15.1:0/1120398619 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89e4007740 con 0x7f89ec0c7a40 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.083+0000 7f89eb7fe640 1 -- 172.21.15.1:0/1120398619 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e4007b10 con 0x7f89ec0c7a40 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.084+0000 7f89f2de0640 1 -- 172.21.15.1:0/1120398619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec0c7a40 msgr2=0x7f89ec0c7e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.084+0000 7f89f2de0640 1 --2- 172.21.15.1:0/1120398619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec0c7a40 0x7f89ec0c7e50 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f89e4009d90 tx=0x7f89e402a780 comp rx=0 tx=0).stop 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.084+0000 7f89f2de0640 1 -- 172.21.15.1:0/1120398619 shutdown_connections 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.084+0000 7f89f2de0640 1 --2- 172.21.15.1:0/1120398619 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec0c7a40 0x7f89ec0c7e50 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.084+0000 7f89f2de0640 1 -- 172.21.15.1:0/1120398619 >> 172.21.15.1:0/1120398619 conn(0x7f89ec068d90 msgr2=0x7f89ec0d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.084+0000 7f89f2de0640 1 -- 172.21.15.1:0/1120398619 shutdown_connections 2024-05-09T05:06:12.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.084+0000 7f89f2de0640 1 -- 172.21.15.1:0/1120398619 wait complete. 2024-05-09T05:06:12.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.086+0000 7f89f2de0640 1 Processor -- start 2024-05-09T05:06:12.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.086+0000 7f89f2de0640 1 -- start start 2024-05-09T05:06:12.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f2de0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec06efa0 0x7f89ec1e68b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f2de0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89ec0c7a40 0x7f89ec159a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f2de0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f89ec156900 0x7f89ec150010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f2de0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f89ec15cc40 con 0x7f89ec06efa0 2024-05-09T05:06:12.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f2de0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f89ec06db40 con 0x7f89ec0c7a40 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f2de0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f89ec15cf20 con 0x7f89ec156900 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f0b7c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec06efa0 0x7f89ec1e68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f0b7c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec06efa0 0x7f89ec1e68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49134/0 (socket says 172.21.15.1:49134) 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.087+0000 7f89f0b7c640 1 -- 172.21.15.1:0/2115603473 learned_addr learned my addr 172.21.15.1:0/2115603473 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f137d640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f89ec156900 0x7f89ec150010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89ebfff640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89ec0c7a40 0x7f89ec159a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f0b7c640 1 -- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f89ec156900 msgr2=0x7f89ec150010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f0b7c640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f89ec156900 0x7f89ec150010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f0b7c640 1 -- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89ec0c7a40 msgr2=0x7f89ec159a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f0b7c640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89ec0c7a40 0x7f89ec159a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f0b7c640 1 -- 172.21.15.1:0/2115603473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89ec1e6df0 con 0x7f89ec06efa0 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f0b7c640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec06efa0 0x7f89ec1e68b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f89e000c400 tx=0x7f89e000c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89e97fa640 1 -- 172.21.15.1:0/2115603473 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e0017070 con 0x7f89ec06efa0 2024-05-09T05:06:12.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89e97fa640 1 -- 172.21.15.1:0/2115603473 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89e0005ce0 con 0x7f89ec06efa0 2024-05-09T05:06:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89ec1f3430 con 0x7f89ec06efa0 2024-05-09T05:06:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89e97fa640 1 -- 172.21.15.1:0/2115603473 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e00072d0 con 0x7f89ec06efa0 2024-05-09T05:06:12.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.088+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89ec1e60b0 con 0x7f89ec06efa0 2024-05-09T05:06:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.089+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89ec1e5bc0 con 0x7f89ec06efa0 2024-05-09T05:06:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.090+0000 7f89e97fa640 1 -- 172.21.15.1:0/2115603473 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f89e001a070 con 0x7f89ec06efa0 2024-05-09T05:06:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.091+0000 7f89e97fa640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89d00762d0 0x7f89d00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.091+0000 7f89e97fa640 1 -- 172.21.15.1:0/2115603473 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f89e00968a0 con 0x7f89ec06efa0 2024-05-09T05:06:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.091+0000 7f89ebfff640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89d00762d0 0x7f89d00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.091+0000 7f89ebfff640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89d00762d0 0x7f89d00787e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f89ec160c60 tx=0x7f89d8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.095+0000 7f89e97fa640 1 -- 172.21.15.1:0/2115603473 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f89e005fbb0 con 0x7f89ec06efa0 2024-05-09T05:06:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.286+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f89ec050870 con 0x7f89ec06efa0 2024-05-09T05:06:12.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.287+0000 7f89e97fa640 1 -- 172.21.15.1:0/2115603473 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7f89e0063eb0 con 0x7f89ec06efa0 2024-05-09T05:06:12.288 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:12.288 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:06:06.049536+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:06:12.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89d00762d0 msgr2=0x7f89d00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89d00762d0 0x7f89d00787e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f89ec160c60 tx=0x7f89d8009040 comp rx=0 tx=0).stop 2024-05-09T05:06:12.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec06efa0 msgr2=0x7f89ec1e68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec06efa0 0x7f89ec1e68b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f89e000c400 tx=0x7f89e000c8c0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 shutdown_connections 2024-05-09T05:06:12.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89d00762d0 0x7f89d00787e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f89ec0c7a40 0x7f89ec159a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f89ec156900 0x7f89ec150010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 --2- 172.21.15.1:0/2115603473 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f89ec06efa0 0x7f89ec1e68b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 >> 172.21.15.1:0/2115603473 conn(0x7f89ec068d90 msgr2=0x7f89ec0cc230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 shutdown_connections 2024-05-09T05:06:12.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.289+0000 7f89f2de0640 1 -- 172.21.15.1:0/2115603473 wait complete. 2024-05-09T05:06:12.301 INFO:tasks.ceph.ceph_manager.ceph:all up! 2024-05-09T05:06:12.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:06:12.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.415+0000 7efc67927640 1 Processor -- start 2024-05-09T05:06:12.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.415+0000 7efc67927640 1 -- start start 2024-05-09T05:06:12.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.416+0000 7efc67927640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efc60072120 0x7efc60156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.416+0000 7efc67927640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc600c5e00 con 0x7efc6015d1d0 2024-05-09T05:06:12.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.416+0000 7efc67927640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efc600c60e0 con 0x7efc600caab0 2024-05-09T05:06:12.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.416+0000 7efc67927640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7efc600ca340 con 0x7efc60072120 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.416+0000 7efc64ec2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efc6015d1d0 0x7efc60070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35520/0 (socket says 172.21.15.1:35520) 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.416+0000 7efc64ec2640 1 -- 172.21.15.1:0/2228402961 learned_addr learned my addr 172.21.15.1:0/2228402961 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.417+0000 7efc656c3640 1 --2- 172.21.15.1:0/2228402961 >> v2:172.21.15.196:3300/0 conn(0x7efc60072120 0x7efc60156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.417+0000 7efc57fff640 1 -- 172.21.15.1:0/2228402961 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2025853485 0 0) 0x7efc600c5e00 con 0x7efc6015d1d0 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.417+0000 7efc57fff640 1 -- 172.21.15.1:0/2228402961 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efc4c003520 con 0x7efc6015d1d0 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.417+0000 7efc656c3640 1 -- 172.21.15.1:0/2228402961 >> v1:172.21.15.196:6789/0 conn(0x7efc600caab0 legacy=0x7efc600caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.417+0000 7efc656c3640 1 -- 172.21.15.1:0/2228402961 >> v1:172.21.15.1:6789/0 conn(0x7efc6015d1d0 legacy=0x7efc60070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.417+0000 7efc57fff640 1 -- 172.21.15.1:0/2228402961 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 372030730 0 0) 0x7efc4c003520 con 0x7efc6015d1d0 2024-05-09T05:06:12.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.417+0000 7efc656c3640 1 -- 172.21.15.1:0/2228402961 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc600c63c0 con 0x7efc60072120 2024-05-09T05:06:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.418+0000 7efc656c3640 1 --2- 172.21.15.1:0/2228402961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc60072120 0x7efc60156a80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0097a0 tx=0x7efc5c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=58048c67c52b2ef2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.418+0000 7efc57fff640 1 -- 172.21.15.1:0/2228402961 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c037070 con 0x7efc60072120 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.418+0000 7efc57fff640 1 -- 172.21.15.1:0/2228402961 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc5c02ac50 con 0x7efc60072120 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.418+0000 7efc57fff640 1 -- 172.21.15.1:0/2228402961 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc500016f0 con 0x7efc60072120 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.418+0000 7efc67927640 1 -- 172.21.15.1:0/2228402961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc60072120 msgr2=0x7efc60156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.418+0000 7efc67927640 1 --2- 172.21.15.1:0/2228402961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc60072120 0x7efc60156a80 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0097a0 tx=0x7efc5c02a1c0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.419+0000 7efc67927640 1 -- 172.21.15.1:0/2228402961 shutdown_connections 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.419+0000 7efc67927640 1 --2- 172.21.15.1:0/2228402961 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc60072120 0x7efc60156a80 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.419+0000 7efc67927640 1 -- 172.21.15.1:0/2228402961 >> 172.21.15.1:0/2228402961 conn(0x7efc600731d0 msgr2=0x7efc600cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.419+0000 7efc67927640 1 -- 172.21.15.1:0/2228402961 shutdown_connections 2024-05-09T05:06:12.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.419+0000 7efc67927640 1 -- 172.21.15.1:0/2228402961 wait complete. 2024-05-09T05:06:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.420+0000 7efc67927640 1 Processor -- start 2024-05-09T05:06:12.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.420+0000 7efc67927640 1 -- start start 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc67927640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc60072120 0x7efc601e7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc67927640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc600caab0 0x7efc600cecd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc67927640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc6015d1d0 0x7efc600cf210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc67927640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efc6015d5e0 con 0x7efc60072120 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc67927640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efc600ca620 con 0x7efc6015d1d0 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc67927640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efc60071530 con 0x7efc600caab0 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc60072120 0x7efc601e7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc60072120 0x7efc601e7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49146/0 (socket says 172.21.15.1:49146) 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 -- 172.21.15.1:0/4110984962 learned_addr learned my addr 172.21.15.1:0/4110984962 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc64ec2640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc600caab0 0x7efc600cecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc65ec4640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc6015d1d0 0x7efc600cf210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 -- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc600caab0 msgr2=0x7efc600cecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc600caab0 0x7efc600cecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 -- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc6015d1d0 msgr2=0x7efc600cf210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc6015d1d0 0x7efc600cf210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 -- 172.21.15.1:0/4110984962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc601e8330 con 0x7efc60072120 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.422+0000 7efc656c3640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc60072120 0x7efc601e7df0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0098c0 tx=0x7efc5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.423+0000 7efc55ffb640 1 -- 172.21.15.1:0/4110984962 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c041020 con 0x7efc60072120 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.423+0000 7efc55ffb640 1 -- 172.21.15.1:0/4110984962 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc5c0044c0 con 0x7efc60072120 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.423+0000 7efc55ffb640 1 -- 172.21.15.1:0/4110984962 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc5c037040 con 0x7efc60072120 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.423+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc601f31c0 con 0x7efc60072120 2024-05-09T05:06:12.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.423+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc601e75f0 con 0x7efc60072120 2024-05-09T05:06:12.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.424+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc60050870 con 0x7efc60072120 2024-05-09T05:06:12.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.424+0000 7efc55ffb640 1 -- 172.21.15.1:0/4110984962 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efc5c044020 con 0x7efc60072120 2024-05-09T05:06:12.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.425+0000 7efc55ffb640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc440762d0 0x7efc440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.425+0000 7efc55ffb640 1 -- 172.21.15.1:0/4110984962 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efc5c0b78d0 con 0x7efc60072120 2024-05-09T05:06:12.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.425+0000 7efc64ec2640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc440762d0 0x7efc440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.426+0000 7efc64ec2640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc440762d0 0x7efc440787e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efc6006f240 tx=0x7efc50001bb0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.430+0000 7efc55ffb640 1 -- 172.21.15.1:0/4110984962 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efc5c04b030 con 0x7efc60072120 2024-05-09T05:06:12.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.620+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7efc600cba10 con 0x7efc60072120 2024-05-09T05:06:12.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.622+0000 7efc55ffb640 1 -- 172.21.15.1:0/4110984962 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7efc5c080be0 con 0x7efc60072120 2024-05-09T05:06:12.622 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:12.622 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:06:06.049536+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:06:12.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc440762d0 msgr2=0x7efc440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc440762d0 0x7efc440787e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7efc6006f240 tx=0x7efc50001bb0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc60072120 msgr2=0x7efc601e7df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc60072120 0x7efc601e7df0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efc5c0098c0 tx=0x7efc5c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 shutdown_connections 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc440762d0 0x7efc440787e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc6015d1d0 0x7efc600cf210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc600caab0 0x7efc600cecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 --2- 172.21.15.1:0/4110984962 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc60072120 0x7efc601e7df0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 >> 172.21.15.1:0/4110984962 conn(0x7efc600731d0 msgr2=0x7efc600cc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 shutdown_connections 2024-05-09T05:06:12.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.624+0000 7efc67927640 1 -- 172.21.15.1:0/4110984962 wait complete. 2024-05-09T05:06:12.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2024-05-09T05:06:12.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2024-05-09T05:06:12.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2024-05-09T05:06:12.637 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2024-05-09T05:06:12.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2024-05-09T05:06:12.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2024-05-09T05:06:12.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2024-05-09T05:06:12.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2024-05-09T05:06:12.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.861+0000 7f41fd029640 1 Processor -- start 2024-05-09T05:06:12.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.861+0000 7f41fd029640 1 -- start start 2024-05-09T05:06:12.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.863+0000 7f41fd029640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0033510 0x7f41f0006b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7f6ad359e640 1 Processor -- start 2024-05-09T05:06:12.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.863+0000 7f41fd029640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f41f0045ea0 0x7f41f00d5720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7faa0759e640 1 Processor -- start 2024-05-09T05:06:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.863+0000 7f41fd029640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41f0046da0 0x7f41f004c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.863+0000 7f41fd029640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f41f00d5c60 con 0x7f41f0046da0 2024-05-09T05:06:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.863+0000 7f41fd029640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f41f0047570 con 0x7f41f0045ea0 2024-05-09T05:06:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7f6ad359e640 1 -- start start 2024-05-09T05:06:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7faa0759e640 1 -- start start 2024-05-09T05:06:12.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.863+0000 7f0a6d266640 1 Processor -- start 2024-05-09T05:06:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.863+0000 7f0a6d266640 1 -- start start 2024-05-09T05:06:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.863+0000 7f41fd029640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f41f0035010 con 0x7f41f0033510 2024-05-09T05:06:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7f41f6d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0033510 0x7f41f0006b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7f41f759e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41f0046da0 0x7f41f004c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7f41f759e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f41f0046da0 0x7f41f004c250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49154/0 (socket says 172.21.15.1:49154) 2024-05-09T05:06:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7f41f759e640 1 -- 172.21.15.1:0/3241964350 learned_addr learned my addr 172.21.15.1:0/3241964350 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.864+0000 7f41f6d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0033510 0x7f41f0006b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59532/0 (socket says 172.21.15.1:59532) 2024-05-09T05:06:12.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.865+0000 7f41f6d9d640 1 -- 172.21.15.1:0/3241964350 >> v2:172.21.15.196:3300/0 conn(0x7f41f0045ea0 msgr2=0x7f41f00d5720 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:06:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.865+0000 7f41f6d9d640 1 --2- 172.21.15.1:0/3241964350 >> v2:172.21.15.196:3300/0 conn(0x7f41f0045ea0 0x7f41f00d5720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.865+0000 7f41f6d9d640 1 -- 172.21.15.1:0/3241964350 >> v2:172.21.15.1:3300/0 conn(0x7f41f0046da0 msgr2=0x7f41f004c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.865+0000 7f41f6d9d640 1 --2- 172.21.15.1:0/3241964350 >> v2:172.21.15.1:3300/0 conn(0x7f41f0046da0 0x7f41f004c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.865+0000 7f41f6d9d640 1 -- 172.21.15.1:0/3241964350 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f0035870 con 0x7f41f0033510 2024-05-09T05:06:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.865+0000 7f0a6d266640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a6004c940 0x7f0a600060e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.865+0000 7f0a6d266640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a600471b0 0x7f0a600d52f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.865+0000 7f0a6d266640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0a600df6e0 con 0x7f0a600de910 2024-05-09T05:06:12.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7fe3b04a8640 1 Processor -- start 2024-05-09T05:06:12.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f06dd994640 1 Processor -- start 2024-05-09T05:06:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7faa0759e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faa000d9830 0x7faa00037090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f41f6d9d640 1 --2- 172.21.15.1:0/3241964350 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0033510 0x7f41f0006b10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f41f0007c40 tx=0x7f41ec032770 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eae2cffb76019445 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f6ad359e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f6acc006530 0x7f6acc0dda40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f6ad1d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6acc046280 0x7f6acc04bb40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35546/0 (socket says 172.21.15.1:35546) 2024-05-09T05:06:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7faa0759e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7faa000375d0 0x7faa00007120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7faa0759e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faa000d9c40 con 0x7faa000e2a10 2024-05-09T05:06:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7faa0759e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7faa000063f0 con 0x7faa000d9830 2024-05-09T05:06:12.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7faa0759e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7faa00048130 con 0x7faa000375d0 2024-05-09T05:06:12.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f6ad1d9b640 1 -- 172.21.15.1:0/3224804175 learned_addr learned my addr 172.21.15.1:0/3224804175 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:12.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f0a6d266640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f0a60046a40 con 0x7f0a6004c940 2024-05-09T05:06:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f0a6d266640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0a60033570 con 0x7f0a600471b0 2024-05-09T05:06:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f0a67fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a6004c940 0x7f0a600060e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f0a67fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0a6004c940 0x7f0a600060e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49162/0 (socket says 172.21.15.1:49162) 2024-05-09T05:06:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f0a67fff640 1 -- 172.21.15.1:0/2552746088 learned_addr learned my addr 172.21.15.1:0/2552746088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f0a67fff640 1 -- 172.21.15.1:0/2552746088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a600471b0 msgr2=0x7f0a600d52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7faa0659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7faa000e2a10 0x7faa000df4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35550/0 (socket says 172.21.15.1:35550) 2024-05-09T05:06:12.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7faa0659c640 1 -- 172.21.15.1:0/1328297957 learned_addr learned my addr 172.21.15.1:0/1328297957 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f6ad2d9d640 1 --2- 172.21.15.1:0/3224804175 >> v2:172.21.15.196:3300/0 conn(0x7f6acc006530 0x7f6acc0dda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7fe3b04a8640 1 -- start start 2024-05-09T05:06:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7faa06d9d640 1 --2- 172.21.15.1:0/1328297957 >> v2:172.21.15.196:3300/0 conn(0x7faa000375d0 0x7faa00007120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f0e91d10640 1 Processor -- start 2024-05-09T05:06:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7faa05d9b640 1 --2- 172.21.15.1:0/1328297957 >> v2:172.21.15.1:3300/0 conn(0x7faa000d9830 0x7faa00037090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f0e91d10640 1 -- start start 2024-05-09T05:06:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f6ad359e640 1 -- 172.21.15.1:0/3224804175 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6acc006940 con 0x7f6acc046280 2024-05-09T05:06:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f6ad359e640 1 -- 172.21.15.1:0/3224804175 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6acc0e3f00 con 0x7f6acc046880 2024-05-09T05:06:12.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f6ad359e640 1 -- 172.21.15.1:0/3224804175 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f6acc046690 con 0x7f6acc006530 2024-05-09T05:06:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7faa0559a640 1 -- 172.21.15.1:0/1328297957 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4254383199 0 0) 0x7faa000d9c40 con 0x7faa000e2a10 2024-05-09T05:06:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f3ee6de0640 1 Processor -- start 2024-05-09T05:06:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7faa0559a640 1 -- 172.21.15.1:0/1328297957 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa9f0003520 con 0x7faa000e2a10 2024-05-09T05:06:12.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f41f5d9b640 1 -- 172.21.15.1:0/3241964350 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec03e070 con 0x7f41f0033510 2024-05-09T05:06:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.866+0000 7f06dd994640 1 -- start start 2024-05-09T05:06:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f41f5d9b640 1 -- 172.21.15.1:0/3241964350 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41ec037710 con 0x7f41f0033510 2024-05-09T05:06:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f41f5d9b640 1 -- 172.21.15.1:0/3241964350 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec037ae0 con 0x7f41f0033510 2024-05-09T05:06:12.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f6ad159a640 1 -- 172.21.15.1:0/3224804175 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216515527 0 0) 0x7f6ad4050dc0 con 0x7f6acc046280 2024-05-09T05:06:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f0a677fe640 1 --2- 172.21.15.1:0/2552746088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a600471b0 0x7f0a600d52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f6ad159a640 1 -- 172.21.15.1:0/3224804175 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6abc003520 con 0x7f6acc046280 2024-05-09T05:06:12.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f0a67fff640 1 --2- 172.21.15.1:0/2552746088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a600471b0 0x7f0a600d52f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7faa06d9d640 1 -- 172.21.15.1:0/1328297957 >> v2:172.21.15.1:3300/0 conn(0x7faa000d9830 msgr2=0x7faa00037090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:12.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7faa06d9d640 1 --2- 172.21.15.1:0/1328297957 >> v2:172.21.15.1:3300/0 conn(0x7faa000d9830 0x7faa00037090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f0a67fff640 1 -- 172.21.15.1:0/2552746088 >> v1:172.21.15.1:6789/0 conn(0x7f0a600de910 legacy=0x7f0a600db320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:12.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.867+0000 7f0a67fff640 1 -- 172.21.15.1:0/2552746088 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a600df440 con 0x7f0a6004c940 2024-05-09T05:06:12.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f0a67fff640 1 --2- 172.21.15.1:0/2552746088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 0x7f0a600060e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00a8f0 tx=0x7f0a5c02e810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dec080dee59ad270 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f6ad159a640 1 -- 172.21.15.1:0/3224804175 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428177174 0 0) 0x7f6acc0e3f00 con 0x7f6acc046880 2024-05-09T05:06:12.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f6ad2d9d640 1 -- 172.21.15.1:0/3224804175 >> v1:172.21.15.196:6789/0 conn(0x7f6acc046880 legacy=0x7f6acc0e36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f6ad2d9d640 1 -- 172.21.15.1:0/3224804175 >> v1:172.21.15.1:6789/0 conn(0x7f6acc046280 legacy=0x7f6acc04bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f6ad2d9d640 1 -- 172.21.15.1:0/3224804175 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6acc0e41e0 con 0x7f6acc006530 2024-05-09T05:06:12.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7faa06d9d640 1 -- 172.21.15.1:0/1328297957 >> v1:172.21.15.1:6789/0 conn(0x7faa000e2a10 legacy=0x7faa000df4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f06dd994640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f06d00076a0 0x7f06d00e3700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f06dd994640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d004b720 0x7f06d004bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f06dd994640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f06d0006340 con 0x7f06d0045800 2024-05-09T05:06:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f06dd994640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f06d0006110 con 0x7f06d00076a0 2024-05-09T05:06:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f06dd994640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f06d0007210 con 0x7f06d004b720 2024-05-09T05:06:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.868+0000 7f3ee6de0640 1 -- start start 2024-05-09T05:06:12.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7faa06d9d640 1 -- 172.21.15.1:0/1328297957 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa000dff50 con 0x7faa000375d0 2024-05-09T05:06:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7fe3b04a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80cbf60 0x7fe3a80cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f0a66ffd640 1 -- 172.21.15.1:0/2552746088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c03d070 con 0x7f0a6004c940 2024-05-09T05:06:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f0a66ffd640 1 -- 172.21.15.1:0/2552746088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a5c035480 con 0x7f0a6004c940 2024-05-09T05:06:12.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7fe3b04a8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe3a80cc370 0x7fe3a80cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f41fd029640 1 -- 172.21.15.1:0/3241964350 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0033510 msgr2=0x7f41f0006b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7fe3b04a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe3a80c7970 con 0x7fe3a80747f0 2024-05-09T05:06:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7fe3ada43640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80cbf60 0x7fe3a80cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7fe3ada43640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80cbf60 0x7fe3a80cd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59554/0 (socket says 172.21.15.1:59554) 2024-05-09T05:06:12.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7fe3ada43640 1 -- 172.21.15.1:0/1246227486 learned_addr learned my addr 172.21.15.1:0/1246227486 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7fe3b04a8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe3a80c5600 con 0x7fe3a80cc370 2024-05-09T05:06:12.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f06d67fc640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f06d0045800 0x7f06d00dd790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:57622/0 (socket says 172.21.15.1:57622) 2024-05-09T05:06:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1246227486 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a80c7be0 con 0x7fe3a80cbf60 2024-05-09T05:06:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7fe3ada43640 1 -- 172.21.15.1:0/1246227486 >> v2:172.21.15.1:3300/0 conn(0x7fe3a80cc370 msgr2=0x7fe3a80cda00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7fe3ada43640 1 --2- 172.21.15.1:0/1246227486 >> v2:172.21.15.1:3300/0 conn(0x7fe3a80cc370 0x7fe3a80cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7fe3ada43640 1 -- 172.21.15.1:0/1246227486 >> v1:172.21.15.1:6789/0 conn(0x7fe3a80747f0 legacy=0x7fe3a8152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7fe3ad242640 1 -- 172.21.15.1:0/1246227486 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2390254081 0 0) 0x7fe3a80c7970 con 0x7fe3a80747f0 2024-05-09T05:06:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.869+0000 7f41fd029640 1 --2- 172.21.15.1:0/3241964350 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0033510 0x7f41f0006b10 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f41f0007c40 tx=0x7f41ec032770 comp rx=0 tx=0).stop 2024-05-09T05:06:12.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0e91d10640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 0x7f0e840060e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f6ad2d9d640 1 --2- 172.21.15.1:0/3224804175 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 0x7f6acc0dda40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80097a0 tx=0x7f6ac802a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2053838a947489fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7fe3ada43640 1 -- 172.21.15.1:0/1246227486 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a80c7ec0 con 0x7fe3a80cbf60 2024-05-09T05:06:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0e8bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0e840d7430 0x7f0e840e07d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35568/0 (socket says 172.21.15.1:35568) 2024-05-09T05:06:12.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0e8bfff640 1 -- 172.21.15.1:0/817352496 learned_addr learned my addr 172.21.15.1:0/817352496 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f41fd029640 1 -- 172.21.15.1:0/3241964350 shutdown_connections 2024-05-09T05:06:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f41fd029640 1 --2- 172.21.15.1:0/3241964350 >> v2:172.21.15.1:3300/0 conn(0x7f41f0046da0 0x7f41f004c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0e90d0e640 1 --2- 172.21.15.1:0/817352496 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 0x7f0e840060e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f06d67fc640 1 -- 172.21.15.1:0/1465119972 learned_addr learned my addr 172.21.15.1:0/1465119972 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:12.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f41fd029640 1 --2- 172.21.15.1:0/3241964350 >> v2:172.21.15.196:3300/0 conn(0x7f41f0045ea0 0x7f41f00d5720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f41fd029640 1 --2- 172.21.15.1:0/3241964350 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0033510 0x7f41f0006b10 secure :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f41f0007c40 tx=0x7f41ec032770 comp rx=0 tx=0).stop 2024-05-09T05:06:12.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f41fd029640 1 -- 172.21.15.1:0/3241964350 >> 172.21.15.1:0/3241964350 conn(0x7f41f00503e0 msgr2=0x7f41f00507f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f06d77fe640 1 --2- 172.21.15.1:0/1465119972 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d004b720 0x7f06d004bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f06d6ffd640 1 --2- 172.21.15.1:0/1465119972 >> v2:172.21.15.196:3300/0 conn(0x7f06d00076a0 0x7f06d00e3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0e91d10640 1 --2- 172.21.15.1:0/817352496 >> v2:172.21.15.196:3300/0 conn(0x7f0e840e0ee0 0x7f0e840e7b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7faa06d9d640 1 --2- 172.21.15.1:0/1328297957 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa000375d0 0x7faa00007120 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faa00010150 tx=0x7fa9f802a250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c83abc476f56610b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f41fd029640 1 -- 172.21.15.1:0/3241964350 shutdown_connections 2024-05-09T05:06:12.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0e91d10640 1 -- 172.21.15.1:0/817352496 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0e8404b5d0 con 0x7f0e840d7430 2024-05-09T05:06:12.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0e91d10640 1 -- 172.21.15.1:0/817352496 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0e84036420 con 0x7f0e840e0ee0 2024-05-09T05:06:12.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0e91d10640 1 -- 172.21.15.1:0/817352496 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0e8404b8b0 con 0x7f0e84035050 2024-05-09T05:06:12.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0a66ffd640 1 -- 172.21.15.1:0/2552746088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a5c035870 con 0x7f0a6004c940 2024-05-09T05:06:12.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0a6d266640 1 -- 172.21.15.1:0/2552746088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 msgr2=0x7f0a600060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f0a6d266640 1 --2- 172.21.15.1:0/2552746088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 0x7f0a600060e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c00a8f0 tx=0x7f0a5c02e810 comp rx=0 tx=0).stop 2024-05-09T05:06:12.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f06d5ffb640 1 -- 172.21.15.1:0/1465119972 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 877941613 0 0) 0x7f06d0006340 con 0x7f06d0045800 2024-05-09T05:06:12.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f06d5ffb640 1 -- 172.21.15.1:0/1465119972 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f06c0003520 con 0x7f06d0045800 2024-05-09T05:06:12.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f06d77fe640 1 -- 172.21.15.1:0/1465119972 >> v2:172.21.15.196:3300/0 conn(0x7f06d00076a0 msgr2=0x7f06d00e3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:12.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f06d77fe640 1 --2- 172.21.15.1:0/1465119972 >> v2:172.21.15.196:3300/0 conn(0x7f06d00076a0 0x7f06d00e3700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f06d77fe640 1 -- 172.21.15.1:0/1465119972 >> v1:172.21.15.196:6789/0 conn(0x7f06d0045800 legacy=0x7f06d00dd790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0e9150f640 1 --2- 172.21.15.1:0/817352496 >> v2:172.21.15.196:3300/0 conn(0x7f0e840e0ee0 0x7f0e840e7b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0a6d266640 1 -- 172.21.15.1:0/2552746088 shutdown_connections 2024-05-09T05:06:12.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0a6d266640 1 --2- 172.21.15.1:0/2552746088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a600471b0 0x7f0a600d52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0a6d266640 1 --2- 172.21.15.1:0/2552746088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 0x7f0a600060e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f6ad159a640 1 -- 172.21.15.1:0/3224804175 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8037070 con 0x7f6acc006530 2024-05-09T05:06:12.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.870+0000 7f41fd029640 1 -- 172.21.15.1:0/3241964350 wait complete. 2024-05-09T05:06:12.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7faa0559a640 1 -- 172.21.15.1:0/1328297957 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8037070 con 0x7faa000375d0 2024-05-09T05:06:12.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0a6d266640 1 -- 172.21.15.1:0/2552746088 >> 172.21.15.1:0/2552746088 conn(0x7f0a600503b0 msgr2=0x7f0a600507c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0e90d0e640 1 -- 172.21.15.1:0/817352496 >> v2:172.21.15.196:3300/0 conn(0x7f0e840e0ee0 msgr2=0x7f0e840e7b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:12.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0e90d0e640 1 --2- 172.21.15.1:0/817352496 >> v2:172.21.15.196:3300/0 conn(0x7f0e840e0ee0 0x7f0e840e7b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7faa0559a640 1 -- 172.21.15.1:0/1328297957 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f8032440 con 0x7faa000375d0 2024-05-09T05:06:12.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0e90d0e640 1 -- 172.21.15.1:0/817352496 >> v1:172.21.15.1:6789/0 conn(0x7f0e840d7430 legacy=0x7f0e840e07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0e90d0e640 1 -- 172.21.15.1:0/817352496 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e84034570 con 0x7f0e84035050 2024-05-09T05:06:12.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0e8b7fe640 1 -- 172.21.15.1:0/817352496 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2602470409 0 0) 0x7f0e8404b5d0 con 0x7f0e840d7430 2024-05-09T05:06:12.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0e9150f640 1 --2- 172.21.15.1:0/817352496 >> v2:172.21.15.196:3300/0 conn(0x7f0e840e0ee0 0x7f0e840e7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:12.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f6ad159a640 1 -- 172.21.15.1:0/3224804175 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac8032640 con 0x7f6acc006530 2024-05-09T05:06:12.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f6ad159a640 1 -- 172.21.15.1:0/3224804175 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac8032a10 con 0x7f6acc006530 2024-05-09T05:06:12.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7faa0559a640 1 -- 172.21.15.1:0/1328297957 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8032850 con 0x7faa000375d0 2024-05-09T05:06:12.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f06d77fe640 1 -- 172.21.15.1:0/1465119972 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d0045c10 con 0x7f06d004b720 2024-05-09T05:06:12.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0a6d266640 1 -- 172.21.15.1:0/2552746088 shutdown_connections 2024-05-09T05:06:12.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f0a6d266640 1 -- 172.21.15.1:0/2552746088 wait complete. 2024-05-09T05:06:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f3ee6de0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3ed80457d0 0x7f3ed80ef060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f3ee6de0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed8006310 con 0x7f3ed80e36d0 2024-05-09T05:06:12.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7fe3ada43640 1 --2- 172.21.15.1:0/1246227486 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80cbf60 0x7fe3a80cd4c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe39800bb10 tx=0x7fe39802c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ab656c1a71bce23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f3ee6de0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3ed8045be0 con 0x7f3ed8007670 2024-05-09T05:06:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f3ee6de0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f3ed80060e0 con 0x7f3ed80457d0 2024-05-09T05:06:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f3ee55dd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3ed80e36d0 0x7f3ed804b6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35580/0 (socket says 172.21.15.1:35580) 2024-05-09T05:06:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f06d77fe640 1 --2- 172.21.15.1:0/1465119972 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d004b720 0x7f06d004bb30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f06c80097a0 tx=0x7f06c802a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f41014f4c7777c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.871+0000 7f3ee55dd640 1 -- 172.21.15.1:0/1092687240 learned_addr learned my addr 172.21.15.1:0/1092687240 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3191018050 0 0) 0x7f3ed8006310 con 0x7f3ed80e36d0 2024-05-09T05:06:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7fe3ad242640 1 -- 172.21.15.1:0/1246227486 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398033640 con 0x7fe3a80cbf60 2024-05-09T05:06:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7fe3ad242640 1 -- 172.21.15.1:0/1246227486 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe398038070 con 0x7fe3a80cbf60 2024-05-09T05:06:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7fe3ad242640 1 -- 172.21.15.1:0/1246227486 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398033a00 con 0x7fe3a80cbf60 2024-05-09T05:06:12.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f0e90d0e640 1 --2- 172.21.15.1:0/817352496 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 0x7f0e840060e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009790 tx=0x7f0e8002a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5960663f2a316f22 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3ecc003520 con 0x7f3ed80e36d0 2024-05-09T05:06:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f06d5ffb640 1 -- 172.21.15.1:0/1465119972 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8037070 con 0x7f06d004b720 2024-05-09T05:06:12.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3181322692 0 0) 0x7f3ecc003520 con 0x7f3ed80e36d0 2024-05-09T05:06:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f06d5ffb640 1 -- 172.21.15.1:0/1465119972 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06c8032440 con 0x7f06d004b720 2024-05-09T05:06:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f06d5ffb640 1 -- 172.21.15.1:0/1465119972 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06c8037070 con 0x7f06d004b720 2024-05-09T05:06:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 >> v2:172.21.15.196:3300/0 conn(0x7f3ed80457d0 msgr2=0x7f3ed80ef060 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:06:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 --2- 172.21.15.1:0/1092687240 >> v2:172.21.15.196:3300/0 conn(0x7f3ed80457d0 0x7f3ed80ef060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 >> v1:172.21.15.196:6789/0 conn(0x7f3ed8007670 legacy=0x7f3ed80dd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed80065d0 con 0x7f3ed80e36d0 2024-05-09T05:06:12.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7faa0759e640 1 -- 172.21.15.1:0/1328297957 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa000375d0 msgr2=0x7faa00007120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7faa0759e640 1 --2- 172.21.15.1:0/1328297957 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa000375d0 0x7faa00007120 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7faa00010150 tx=0x7fa9f802a250 comp rx=0 tx=0).stop 2024-05-09T05:06:12.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f3edc0016d0 con 0x7f3ed80e36d0 2024-05-09T05:06:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7faa0759e640 1 -- 172.21.15.1:0/1328297957 shutdown_connections 2024-05-09T05:06:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7faa0759e640 1 --2- 172.21.15.1:0/1328297957 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa000375d0 0x7faa00007120 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7faa0759e640 1 --2- 172.21.15.1:0/1328297957 >> v2:172.21.15.1:3300/0 conn(0x7faa000d9830 0x7faa00037090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7faa0759e640 1 -- 172.21.15.1:0/1328297957 >> 172.21.15.1:0/1328297957 conn(0x7faa00054840 msgr2=0x7faa00054c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f6ad359e640 1 -- 172.21.15.1:0/3224804175 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 msgr2=0x7f6acc0dda40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f6ad359e640 1 --2- 172.21.15.1:0/3224804175 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 0x7f6acc0dda40 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80097a0 tx=0x7f6ac802a2c0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 >> v1:172.21.15.1:6789/0 conn(0x7f3ed80e36d0 legacy=0x7f3ed804b6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ecc003a60 0x7f3ecc003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc004800 0x7f3ecc004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3ecc0099b0 0x7f3ecc00bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc00c3a0 con 0x7f3ecc004800 2024-05-09T05:06:12.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3ecc00c550 con 0x7f3ecc003a60 2024-05-09T05:06:12.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.872+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3ecc00c750 con 0x7f3ecc0099b0 2024-05-09T05:06:12.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7faa0759e640 1 -- 172.21.15.1:0/1328297957 shutdown_connections 2024-05-09T05:06:12.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f6ad359e640 1 -- 172.21.15.1:0/3224804175 shutdown_connections 2024-05-09T05:06:12.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f6ad359e640 1 --2- 172.21.15.1:0/3224804175 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 0x7f6acc0dda40 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f6ad359e640 1 -- 172.21.15.1:0/3224804175 >> 172.21.15.1:0/3224804175 conn(0x7f6acc050800 msgr2=0x7f6acc050c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f0e8b7fe640 1 -- 172.21.15.1:0/817352496 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e80037070 con 0x7f0e84035050 2024-05-09T05:06:12.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f0e8b7fe640 1 -- 172.21.15.1:0/817352496 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e80032440 con 0x7f0e84035050 2024-05-09T05:06:12.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f0e8b7fe640 1 -- 172.21.15.1:0/817352496 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e80032810 con 0x7f0e84035050 2024-05-09T05:06:12.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f3ee55dd640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc004800 0x7f3ecc004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7faa0759e640 1 -- 172.21.15.1:0/1328297957 wait complete. 2024-05-09T05:06:12.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f3ee5dde640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ecc003a60 0x7f3ecc003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f6ad359e640 1 -- 172.21.15.1:0/3224804175 shutdown_connections 2024-05-09T05:06:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.873+0000 7f6ad359e640 1 -- 172.21.15.1:0/3224804175 wait complete. 2024-05-09T05:06:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1246227486 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80cbf60 msgr2=0x7fe3a80cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7fe3b04a8640 1 --2- 172.21.15.1:0/1246227486 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80cbf60 0x7fe3a80cd4c0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fe39800bb10 tx=0x7fe39802c8b0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f41fd029640 1 Processor -- start 2024-05-09T05:06:12.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f0a6d266640 1 Processor -- start 2024-05-09T05:06:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f06dd994640 1 -- 172.21.15.1:0/1465119972 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d004b720 msgr2=0x7f06d004bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f06dd994640 1 --2- 172.21.15.1:0/1465119972 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d004b720 0x7f06d004bb30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f06c80097a0 tx=0x7f06c802a230 comp rx=0 tx=0).stop 2024-05-09T05:06:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f06dd994640 1 -- 172.21.15.1:0/1465119972 shutdown_connections 2024-05-09T05:06:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f06dd994640 1 --2- 172.21.15.1:0/1465119972 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d004b720 0x7f06d004bb30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f06dd994640 1 --2- 172.21.15.1:0/1465119972 >> v2:172.21.15.196:3300/0 conn(0x7f06d00076a0 0x7f06d00e3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f06dd994640 1 -- 172.21.15.1:0/1465119972 >> 172.21.15.1:0/1465119972 conn(0x7f06d00503e0 msgr2=0x7f06d00507f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f0a6d266640 1 -- start start 2024-05-09T05:06:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1246227486 shutdown_connections 2024-05-09T05:06:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7fe3b04a8640 1 --2- 172.21.15.1:0/1246227486 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80cbf60 0x7fe3a80cd4c0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7fe3b04a8640 1 --2- 172.21.15.1:0/1246227486 >> v2:172.21.15.1:3300/0 conn(0x7fe3a80cc370 0x7fe3a80cda00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1246227486 >> 172.21.15.1:0/1246227486 conn(0x7fe3a80ce180 msgr2=0x7fe3a80ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f0e91d10640 1 -- 172.21.15.1:0/817352496 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 msgr2=0x7f0e840060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f0e91d10640 1 --2- 172.21.15.1:0/817352496 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 0x7f0e840060e0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0e80009790 tx=0x7f0e8002a160 comp rx=0 tx=0).stop 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f3ee55dd640 1 -- 172.21.15.1:0/1092687240 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3ecc0099b0 msgr2=0x7f3ecc00bdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f3ee55dd640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3ecc0099b0 0x7f3ecc00bdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f3ee55dd640 1 -- 172.21.15.1:0/1092687240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ecc003a60 msgr2=0x7f3ecc003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f3ee55dd640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ecc003a60 0x7f3ecc003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f3ee55dd640 1 -- 172.21.15.1:0/1092687240 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed80065d0 con 0x7f3ecc004800 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f3ee65df640 1 -- 172.21.15.1:0/1092687240 reap_dead start 2024-05-09T05:06:12.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f41fd029640 1 -- start start 2024-05-09T05:06:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f0e91d10640 1 -- 172.21.15.1:0/817352496 shutdown_connections 2024-05-09T05:06:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f0e91d10640 1 --2- 172.21.15.1:0/817352496 >> v2:172.21.15.196:3300/0 conn(0x7f0e840e0ee0 0x7f0e840e7b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f0e91d10640 1 --2- 172.21.15.1:0/817352496 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 0x7f0e840060e0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.874+0000 7f0e91d10640 1 -- 172.21.15.1:0/817352496 >> 172.21.15.1:0/817352496 conn(0x7f0e840503b0 msgr2=0x7f0e840507c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.875+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1246227486 shutdown_connections 2024-05-09T05:06:12.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.875+0000 7f06dd994640 1 -- 172.21.15.1:0/1465119972 shutdown_connections 2024-05-09T05:06:12.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.875+0000 7f06dd994640 1 -- 172.21.15.1:0/1465119972 wait complete. 2024-05-09T05:06:12.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.875+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1246227486 wait complete. 2024-05-09T05:06:12.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.875+0000 7f0e91d10640 1 -- 172.21.15.1:0/817352496 shutdown_connections 2024-05-09T05:06:12.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.875+0000 7f0e91d10640 1 -- 172.21.15.1:0/817352496 wait complete. 2024-05-09T05:06:12.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.875+0000 7f3ee55dd640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc004800 0x7f3ecc004c10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00c2f0 tx=0x7f3ecc005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8db285854ff045fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.876+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc036070 con 0x7f3ecc004800 2024-05-09T05:06:12.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.876+0000 7f0a6d266640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a600471b0 0x7f0a6016e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.876+0000 7f0a6d266640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 0x7f0a6016ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.876+0000 7f0a677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 0x7f0a6016ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f0a677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 0x7f0a6016ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49188/0 (socket says 172.21.15.1:49188) 2024-05-09T05:06:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7faa0759e640 1 Processor -- start 2024-05-09T05:06:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f0a67fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a600471b0 0x7f0a6016e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7faa0759e640 1 -- start start 2024-05-09T05:06:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.876+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3edc005ad0 con 0x7f3ecc004800 2024-05-09T05:06:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f0a67fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a600471b0 0x7f0a6016e2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43894/0 (socket says 172.21.15.1:43894) 2024-05-09T05:06:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f0a677fe640 1 -- 172.21.15.1:0/3852886321 learned_addr learned my addr 172.21.15.1:0/3852886321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f6ad359e640 1 Processor -- start 2024-05-09T05:06:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f41fd029640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0045ea0 0x7f41f00d4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f3ee6de0640 1 -- 172.21.15.1:0/1092687240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc004800 msgr2=0x7f3ecc004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f6ad359e640 1 -- start start 2024-05-09T05:06:12.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f41fd029640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f0046da0 0x7f41f01718a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f3ee4ddc640 1 -- 172.21.15.1:0/1092687240 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3edc036070 con 0x7f3ecc004800 2024-05-09T05:06:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f41fd029640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f00d1590 0x7f41f00ca1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f3ee6de0640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc004800 0x7f3ecc004c10 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f3edc00c2f0 tx=0x7f3ecc005830 comp rx=0 tx=0).stop 2024-05-09T05:06:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f41f659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f0046da0 0x7f41f01718a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f41f659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f0046da0 0x7f41f01718a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49202/0 (socket says 172.21.15.1:49202) 2024-05-09T05:06:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41f6d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0045ea0 0x7f41f00d4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f3ee6de0640 1 -- 172.21.15.1:0/1092687240 shutdown_connections 2024-05-09T05:06:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f3ee6de0640 1 --2- 172.21.15.1:0/1092687240 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ecc004800 0x7f3ecc004c10 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f3ee6de0640 1 -- 172.21.15.1:0/1092687240 >> 172.21.15.1:0/1092687240 conn(0x7f3ed80503b0 msgr2=0x7f3ed80507c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41f759e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f00d1590 0x7f41f00ca1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41f759e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f00d1590 0x7f41f00ca1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43906/0 (socket says 172.21.15.1:43906) 2024-05-09T05:06:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41fd029640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f41f0035590 con 0x7f41f0046da0 2024-05-09T05:06:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41fd029640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f41f00352b0 con 0x7f41f00d1590 2024-05-09T05:06:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41fd029640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f41f00065f0 con 0x7f41f0045ea0 2024-05-09T05:06:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41f659c640 1 -- 172.21.15.1:0/4064073374 learned_addr learned my addr 172.21.15.1:0/4064073374 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f3ee6de0640 1 -- 172.21.15.1:0/1092687240 shutdown_connections 2024-05-09T05:06:12.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f06dd994640 1 Processor -- start 2024-05-09T05:06:12.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f3ee6de0640 1 -- 172.21.15.1:0/1092687240 wait complete. 2024-05-09T05:06:12.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7fe3b04a8640 1 Processor -- start 2024-05-09T05:06:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41f6d9d640 1 -- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f00d1590 msgr2=0x7f41f00ca1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f41f6d9d640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41f00d1590 0x7f41f00ca1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7f41f6d9d640 1 -- 172.21.15.1:0/4064073374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f0046da0 msgr2=0x7f41f01718a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7f41f6d9d640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f0046da0 0x7f41f01718a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f0a6d266640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a600de910 0x7f0a600cd760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0a60046c90 con 0x7f0a6004c940 2024-05-09T05:06:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0a6004bf80 con 0x7f0a600471b0 2024-05-09T05:06:12.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.877+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0a60034480 con 0x7f0a600de910 2024-05-09T05:06:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f0a677fe640 1 -- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a600de910 msgr2=0x7f0a600cd760 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:06:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f0a677fe640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0a600de910 0x7f0a600cd760 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f0a677fe640 1 -- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a600471b0 msgr2=0x7f0a6016e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f0a677fe640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a600471b0 0x7f0a6016e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f0a677fe640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a6016f380 con 0x7f0a6004c940 2024-05-09T05:06:12.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7f41f6d9d640 1 -- 172.21.15.1:0/4064073374 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f0171de0 con 0x7f41f0045ea0 2024-05-09T05:06:12.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f06dd994640 1 -- start start 2024-05-09T05:06:12.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7f0a67fff640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0a600471b0 0x7f0a6016e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:12.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7f41f659c640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41f0046da0 0x7f41f01718a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:12.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f0e91d10640 1 Processor -- start 2024-05-09T05:06:12.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.878+0000 7f0e91d10640 1 -- start start 2024-05-09T05:06:12.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7f0a677fe640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 0x7f0a6016ee40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0a68051dc0 tx=0x7f0a6806dad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7fe3b04a8640 1 -- start start 2024-05-09T05:06:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7faa0759e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa000375d0 0x7faa0004e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7faa0759e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7faa000d9830 0x7faa0004caf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68096670 con 0x7f0a6004c940 2024-05-09T05:06:12.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7faa0759e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa000e2a10 0x7faa000d60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.879+0000 7f6ad359e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 0x7f6acc1702e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7f6ad359e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6acc046280 0x7f6acc0e2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7f6ad259c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 0x7f6acc1702e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7faa06d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa000e2a10 0x7faa000d60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7faa06d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa000e2a10 0x7faa000d60e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49218/0 (socket says 172.21.15.1:49218) 2024-05-09T05:06:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7faa06d9d640 1 -- 172.21.15.1:0/1080556712 learned_addr learned my addr 172.21.15.1:0/1080556712 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faa00006920 con 0x7faa000e2a10 2024-05-09T05:06:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7faa00006640 con 0x7faa000375d0 2024-05-09T05:06:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7faa00006c00 con 0x7faa000d9830 2024-05-09T05:06:12.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7f6ad259c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 0x7f6acc1702e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43914/0 (socket says 172.21.15.1:43914) 2024-05-09T05:06:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7faa0659c640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa000375d0 0x7faa0004e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06dd994640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06d00076a0 0x7f06d0182980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06dd994640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d0045800 0x7f06d00e6e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06dd994640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06d004b720 0x7f06d00df930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06dd994640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f06d0006bc0 con 0x7f06d004b720 2024-05-09T05:06:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f3ee6de0640 1 Processor -- start 2024-05-09T05:06:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7faa06d9d640 1 -- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7faa000d9830 msgr2=0x7faa0004caf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:06:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7faa06d9d640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7faa000d9830 0x7faa0004caf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7faa06d9d640 1 -- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa000375d0 msgr2=0x7faa0004e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7faa06d9d640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faa000375d0 0x7faa0004e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06dd994640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f06d00068e0 con 0x7f06d00076a0 2024-05-09T05:06:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06dd994640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f06d0006ea0 con 0x7f06d0045800 2024-05-09T05:06:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7faa06d9d640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa0004d030 con 0x7faa000e2a10 2024-05-09T05:06:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f6ad1d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6acc046280 0x7f6acc0e2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f6ad1d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6acc046280 0x7f6acc0e2e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59590/0 (socket says 172.21.15.1:59590) 2024-05-09T05:06:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f3ee6de0640 1 -- start start 2024-05-09T05:06:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f6ad359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6acc046880 0x7f6acc0e3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f6ad259c640 1 -- 172.21.15.1:0/2433381124 learned_addr learned my addr 172.21.15.1:0/2433381124 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a6004e8f0 con 0x7f0a6004c940 2024-05-09T05:06:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.880+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a6004e330 con 0x7f0a6004c940 2024-05-09T05:06:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0a600054a0 con 0x7f0a6004c940 2024-05-09T05:06:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0a68091070 con 0x7f0a6004c940 2024-05-09T05:06:12.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0a68096a30 con 0x7f0a6004c940 2024-05-09T05:06:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6acc045a80 con 0x7f6acc046880 2024-05-09T05:06:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6acc0457a0 con 0x7f6acc006530 2024-05-09T05:06:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6acc045d60 con 0x7f6acc046280 2024-05-09T05:06:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06d00076a0 0x7f06d0182980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06d6ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06d00076a0 0x7f06d0182980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43926/0 (socket says 172.21.15.1:43926) 2024-05-09T05:06:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06d6ffd640 1 -- 172.21.15.1:0/2021608557 learned_addr learned my addr 172.21.15.1:0/2021608557 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06d67fc640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d0045800 0x7f06d00e6e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f41f6d9d640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0045ea0 0x7f41f00d4860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f41f0035190 tx=0x7f41ec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f6ad2d9d640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6acc046880 0x7f6acc0e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e91d10640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 0x7f0e840df2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e91d10640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e840d7430 0x7f0e8417d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e91d10640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e840e0ee0 0x7f0e8417dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e91d10640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0e840334e0 con 0x7f0e840e0ee0 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e91d10640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0e84034b30 con 0x7f0e840d7430 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e91d10640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0e84006620 con 0x7f0e84035050 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e8bfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e840d7430 0x7f0e8417d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e8bfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e840d7430 0x7f0e8417d760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43934/0 (socket says 172.21.15.1:43934) 2024-05-09T05:06:12.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f0e8bfff640 1 -- 172.21.15.1:0/772851468 learned_addr learned my addr 172.21.15.1:0/772851468 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7faa06d9d640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa000e2a10 0x7faa000d60e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009860 tx=0x7fa9f80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8037070 con 0x7faa000e2a10 2024-05-09T05:06:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa9f8042070 con 0x7faa000e2a10 2024-05-09T05:06:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa0004c330 con 0x7faa000e2a10 2024-05-09T05:06:12.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7fe3b04a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80747f0 0x7fe3a81ede60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f06d77fe640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06d004b720 0x7f06d00df930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f8030ae0 con 0x7faa000e2a10 2024-05-09T05:06:12.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7fe3b04a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a80cbf60 0x7fe3a8151030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7fe3b04a8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3a80cc370 0x7fe3a814a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.881+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec047030 con 0x7f41f0045ea0 2024-05-09T05:06:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f41ec005990 con 0x7f41f0045ea0 2024-05-09T05:06:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec03e040 con 0x7f41f0045ea0 2024-05-09T05:06:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f00cae30 con 0x7f41f0045ea0 2024-05-09T05:06:12.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f00de7d0 con 0x7f41f0045ea0 2024-05-09T05:06:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f6ad259c640 1 -- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6acc046280 msgr2=0x7f6acc0e2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f6ad259c640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6acc046280 0x7f6acc0e2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f6ad259c640 1 -- 172.21.15.1:0/2433381124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6acc046880 msgr2=0x7f6acc0e3380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f6ad259c640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6acc046880 0x7f6acc0e3380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f6ad259c640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6acc17cb80 con 0x7f6acc006530 2024-05-09T05:06:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7fe3b04a8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a80740c0 con 0x7fe3a80cbf60 2024-05-09T05:06:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7fe3b04a8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe3a80c8480 con 0x7fe3a80cc370 2024-05-09T05:06:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7fe3b04a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe3a8068650 con 0x7fe3a80747f0 2024-05-09T05:06:12.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f0e8bfff640 1 -- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 msgr2=0x7f0e840df2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f0e8bfff640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0e84035050 0x7f0e840df2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f0e8bfff640 1 -- 172.21.15.1:0/772851468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e840e0ee0 msgr2=0x7f0e8417dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f0e8bfff640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0e840e0ee0 0x7f0e8417dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f0e8bfff640 1 -- 172.21.15.1:0/772851468 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0e840cb1b0 con 0x7f0e840d7430 2024-05-09T05:06:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa0004de00 con 0x7faa000e2a10 2024-05-09T05:06:12.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f06d67fc640 1 -- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06d00076a0 msgr2=0x7f06d0182980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f06d67fc640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f06d00076a0 0x7f06d0182980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f06d67fc640 1 -- 172.21.15.1:0/2021608557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06d004b720 msgr2=0x7f06d00df930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f06d67fc640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06d004b720 0x7f06d00df930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.882+0000 7f06d67fc640 1 -- 172.21.15.1:0/2021608557 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06d00ddf00 con 0x7f06d0045800 2024-05-09T05:06:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f06d77fe640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f06d004b720 0x7f06d00df930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f41dd7fa640 1 -- 172.21.15.1:0/4064073374 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f41f00054d0 con 0x7f41f0045ea0 2024-05-09T05:06:12.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f6ad259c640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 0x7f6acc1702e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400c3c0 tx=0x7f6ac400c7c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ada43640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a80cbf60 0x7fe3a8151030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ada43640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a80cbf60 0x7fe3a8151030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49242/0 (socket says 172.21.15.1:49242) 2024-05-09T05:06:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ada43640 1 -- 172.21.15.1:0/1140921640 learned_addr learned my addr 172.21.15.1:0/1140921640 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3aea45640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3a80cc370 0x7fe3a814a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ae244640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80747f0 0x7fe3a81ede60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f0e8bfff640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e840d7430 0x7f0e8417d760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400e8f0 tx=0x7f0e7800c820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f06d67fc640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d0045800 0x7f06d00e6e80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06d8000b60 tx=0x7f06d8069e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f3ee6de0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3ecc004800 0x7f3ed80deac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f3ee6de0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed80de0e0 0x7f3ed8049520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ada43640 1 -- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80747f0 msgr2=0x7fe3a81ede60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ada43640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe3a80747f0 0x7fe3a81ede60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ada43640 1 -- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3a80cc370 msgr2=0x7fe3a814a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ada43640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe3a80cc370 0x7fe3a814a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7fe3ada43640 1 -- 172.21.15.1:0/1140921640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe3a8149ac0 con 0x7fe3a80cbf60 2024-05-09T05:06:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4017070 con 0x7f6acc006530 2024-05-09T05:06:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6ac40035f0 con 0x7f6acc006530 2024-05-09T05:06:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f3ee6de0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed8049a60 0x7f3ed817d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f3ee6de0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed8006b90 con 0x7f3ed8049a60 2024-05-09T05:06:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f3ee6de0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3ed80071e0 con 0x7f3ed80de0e0 2024-05-09T05:06:12.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f3ee6de0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3ed8006e70 con 0x7f3ecc004800 2024-05-09T05:06:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e780074e0 con 0x7f0e840d7430 2024-05-09T05:06:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0e780047e0 con 0x7f0e840d7430 2024-05-09T05:06:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0e78004bb0 con 0x7f0e840d7430 2024-05-09T05:06:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6acc0d3750 con 0x7f6acc006530 2024-05-09T05:06:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6acc17d120 con 0x7f6acc006530 2024-05-09T05:06:12.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ac4003a00 con 0x7f6acc006530 2024-05-09T05:06:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f3ee5dde640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3ecc004800 0x7f3ed80deac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f3ee5dde640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3ecc004800 0x7f3ed80deac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59628/0 (socket says 172.21.15.1:59628) 2024-05-09T05:06:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f3ee5dde640 1 -- 172.21.15.1:0/1132563465 learned_addr learned my addr 172.21.15.1:0/1132563465 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f3ee65df640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed8049a60 0x7f3ed817d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0a68096bd0 con 0x7f0a6004c940 2024-05-09T05:06:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f0a64ff9640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0a4c076320 0x7f0a4c078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7fe3ada43640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a80cbf60 0x7fe3a8151030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe39800bb10 tx=0x7fe398004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f0e91d10640 1 -- 172.21.15.1:0/772851468 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0e840cb470 con 0x7f0e840d7430 2024-05-09T05:06:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f0e91d10640 1 -- 172.21.15.1:0/772851468 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0e840cba30 con 0x7f0e840d7430 2024-05-09T05:06:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398043070 con 0x7fe3a80cbf60 2024-05-09T05:06:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f3ee55dd640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed80de0e0 0x7f3ed8049520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f3ee65df640 1 -- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3ecc004800 msgr2=0x7f3ed80deac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f3ee65df640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3ecc004800 0x7f3ed80deac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f3ee65df640 1 -- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed80de0e0 msgr2=0x7f3ed8049520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f3ee65df640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed80de0e0 0x7f3ed8049520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f3ee65df640 1 -- 172.21.15.1:0/1132563465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3ed80df000 con 0x7f3ed8049a60 2024-05-09T05:06:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f3ee55dd640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3ed80de0e0 0x7f3ed8049520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.883+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7faa00175950 con 0x7faa000e2a10 2024-05-09T05:06:12.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f3ee65df640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed8049a60 0x7f3ed817d4c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3ee006b1b0 tx=0x7f3ee0069e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7f0a67fff640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0a4c076320 0x7f0a4c078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f41ec04a070 con 0x7f41f0045ea0 2024-05-09T05:06:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f06bffff640 1 -- 172.21.15.1:0/2021608557 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8095600 con 0x7f06d0045800 2024-05-09T05:06:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f06bffff640 1 -- 172.21.15.1:0/2021608557 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f06d80957c0 con 0x7f06d0045800 2024-05-09T05:06:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f06bffff640 1 -- 172.21.15.1:0/2021608557 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f06d8095bb0 con 0x7f06d0045800 2024-05-09T05:06:12.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f06dd994640 1 -- 172.21.15.1:0/2021608557 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06d0182ec0 con 0x7f06d0045800 2024-05-09T05:06:12.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.884+0000 7f06dd994640 1 -- 172.21.15.1:0/2021608557 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f06d00e0d50 con 0x7f06d0045800 2024-05-09T05:06:12.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0a68113730 con 0x7f0a6004c940 2024-05-09T05:06:12.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7f0a64ff9640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f0a4c07e5f0 0x7f0a4c080a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe3a814c0e0 con 0x7fe3a80cbf60 2024-05-09T05:06:12.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.885+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe3a8149d80 con 0x7fe3a80cbf60 2024-05-09T05:06:12.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe398032be0 con 0x7fe3a80cbf60 2024-05-09T05:06:12.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe398043070 con 0x7fe3a80cbf60 2024-05-09T05:06:12.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7f41df7fe640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41e40763d0 0x7f41e40788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.887+0000 7f41f659c640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41e40763d0 0x7f41e40788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.887+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f41ec0c0100 con 0x7f41f0045ea0 2024-05-09T05:06:12.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.887+0000 7f41f659c640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41e40763d0 0x7f41e40788e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f41f004d7a0 tx=0x7f41e80090e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f6ab8001030 con 0x7f6acc006530 2024-05-09T05:06:12.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee00743c0 con 0x7f3ed8049a60 2024-05-09T05:06:12.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7f3ee6de0640 1 -- 172.21.15.1:0/1132563465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3ed80dcbb0 con 0x7f3ed8049a60 2024-05-09T05:06:12.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.886+0000 7f3ee6de0640 1 -- 172.21.15.1:0/1132563465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3ed8035920 con 0x7f3ed8049a60 2024-05-09T05:06:12.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.887+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3ee0095070 con 0x7f3ed8049a60 2024-05-09T05:06:12.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.887+0000 7f41df7fe640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f41e407e640 0x7f41e4080a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.887+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ee00747e0 con 0x7f3ed8049a60 2024-05-09T05:06:12.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f41f759e640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f41e407e640 0x7f41e4080a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f0a6ca65640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f0a4c07e5f0 0x7f0a4c080a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3ed80054a0 con 0x7f3ed8049a60 2024-05-09T05:06:12.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6ac401a070 con 0x7f6acc006530 2024-05-09T05:06:12.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 --> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f41e4081160 con 0x7f41e407e640 2024-05-09T05:06:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f41ec0c04f0 con 0x7f41f0045ea0 2024-05-09T05:06:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa9f8030c80 con 0x7faa000e2a10 2024-05-09T05:06:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.887+0000 7f06bffff640 1 -- 172.21.15.1:0/2021608557 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f06d809e3f0 con 0x7f06d0045800 2024-05-09T05:06:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f06bffff640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f06c40763d0 0x7f06c40788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f06bffff640 1 -- 172.21.15.1:0/2021608557 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f06d8114560 con 0x7f06d0045800 2024-05-09T05:06:12.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f06d6ffd640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f06c40763d0 0x7f06c40788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f41f759e640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f41e407e640 0x7f41e4080a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.887+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0e78011070 con 0x7f0e840d7430 2024-05-09T05:06:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.888+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0e8417e1e0 con 0x7f0e840d7430 2024-05-09T05:06:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f06dd994640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7f06b8001650 0x7f06b8003b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f6ac2ffd640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6ab40763d0 0x7f6ab40788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f06dd994640 1 -- 172.21.15.1:0/2021608557 --> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f06b8006c00 con 0x7f06b8001650 2024-05-09T05:06:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7fa9f6ffd640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9f0076820 0x7fa9f0078d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f06d6ffd640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f06c40763d0 0x7f06c40788e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f06cc006a30 tx=0x7f06cc009110 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0a68076b00 con 0x7f0a4c07e5f0 2024-05-09T05:06:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0a6ca65640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f0a4c07e5f0 0x7f0a4c080a40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0e897fa640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0e700763a0 0x7f0e700788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f0e78096fb0 con 0x7f0e840d7430 2024-05-09T05:06:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0e897fa640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7f0e7007e670 0x7f0e70080ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 --> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0e70081190 con 0x7f0e7007e670 2024-05-09T05:06:12.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0e7809c5a0 con 0x7f0e840d7430 2024-05-09T05:06:12.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f0a680da9a0 con 0x7f0a6004c940 2024-05-09T05:06:12.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa9f80b7b80 con 0x7faa000e2a10 2024-05-09T05:06:12.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f06d77fe640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7f06b8001650 0x7f06b8003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f6ad1d9b640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6ab40763d0 0x7f6ab40788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe390001030 con 0x7fe3a80cbf60 2024-05-09T05:06:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7fa9f6ffd640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7fa9f007ea70 0x7fa9f0080ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 <== osd.3 v2:172.21.15.1:6808/3991537322 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f41e4081160 con 0x7f41e407e640 2024-05-09T05:06:12.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6ac4096e20 con 0x7f6acc006530 2024-05-09T05:06:12.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa9f8032440 con 0x7fa9f007ea70 2024-05-09T05:06:12.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7faa0659c640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9f0076820 0x7fa9f0078d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fa9f80be050 con 0x7faa000e2a10 2024-05-09T05:06:12.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f6ac2ffd640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f6ab407e6a0 0x7f6ab4080af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7faa05d9b640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7fa9f007ea70 0x7fa9f0080ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f6ad1d9b640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6ab40763d0 0x7f6ab40788e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4064290 tx=0x7f6ad4068550 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe398046070 con 0x7fe3a80cbf60 2024-05-09T05:06:12.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ab40811c0 con 0x7f6ab407e6a0 2024-05-09T05:06:12.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f0a67fff640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0a4c076320 0x7f0a4c078830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c03a000 tx=0x7f0a5c033000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.889+0000 7f0e90d0e640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0e700763a0 0x7f0e700788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f0e9150f640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7f0e7007e670 0x7f0e70080ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f0e90d0e640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0e700763a0 0x7f0e700788b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0e80034000 tx=0x7f0e8002c000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.890+0000 7f6ad2d9d640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f6ab407e6a0 0x7f6ab4080af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f6ac4097210 con 0x7f6acc006530 2024-05-09T05:06:12.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7faa0659c640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9f0076820 0x7fa9f0078d30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa08064290 tx=0x7faa08068550 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7f06d77fe640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7f06b8001650 0x7f06b8003b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7f0e9150f640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7f0e7007e670 0x7f0e70080ac0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7fe39effd640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe38c076500 0x7fe38c078a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7faa05d9b640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7fa9f007ea70 0x7fa9f0080ec0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 <== osd.1 v2:172.21.15.1:6811/2455090880 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0a68076b00 con 0x7f0a4c07e5f0 2024-05-09T05:06:12.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe3980b7fd0 con 0x7fe3a80cbf60 2024-05-09T05:06:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7fe3ae244640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe38c076500 0x7fe38c078a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7fe39effd640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7fe38c07e7d0 0x7fe38c080c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 --> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe398034e90 con 0x7fe38c07e7d0 2024-05-09T05:06:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7fe3980ba5d0 con 0x7fe3a80cbf60 2024-05-09T05:06:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7f6ad2d9d640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f6ab407e6a0 0x7f6ab4080af0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3ee0074980 con 0x7f3ed8049a60 2024-05-09T05:06:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.891+0000 7fe3aea45640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7fe38c07e7d0 0x7fe38c080c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f06bffff640 1 -- 172.21.15.1:0/2021608557 <== osd.4 v2:172.21.15.196:6808/4274960119 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f06b8006c00 con 0x7f06b8001650 2024-05-09T05:06:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f3ed67fc640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3ec4076320 0x7f3ec4078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 <== osd.7 v2:172.21.15.196:6816/747967558 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0e70081190 con 0x7f0e7007e670 2024-05-09T05:06:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3ee01147d0 con 0x7f3ed8049a60 2024-05-09T05:06:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 <== osd.0 v2:172.21.15.1:6824/1100742567 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa9f8032440 con 0x7fa9f007ea70 2024-05-09T05:06:12.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f3ed67fc640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7f3ec407e5f0 0x7f3ec4080a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:12.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 <== osd.5 v2:172.21.15.196:6823/3481343234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f6ab40811c0 con 0x7f6ab407e6a0 2024-05-09T05:06:12.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 --> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3ee008ee50 con 0x7f3ec407e5f0 2024-05-09T05:06:12.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=21) ==== 24+0+0 (secure 0 0 0) 0x7f3ee0114cb0 con 0x7f3ed8049a60 2024-05-09T05:06:12.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7f3ee55dd640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7f3ec407e5f0 0x7f3ec4080a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.893+0000 7f3ee5dde640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3ec4076320 0x7f3ec4078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:12.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.892+0000 7fe3aea45640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7fe38c07e7d0 0x7fe38c080c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.893+0000 7fe3ae244640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe38c076500 0x7fe38c078a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0000f60 tx=0x7fe3a0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.893+0000 7f3ee55dd640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7f3ec407e5f0 0x7f3ec4080a40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.893+0000 7f3ee5dde640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3ec4076320 0x7f3ec4078830 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00079a0 tx=0x7f3ed0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:12.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.893+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 <== osd.2 v2:172.21.15.1:6800/239705464 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe398034e90 con 0x7fe38c07e7d0 2024-05-09T05:06:12.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.894+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 <== osd.6 v2:172.21.15.196:6800/3854610795 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3ee008ee50 con 0x7f3ec407e5f0 2024-05-09T05:06:12.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.922+0000 7f41dd7fa640 1 -- 172.21.15.1:0/4064073374 --> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f41f0045c00 con 0x7f41e407e640 2024-05-09T05:06:12.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.923+0000 7f41df7fe640 1 -- 172.21.15.1:0/4064073374 <== osd.3 v2:172.21.15.1:6808/3991537322 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f41f0045c00 con 0x7f41e407e640 2024-05-09T05:06:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.924+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f41e407e640 msgr2=0x7f41e4080a90 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.924+0000 7f41fd029640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f41e407e640 0x7f41e4080a90 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.924+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41e40763d0 msgr2=0x7f41e40788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.924+0000 7f41fd029640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41e40763d0 0x7f41e40788e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f41f004d7a0 tx=0x7f41e80090e0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.924+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0045ea0 msgr2=0x7f41f00d4860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.924+0000 7f41fd029640 1 --2- 172.21.15.1:0/4064073374 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f41f0045ea0 0x7f41f00d4860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f41f0035190 tx=0x7f41ec0046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.925+0000 7f41f759e640 1 -- 172.21.15.1:0/4064073374 reap_dead start 2024-05-09T05:06:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.925+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 shutdown_connections 2024-05-09T05:06:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.925+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 >> 172.21.15.1:0/4064073374 conn(0x7f41f00503e0 msgr2=0x7f41f004ea50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.925+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 --> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0a6004f660 con 0x7f0a4c07e5f0 2024-05-09T05:06:12.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.925+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 shutdown_connections 2024-05-09T05:06:12.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.925+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 --> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f6ab8002dd0 con 0x7f6ab407e6a0 2024-05-09T05:06:12.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.925+0000 7f0a64ff9640 1 -- 172.21.15.1:0/3852886321 <== osd.1 v2:172.21.15.1:6811/2455090880 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0a6004f660 con 0x7f0a4c07e5f0 2024-05-09T05:06:12.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.925+0000 7f41fd029640 1 -- 172.21.15.1:0/4064073374 wait complete. 2024-05-09T05:06:12.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f06dd994640 1 -- 172.21.15.1:0/2021608557 --> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f06b8005c90 con 0x7f06b8001650 2024-05-09T05:06:12.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 --> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0e840dfe10 con 0x7f0e7007e670 2024-05-09T05:06:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f0a4c07e5f0 msgr2=0x7f0a4c080a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6d266640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f0a4c07e5f0 0x7f0a4c080a40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f6ac2ffd640 1 -- 172.21.15.1:0/2433381124 <== osd.5 v2:172.21.15.196:6823/3481343234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f6ab8002dd0 con 0x7f6ab407e6a0 2024-05-09T05:06:12.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 --> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7faa00175c40 con 0x7fa9f007ea70 2024-05-09T05:06:12.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0a4c076320 msgr2=0x7f0a4c078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6d266640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0a4c076320 0x7f0a4c078830 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0a5c03a000 tx=0x7f0a5c033000 comp rx=0 tx=0).stop 2024-05-09T05:06:12.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 msgr2=0x7f0a6016ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6d266640 1 --2- 172.21.15.1:0/3852886321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0a6004c940 0x7f0a6016ee40 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0a68051dc0 tx=0x7f0a6806dad0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f06bffff640 1 -- 172.21.15.1:0/2021608557 <== osd.4 v2:172.21.15.196:6808/4274960119 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f06b8005c90 con 0x7f06b8001650 2024-05-09T05:06:12.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6ca65640 1 -- 172.21.15.1:0/3852886321 reap_dead start 2024-05-09T05:06:12.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7fa9f6ffd640 1 -- 172.21.15.1:0/1080556712 <== osd.0 v2:172.21.15.1:6824/1100742567 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7faa00175c40 con 0x7fa9f007ea70 2024-05-09T05:06:12.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 shutdown_connections 2024-05-09T05:06:12.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0e897fa640 1 -- 172.21.15.1:0/772851468 <== osd.7 v2:172.21.15.196:6816/747967558 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0e840dfe10 con 0x7f0e7007e670 2024-05-09T05:06:12.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.926+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 >> 172.21.15.1:0/3852886321 conn(0x7f0a600503b0 msgr2=0x7f0a600056b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f6ab407e6a0 msgr2=0x7f6ab4080af0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f6ab407e6a0 0x7f6ab4080af0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 shutdown_connections 2024-05-09T05:06:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7fa9f007ea70 msgr2=0x7fa9f0080ec0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa0759e640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7fa9f007ea70 0x7fa9f0080ec0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06bdffb640 1 -- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7f06b8001650 msgr2=0x7f06b8003b60 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06bdffb640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7f06b8001650 0x7f06b8003b60 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06bdffb640 1 -- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f06c40763d0 msgr2=0x7f06c40788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06bdffb640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f06c40763d0 0x7f06c40788e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f06cc006a30 tx=0x7f06cc009110 comp rx=0 tx=0).stop 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06bdffb640 1 -- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d0045800 msgr2=0x7f06d00e6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06bdffb640 1 --2- 172.21.15.1:0/2021608557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f06d0045800 0x7f06d00e6e80 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f06d8000b60 tx=0x7f06d8069e50 comp rx=0 tx=0).stop 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9f0076820 msgr2=0x7fa9f0078d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa0759e640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9f0076820 0x7fa9f0078d30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7faa08064290 tx=0x7faa08068550 comp rx=0 tx=0).stop 2024-05-09T05:06:12.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa000e2a10 msgr2=0x7faa000d60e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa0759e640 1 --2- 172.21.15.1:0/1080556712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faa000e2a10 0x7faa000d60e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa9f8009860 tx=0x7fa9f80046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 --> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe390002dd0 con 0x7fe38c07e7d0 2024-05-09T05:06:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6ab40763d0 msgr2=0x7f6ab40788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6ab40763d0 0x7f6ab40788e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4064290 tx=0x7f6ad4068550 comp rx=0 tx=0).stop 2024-05-09T05:06:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 msgr2=0x7f6acc1702e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 --2- 172.21.15.1:0/2433381124 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6acc006530 0x7f6acc1702e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f6ac400c3c0 tx=0x7f6ac400c7c0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06d77fe640 1 -- 172.21.15.1:0/2021608557 reap_dead start 2024-05-09T05:06:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad2d9d640 1 -- 172.21.15.1:0/2433381124 reap_dead start 2024-05-09T05:06:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 shutdown_connections 2024-05-09T05:06:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 >> 172.21.15.1:0/2433381124 conn(0x7f6acc050800 msgr2=0x7f6acc04ae70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa06d9d640 1 -- 172.21.15.1:0/1080556712 reap_dead start 2024-05-09T05:06:12.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7f0e7007e670 msgr2=0x7f0e70080ac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 shutdown_connections 2024-05-09T05:06:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 >> 172.21.15.1:0/1080556712 conn(0x7faa00054840 msgr2=0x7faa00052900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f0e76ffd640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7f0e7007e670 0x7f0e70080ac0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 shutdown_connections 2024-05-09T05:06:12.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06bdffb640 1 -- 172.21.15.1:0/2021608557 shutdown_connections 2024-05-09T05:06:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f06bdffb640 1 -- 172.21.15.1:0/2021608557 >> 172.21.15.1:0/2021608557 conn(0x7f06d00503e0 msgr2=0x7f06d004d260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f0a6d266640 1 -- 172.21.15.1:0/3852886321 wait complete. 2024-05-09T05:06:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.927+0000 7f6ad359e640 1 -- 172.21.15.1:0/2433381124 wait complete. 2024-05-09T05:06:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0e700763a0 msgr2=0x7f0e700788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e76ffd640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0e700763a0 0x7f0e700788b0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0e80034000 tx=0x7f0e8002c000 comp rx=0 tx=0).stop 2024-05-09T05:06:12.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e840d7430 msgr2=0x7f0e8417d760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e76ffd640 1 --2- 172.21.15.1:0/772851468 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0e840d7430 0x7f0e8417d760 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0e8400e8f0 tx=0x7f0e7800c820 comp rx=0 tx=0).stop 2024-05-09T05:06:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e9150f640 1 -- 172.21.15.1:0/772851468 reap_dead start 2024-05-09T05:06:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 shutdown_connections 2024-05-09T05:06:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7faa0759e640 1 -- 172.21.15.1:0/1080556712 wait complete. 2024-05-09T05:06:12.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7fe39effd640 1 -- 172.21.15.1:0/1140921640 <== osd.2 v2:172.21.15.1:6800/239705464 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7fe390002dd0 con 0x7fe38c07e7d0 2024-05-09T05:06:12.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 shutdown_connections 2024-05-09T05:06:12.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 >> 172.21.15.1:0/772851468 conn(0x7f0e840503b0 msgr2=0x7f0e840d63e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 --> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f3ed80363e0 con 0x7f3ec407e5f0 2024-05-09T05:06:12.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f06bdffb640 1 -- 172.21.15.1:0/2021608557 shutdown_connections 2024-05-09T05:06:12.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f06bdffb640 1 -- 172.21.15.1:0/2021608557 wait complete. 2024-05-09T05:06:12.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f3ed67fc640 1 -- 172.21.15.1:0/1132563465 <== osd.6 v2:172.21.15.196:6800/3854610795 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f3ed80363e0 con 0x7f3ec407e5f0 2024-05-09T05:06:12.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7fe38c07e7d0 msgr2=0x7fe38c080c20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7fe38c07e7d0 0x7fe38c080c20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 shutdown_connections 2024-05-09T05:06:12.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.928+0000 7f0e76ffd640 1 -- 172.21.15.1:0/772851468 wait complete. 2024-05-09T05:06:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe38c076500 msgr2=0x7fe38c078a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe38c076500 0x7fe38c078a10 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0000f60 tx=0x7fe3a0009040 comp rx=0 tx=0).stop 2024-05-09T05:06:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a80cbf60 msgr2=0x7fe3a8151030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 --2- 172.21.15.1:0/1140921640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe3a80cbf60 0x7fe3a8151030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe39800bb10 tx=0x7fe398004730 comp rx=0 tx=0).stop 2024-05-09T05:06:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3aea45640 1 -- 172.21.15.1:0/1140921640 reap_dead start 2024-05-09T05:06:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 shutdown_connections 2024-05-09T05:06:12.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 >> 172.21.15.1:0/1140921640 conn(0x7fe3a80ce180 msgr2=0x7fe3a8148ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 shutdown_connections 2024-05-09T05:06:12.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7fe3b04a8640 1 -- 172.21.15.1:0/1140921640 wait complete. 2024-05-09T05:06:12.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7f3ec407e5f0 msgr2=0x7f3ec4080a40 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7f3ec3fff640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7f3ec407e5f0 0x7f3ec4080a40 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:12.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3ec4076320 msgr2=0x7f3ec4078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7f3ec3fff640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3ec4076320 0x7f3ec4078830 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f3ed00079a0 tx=0x7f3ed0009040 comp rx=0 tx=0).stop 2024-05-09T05:06:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed8049a60 msgr2=0x7f3ed817d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7f3ec3fff640 1 --2- 172.21.15.1:0/1132563465 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3ed8049a60 0x7f3ed817d4c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3ee006b1b0 tx=0x7f3ee0069e50 comp rx=0 tx=0).stop 2024-05-09T05:06:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.929+0000 7f3ee65df640 1 -- 172.21.15.1:0/1132563465 reap_dead start 2024-05-09T05:06:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.930+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 shutdown_connections 2024-05-09T05:06:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.930+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 >> 172.21.15.1:0/1132563465 conn(0x7f3ed80503b0 msgr2=0x7f3ed804bb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.930+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 shutdown_connections 2024-05-09T05:06:12.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:12.930+0000 7f3ec3fff640 1 -- 172.21.15.1:0/1132563465 wait complete. 2024-05-09T05:06:12.996 INFO:teuthology.orchestra.run.smithi001.stdout:60129542152 2024-05-09T05:06:12.996 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-09T05:06:12.997 INFO:teuthology.orchestra.run.smithi001.stdout:60129542152 2024-05-09T05:06:12.997 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-09T05:06:12.998 INFO:teuthology.orchestra.run.smithi001.stdout:64424509447 2024-05-09T05:06:12.998 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-09T05:06:12.999 INFO:teuthology.orchestra.run.smithi001.stdout:60129542152 2024-05-09T05:06:12.999 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-09T05:06:13.000 INFO:teuthology.orchestra.run.smithi001.stdout:60129542151 2024-05-09T05:06:13.000 INFO:teuthology.orchestra.run.smithi001.stdout:64424509446 2024-05-09T05:06:13.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-09T05:06:13.001 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-09T05:06:13.002 INFO:teuthology.orchestra.run.smithi001.stdout:60129542151 2024-05-09T05:06:13.002 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-09T05:06:13.003 INFO:teuthology.orchestra.run.smithi001.stdout:60129542153 2024-05-09T05:06:13.004 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-09T05:06:13.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.254+0000 7f1857534640 1 Processor -- start 2024-05-09T05:06:13.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.254+0000 7f1857534640 1 -- start start 2024-05-09T05:06:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.256+0000 7f11b9c9f640 1 Processor -- start 2024-05-09T05:06:13.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.256+0000 7f11b9c9f640 1 -- start start 2024-05-09T05:06:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.257+0000 7f1857534640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1848045350 0x7f18480060e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.257+0000 7f1857534640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18480349e0 0x7f184804c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.257+0000 7f1856532640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1848045350 0x7f18480060e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.257+0000 7f1856532640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1848045350 0x7f18480060e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49268/0 (socket says 172.21.15.1:49268) 2024-05-09T05:06:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.257+0000 7f1856532640 1 -- 172.21.15.1:0/2549187578 learned_addr learned my addr 172.21.15.1:0/2549187578 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.257+0000 7f1857534640 1 -- 172.21.15.1:0/2549187578 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18480352b0 con 0x7f1848036070 2024-05-09T05:06:13.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.257+0000 7f1857534640 1 -- 172.21.15.1:0/2549187578 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f1848033150 con 0x7f1848045350 2024-05-09T05:06:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.257+0000 7f1857534640 1 -- 172.21.15.1:0/2549187578 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1848035be0 con 0x7f18480349e0 2024-05-09T05:06:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f1856532640 1 -- 172.21.15.1:0/2549187578 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18480349e0 msgr2=0x7f184804c8c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:06:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f1856532640 1 --2- 172.21.15.1:0/2549187578 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18480349e0 0x7f184804c8c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f1856532640 1 -- 172.21.15.1:0/2549187578 >> v1:172.21.15.196:6789/0 conn(0x7f1848036070 legacy=0x7f18481004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f1856532640 1 -- 172.21.15.1:0/2549187578 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1848035590 con 0x7f1848045350 2024-05-09T05:06:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f11b9c9f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11ac00e4b0 0x7f11ac006110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f11b9c9f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f11ac04c970 0x7f11ac0d5320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f11b9c9f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac0471e0 0x7f11ac0de940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f11b9c9f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f11ac046a70 con 0x7f11ac00e4b0 2024-05-09T05:06:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f11b9c9f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f11ac0335a0 con 0x7f11ac04c970 2024-05-09T05:06:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f11b9c9f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f11ac0df710 con 0x7f11ac0471e0 2024-05-09T05:06:13.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11ac00e4b0 0x7f11ac006110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f11ac00e4b0 0x7f11ac006110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49272/0 (socket says 172.21.15.1:49272) 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b37fe640 1 -- 172.21.15.1:0/3608609173 learned_addr learned my addr 172.21.15.1:0/3608609173 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b3fff640 1 --2- 172.21.15.1:0/3608609173 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac0471e0 0x7f11ac0de940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b2ffd640 1 --2- 172.21.15.1:0/3608609173 >> v2:172.21.15.196:3300/0 conn(0x7f11ac04c970 0x7f11ac0d5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b37fe640 1 -- 172.21.15.1:0/3608609173 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac0471e0 msgr2=0x7f11ac0de940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b37fe640 1 --2- 172.21.15.1:0/3608609173 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac0471e0 0x7f11ac0de940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b37fe640 1 -- 172.21.15.1:0/3608609173 >> v2:172.21.15.196:3300/0 conn(0x7f11ac04c970 msgr2=0x7f11ac0d5320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.260+0000 7f1856532640 1 --2- 172.21.15.1:0/2549187578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480060e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f184c009ad0 tx=0x7f184c03b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf4e7b7375de9b4f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.260+0000 7f1855530640 1 -- 172.21.15.1:0/2549187578 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c047070 con 0x7f1848045350 2024-05-09T05:06:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.260+0000 7f1855530640 1 -- 172.21.15.1:0/2549187578 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f184c042600 con 0x7f1848045350 2024-05-09T05:06:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b37fe640 1 --2- 172.21.15.1:0/3608609173 >> v2:172.21.15.196:3300/0 conn(0x7f11ac04c970 0x7f11ac0d5320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f11b37fe640 1 -- 172.21.15.1:0/3608609173 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ac0df470 con 0x7f11ac00e4b0 2024-05-09T05:06:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.260+0000 7f11b37fe640 1 --2- 172.21.15.1:0/3608609173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac00e4b0 0x7f11ac006110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f11ac0df640 tx=0x7f11b40a78b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f584f37a012ab1f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f1855530640 1 -- 172.21.15.1:0/2549187578 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f184c0429d0 con 0x7f1848045350 2024-05-09T05:06:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.258+0000 7f8a05f66640 1 Processor -- start 2024-05-09T05:06:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.259+0000 7f8a05f66640 1 -- start start 2024-05-09T05:06:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f8a05f66640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a000c4d70 0x7f8a000c9510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f8a05f66640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a00073210 con 0x7f8a0015a6b0 2024-05-09T05:06:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f8a05f66640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a000c68e0 con 0x7f8a001612b0 2024-05-09T05:06:13.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f8a05f66640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8a000734f0 con 0x7f8a000c4d70 2024-05-09T05:06:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f89ff7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a000c4d70 0x7f8a000c9510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f89ff7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a000c4d70 0x7f8a000c9510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49278/0 (socket says 172.21.15.1:49278) 2024-05-09T05:06:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f89ff7fe640 1 -- 172.21.15.1:0/2897245454 learned_addr learned my addr 172.21.15.1:0/2897245454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f11b27fc640 1 -- 172.21.15.1:0/3608609173 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b40ad420 con 0x7f11ac00e4b0 2024-05-09T05:06:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f11b27fc640 1 -- 172.21.15.1:0/3608609173 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b40ad5c0 con 0x7f11ac00e4b0 2024-05-09T05:06:13.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f11b27fc640 1 -- 172.21.15.1:0/3608609173 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b40ada10 con 0x7f11ac00e4b0 2024-05-09T05:06:13.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.262+0000 7f1857534640 1 -- 172.21.15.1:0/2549187578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 msgr2=0x7f18480060e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.262+0000 7f1857534640 1 --2- 172.21.15.1:0/2549187578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480060e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f184c009ad0 tx=0x7f184c03b4d0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.262+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3608609173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac00e4b0 msgr2=0x7f11ac006110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.262+0000 7f11b9c9f640 1 --2- 172.21.15.1:0/3608609173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac00e4b0 0x7f11ac006110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f11ac0df640 tx=0x7f11b40a78b0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7f89ff7fe640 1 -- 172.21.15.1:0/2897245454 >> v1:172.21.15.196:6789/0 conn(0x7f8a001612b0 legacy=0x7f8a0016b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.262+0000 7f89ff7fe640 1 -- 172.21.15.1:0/2897245454 >> v1:172.21.15.1:6789/0 conn(0x7f8a0015a6b0 legacy=0x7f8a00160ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.262+0000 7f89ff7fe640 1 -- 172.21.15.1:0/2897245454 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a000c6bc0 con 0x7f8a000c4d70 2024-05-09T05:06:13.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.261+0000 7fb232b4d640 1 Processor -- start 2024-05-09T05:06:13.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.262+0000 7fb232b4d640 1 -- start start 2024-05-09T05:06:13.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f89ff7fe640 1 --2- 172.21.15.1:0/2897245454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000c4d70 0x7f8a000c9510 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009e80 tx=0x7f89f002a870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d528d7ad842f8bca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f1857534640 1 -- 172.21.15.1:0/2549187578 shutdown_connections 2024-05-09T05:06:13.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f1857534640 1 --2- 172.21.15.1:0/2549187578 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18480349e0 0x7f184804c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f1857534640 1 --2- 172.21.15.1:0/2549187578 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480060e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f1857534640 1 -- 172.21.15.1:0/2549187578 >> 172.21.15.1:0/2549187578 conn(0x7f18480503b0 msgr2=0x7f18480507c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3608609173 shutdown_connections 2024-05-09T05:06:13.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f1857534640 1 -- 172.21.15.1:0/2549187578 shutdown_connections 2024-05-09T05:06:13.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f11b9c9f640 1 --2- 172.21.15.1:0/3608609173 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac0471e0 0x7f11ac0de940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f11b9c9f640 1 --2- 172.21.15.1:0/3608609173 >> v2:172.21.15.196:3300/0 conn(0x7f11ac04c970 0x7f11ac0d5320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f11b9c9f640 1 --2- 172.21.15.1:0/3608609173 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac00e4b0 0x7f11ac006110 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3608609173 >> 172.21.15.1:0/3608609173 conn(0x7f11ac0503e0 msgr2=0x7f11ac0507f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f89fe7fc640 1 -- 172.21.15.1:0/2897245454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f0037070 con 0x7f8a000c4d70 2024-05-09T05:06:13.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f89fe7fc640 1 -- 172.21.15.1:0/2897245454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89f0032420 con 0x7f8a000c4d70 2024-05-09T05:06:13.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f89fe7fc640 1 -- 172.21.15.1:0/2897245454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89f00327f0 con 0x7f8a000c4d70 2024-05-09T05:06:13.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.263+0000 7f1857534640 1 -- 172.21.15.1:0/2549187578 wait complete. 2024-05-09T05:06:13.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.264+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3608609173 shutdown_connections 2024-05-09T05:06:13.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.264+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3608609173 wait complete. 2024-05-09T05:06:13.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.264+0000 7fb2308e9640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb22c08d190 0x7fb22c06c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:57652/0 (socket says 172.21.15.1:57652) 2024-05-09T05:06:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8a05f66640 1 -- 172.21.15.1:0/2897245454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000c4d70 msgr2=0x7f8a000c9510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8a05f66640 1 --2- 172.21.15.1:0/2897245454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000c4d70 0x7f8a000c9510 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f89f0009e80 tx=0x7f89f002a870 comp rx=0 tx=0).stop 2024-05-09T05:06:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.264+0000 7fb22bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb22c08d6e0 0x7fb22c0c6ab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35584/0 (socket says 172.21.15.1:35584) 2024-05-09T05:06:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8009f00640 1 Processor -- start 2024-05-09T05:06:13.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.264+0000 7fb232b4d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb22c0c71c0 0x7fb22c174d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8a05f66640 1 -- 172.21.15.1:0/2897245454 shutdown_connections 2024-05-09T05:06:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8a05f66640 1 --2- 172.21.15.1:0/2897245454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a000c4d70 0x7f8a000c9510 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8a05f66640 1 -- 172.21.15.1:0/2897245454 >> 172.21.15.1:0/2897245454 conn(0x7f8a00070fa0 msgr2=0x7f8a000cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.264+0000 7fb2308e9640 1 -- 172.21.15.1:0/20053188 learned_addr learned my addr 172.21.15.1:0/20053188 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:13.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8009f00640 1 -- start start 2024-05-09T05:06:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8a05f66640 1 -- 172.21.15.1:0/2897245454 shutdown_connections 2024-05-09T05:06:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.264+0000 7fb232b4d640 1 -- 172.21.15.1:0/20053188 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c0731b0 con 0x7fb22c08d6e0 2024-05-09T05:06:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.264+0000 7fb2310ea640 1 --2- 172.21.15.1:0/20053188 >> v2:172.21.15.196:3300/0 conn(0x7fb22c0c71c0 0x7fb22c174d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7fb232b4d640 1 -- 172.21.15.1:0/20053188 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb22c073470 con 0x7fb22c08d190 2024-05-09T05:06:13.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7fb232b4d640 1 -- 172.21.15.1:0/20053188 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb22c073990 con 0x7fb22c0c71c0 2024-05-09T05:06:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3179846253 0 0) 0x7fb22c0731b0 con 0x7fb22c08d6e0 2024-05-09T05:06:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb214007930 con 0x7fb22c08d6e0 2024-05-09T05:06:13.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7f8a05f66640 1 -- 172.21.15.1:0/2897245454 wait complete. 2024-05-09T05:06:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1115328171 0 0) 0x7fb22c073470 con 0x7fb22c08d190 2024-05-09T05:06:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb22c0731b0 con 0x7fb22c08d190 2024-05-09T05:06:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3816506565 0 0) 0x7fb214007930 con 0x7fb22c08d6e0 2024-05-09T05:06:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 >> v2:172.21.15.196:3300/0 conn(0x7fb22c0c71c0 msgr2=0x7fb22c174d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 --2- 172.21.15.1:0/20053188 >> v2:172.21.15.196:3300/0 conn(0x7fb22c0c71c0 0x7fb22c174d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 >> v1:172.21.15.196:6789/0 conn(0x7fb22c08d190 legacy=0x7fb22c06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c073c70 con 0x7fb22c08d6e0 2024-05-09T05:06:13.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7f1857534640 1 Processor -- start 2024-05-09T05:06:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb2310ea640 1 --2- 172.21.15.1:0/20053188 >> v2:172.21.15.196:3300/0 conn(0x7fb22c0c71c0 0x7fb22c174d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:13.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7f1857534640 1 -- start start 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fb21c0015a0 con 0x7fb22c08d6e0 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 >> v1:172.21.15.1:6789/0 conn(0x7fb22c08d6e0 legacy=0x7fb22c0c6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb22b7fe640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 0x7fb214008590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb22b7fe640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb214008c10 0x7fb2140090e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb2308e9640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 0x7fb214008590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb22bfff640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb214008c10 0x7fb2140090e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb22b7fe640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2140097a0 0x7fb21400fed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c073990 con 0x7fb214008c10 2024-05-09T05:06:13.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb2140107b0 con 0x7fb2140097a0 2024-05-09T05:06:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb2140109b0 con 0x7fb214008160 2024-05-09T05:06:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7ff2d9f6d640 1 Processor -- start 2024-05-09T05:06:13.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.265+0000 7ff2d9f6d640 1 -- start start 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7f8009f00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 0x7f800415adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7f8009f00640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8004160b30 0x7f80040d1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7f8009f00640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80040cd320 con 0x7f8004160f40 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7f8009f00640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f80040cd8c0 con 0x7f8004160b30 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7f8009f00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f80040c77e0 con 0x7f80040c8990 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7f8003fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8004160f40 0x7f8004164120 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35592/0 (socket says 172.21.15.1:35592) 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7f8003fff640 1 -- 172.21.15.1:0/844761180 learned_addr learned my addr 172.21.15.1:0/844761180 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2298686519 0 0) 0x7f80040cd320 con 0x7f8004160f40 2024-05-09T05:06:13.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7f259a446640 1 Processor -- start 2024-05-09T05:06:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.266+0000 7f259a446640 1 -- start start 2024-05-09T05:06:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7f259a446640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c2c80 0x7f259406fba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7f259a446640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2594155cb0 0x7f259406ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f259a446640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25940c9f90 con 0x7f25941587d0 2024-05-09T05:06:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f259a446640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f25940ca510 con 0x7f2594155cb0 2024-05-09T05:06:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f259a446640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f25940ca230 con 0x7f25940c2c80 2024-05-09T05:06:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7fec003520 con 0x7f8004160f40 2024-05-09T05:06:13.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7f80037fe640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 0x7f800415adf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7fb2310ea640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2140097a0 0x7fb21400fed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7fb2308e9640 1 -- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2140097a0 msgr2=0x7fb21400fed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7fb2308e9640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2140097a0 0x7fb21400fed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7fb2308e9640 1 -- 172.21.15.1:0/20053188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb214008c10 msgr2=0x7fb2140090e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7fb2308e9640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb214008c10 0x7fb2140090e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7fb2308e9640 1 -- 172.21.15.1:0/20053188 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb214007930 con 0x7fb214008160 2024-05-09T05:06:13.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7fb2310ea640 1 -- 172.21.15.1:0/20053188 reap_dead start 2024-05-09T05:06:13.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7ff2d9f6d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc04c8b0 0x7ff2cc0f6fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7ff2d9f6d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2cc1004e0 0x7ff2cc1008f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7ff2d976c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2cc1004e0 0x7ff2cc1008f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7ff2d976c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff2cc1004e0 0x7ff2cc1008f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49282/0 (socket says 172.21.15.1:49282) 2024-05-09T05:06:13.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7ff2d976c640 1 -- 172.21.15.1:0/4171654277 learned_addr learned my addr 172.21.15.1:0/4171654277 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/4171654277 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff2cc006530 con 0x7ff2cc045340 2024-05-09T05:06:13.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/4171654277 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff2cc033140 con 0x7ff2cc1004e0 2024-05-09T05:06:13.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.268+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/4171654277 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff2cc006e60 con 0x7ff2cc04c8b0 2024-05-09T05:06:13.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7fb22bfff640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb214008c10 0x7fb2140090e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:13.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2522039694 0 0) 0x7ff2cc006530 con 0x7ff2cc045340 2024-05-09T05:06:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7fb2308e9640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 0x7fb214008590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb224001fd0 tx=0x7fb214009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5531ddcf24ec68d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7f11b9c9f640 1 Processor -- start 2024-05-09T05:06:13.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.267+0000 7f11b9c9f640 1 -- start start 2024-05-09T05:06:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f2599444640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c2c80 0x7f259406fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 854208903 0 0) 0x7f7fec003520 con 0x7f8004160f40 2024-05-09T05:06:13.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff2c4003520 con 0x7ff2cc045340 2024-05-09T05:06:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f8002ffd640 1 --2- 172.21.15.1:0/844761180 >> v2:172.21.15.1:3300/0 conn(0x7f8004160b30 0x7f80040d1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 msgr2=0x7f800415adf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f2599c45640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2594155cb0 0x7f259406ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f2599c45640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2594155cb0 0x7f259406ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49290/0 (socket says 172.21.15.1:49290) 2024-05-09T05:06:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f2599c45640 1 -- 172.21.15.1:0/1358030814 learned_addr learned my addr 172.21.15.1:0/1358030814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f2598c43640 1 --1- 172.21.15.1:0/1358030814 >> v1:172.21.15.196:6789/0 conn(0x7f25941587d0 0x7f259415c700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:57664/0 (socket says 172.21.15.1:57664) 2024-05-09T05:06:13.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f80027fc640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 0x7f800415adf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 >> v2:172.21.15.1:3300/0 conn(0x7f8004160b30 msgr2=0x7f80040d1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f8a75914640 1 Processor -- start 2024-05-09T05:06:13.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f80027fc640 1 --2- 172.21.15.1:0/844761180 >> v2:172.21.15.1:3300/0 conn(0x7f8004160b30 0x7f80040d1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2252175264 0 0) 0x7ff2c4003520 con 0x7ff2cc045340 2024-05-09T05:06:13.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc04c8b0 msgr2=0x7ff2cc0f6fc0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:06:13.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc04c8b0 0x7ff2cc0f6fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80040cd5e0 con 0x7f8004160f40 2024-05-09T05:06:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 >> v2:172.21.15.1:3300/0 conn(0x7ff2cc1004e0 msgr2=0x7ff2cc1008f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f7ffc001a70 con 0x7f8004160f40 2024-05-09T05:06:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80037fe640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 0x7f800415adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:13.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f1857534640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18480349e0 0x7f18480f81b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 >> v1:172.21.15.1:6789/0 conn(0x7f8004160f40 legacy=0x7f8004164120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 --2- 172.21.15.1:0/4171654277 >> v2:172.21.15.1:3300/0 conn(0x7ff2cc1004e0 0x7ff2cc1008f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc006810 con 0x7ff2cc045340 2024-05-09T05:06:13.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7ff2c8001760 con 0x7ff2cc045340 2024-05-09T05:06:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 >> v1:172.21.15.196:6789/0 conn(0x7ff2cc045340 legacy=0x7ff2cc0072f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c4003a00 0x7ff2c4003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4004800 0x7ff2c4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2c4005350 0x7ff2c400b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc033140 con 0x7ff2c4003a00 2024-05-09T05:06:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc006e60 con 0x7ff2c4004800 2024-05-09T05:06:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff2c400c2e0 con 0x7ff2c4005350 2024-05-09T05:06:13.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f1857534640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1848036070 0x7f18480fc790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 0x7f7fec003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7ff2d976c640 1 --2- 172.21.15.1:0/4171654277 >> v2:172.21.15.1:3300/0 conn(0x7ff2cc1004e0 0x7ff2cc1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b9c9f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac00e4b0 0x7f11ac048ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b9c9f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac0471e0 0x7f11ac0d6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b37fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac00e4b0 0x7f11ac048ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb224002010 con 0x7fb214008160 2024-05-09T05:06:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fec004060 0x7f7fec004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7fb22b7fe640 1 -- 172.21.15.1:0/20053188 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb224037070 con 0x7fb214008160 2024-05-09T05:06:13.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b37fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac00e4b0 0x7f11ac048ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43996/0 (socket says 172.21.15.1:43996) 2024-05-09T05:06:13.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b37fe640 1 -- 172.21.15.1:0/3283414143 learned_addr learned my addr 172.21.15.1:0/3283414143 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fec004d40 0x7f7fec005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b9c9f640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac04c970 0x7f11ac047d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3283414143 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11ac046cc0 con 0x7f11ac0471e0 2024-05-09T05:06:13.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3283414143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f11ac04bfb0 con 0x7f11ac00e4b0 2024-05-09T05:06:13.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3283414143 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f11ac0344b0 con 0x7f11ac04c970 2024-05-09T05:06:13.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b3fff640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac04c970 0x7f11ac047d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b2ffd640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac0471e0 0x7f11ac0d6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f8a05f66640 1 Processor -- start 2024-05-09T05:06:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80037fe640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 0x7f7fec003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.269+0000 7f1857534640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480fd0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f8003fff640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fec004d40 0x7f7fec005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f80040cd8c0 con 0x7f7fec004d40 2024-05-09T05:06:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f80040c77e0 con 0x7f7fec004060 2024-05-09T05:06:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7fec00bbd0 con 0x7f80040c8990 2024-05-09T05:06:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f8002ffd640 1 --2- 172.21.15.1:0/844761180 >> v2:172.21.15.1:3300/0 conn(0x7f8004160b30 0x7f80040d1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f1857534640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f18480f7420 con 0x7f1848045350 2024-05-09T05:06:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f1857534640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1848036480 con 0x7f18480349e0 2024-05-09T05:06:13.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f1857534640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f18480f75a0 con 0x7f1848036070 2024-05-09T05:06:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b37fe640 1 -- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac04c970 msgr2=0x7f11ac047d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b37fe640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac04c970 0x7f11ac047d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b37fe640 1 -- 172.21.15.1:0/3283414143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac0471e0 msgr2=0x7f11ac0d6100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f11b37fe640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac0471e0 0x7f11ac0d6100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f11b37fe640 1 -- 172.21.15.1:0/3283414143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11ac0d8040 con 0x7f11ac00e4b0 2024-05-09T05:06:13.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f1856d33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480fd0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f11b3fff640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac04c970 0x7f11ac047d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:13.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f11b37fe640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac00e4b0 0x7f11ac048ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f11b402e1c0 tx=0x7f11b4077ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f8a75914640 1 -- start start 2024-05-09T05:06:13.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f2599c45640 1 -- 172.21.15.1:0/1358030814 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c2c80 msgr2=0x7f259406fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f2599c45640 1 --2- 172.21.15.1:0/1358030814 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c2c80 0x7f259406fba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f2599c45640 1 -- 172.21.15.1:0/1358030814 >> v1:172.21.15.196:6789/0 conn(0x7f25941587d0 legacy=0x7f259415c700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.270+0000 7f2599c45640 1 -- 172.21.15.1:0/1358030814 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25940c48c0 con 0x7f2594155cb0 2024-05-09T05:06:13.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f1856d33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480fd0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49308/0 (socket says 172.21.15.1:49308) 2024-05-09T05:06:13.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f1856d33640 1 -- 172.21.15.1:0/3600545105 learned_addr learned my addr 172.21.15.1:0/3600545105 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f119ffff640 1 -- 172.21.15.1:0/3283414143 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b40be020 con 0x7f11ac00e4b0 2024-05-09T05:06:13.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f119ffff640 1 -- 172.21.15.1:0/3283414143 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11b40ae9c0 con 0x7f11ac00e4b0 2024-05-09T05:06:13.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f119ffff640 1 -- 172.21.15.1:0/3283414143 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11b40ad420 con 0x7f11ac00e4b0 2024-05-09T05:06:13.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3283414143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11ac0d8320 con 0x7f11ac00e4b0 2024-05-09T05:06:13.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f80037fe640 1 -- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fec004060 msgr2=0x7f7fec004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f80037fe640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7fec004060 0x7f7fec004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f80037fe640 1 -- 172.21.15.1:0/844761180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fec004d40 msgr2=0x7f7fec005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f80037fe640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7fec004d40 0x7f7fec005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f80037fe640 1 -- 172.21.15.1:0/844761180 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7fec003520 con 0x7f80040c8990 2024-05-09T05:06:13.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3283414143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11ac0489a0 con 0x7f11ac00e4b0 2024-05-09T05:06:13.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f1856d33640 1 -- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1848036070 msgr2=0x7f18480fc790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f1856d33640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1848036070 0x7f18480fc790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f1856d33640 1 -- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18480349e0 msgr2=0x7f18480f81b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:06:13.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f1856d33640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18480349e0 0x7f18480f81b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f1856d33640 1 -- 172.21.15.1:0/3600545105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18480f93a0 con 0x7f1848045350 2024-05-09T05:06:13.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f1856d33640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480fd0c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f185006bc20 tx=0x7f1850069e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7f8a05f66640 1 -- start start 2024-05-09T05:06:13.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f2599c45640 1 --2- 172.21.15.1:0/1358030814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2594155cb0 0x7f259406ed30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25940ca690 tx=0x7f258c02bae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0dab2a368a3be4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7ff2d976c640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2c4005350 0x7ff2c400b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7ff2d8f6b640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c4003a00 0x7ff2c4003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7ff2d976c640 1 -- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4004800 msgr2=0x7ff2c4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7ff2d976c640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4004800 0x7ff2c4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7ff2d976c640 1 -- 172.21.15.1:0/4171654277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c4003a00 msgr2=0x7ff2c4003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7ff2d976c640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2c4003a00 0x7ff2c4003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7ff2d976c640 1 -- 172.21.15.1:0/4171654277 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc006810 con 0x7ff2c4005350 2024-05-09T05:06:13.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7fb232b4d640 1 -- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 msgr2=0x7fb214008590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7fb232b4d640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 0x7fb214008590 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fb224001fd0 tx=0x7fb214009c40 comp rx=0 tx=0).stop 2024-05-09T05:06:13.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7fb232b4d640 1 -- 172.21.15.1:0/20053188 shutdown_connections 2024-05-09T05:06:13.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7fb232b4d640 1 --2- 172.21.15.1:0/20053188 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 0x7fb214008590 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.271+0000 7fb232b4d640 1 -- 172.21.15.1:0/20053188 >> 172.21.15.1:0/20053188 conn(0x7fb22c070fa0 msgr2=0x7fb22c0cd870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7fb232b4d640 1 -- 172.21.15.1:0/20053188 shutdown_connections 2024-05-09T05:06:13.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f8a75914640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8a7015ae30 0x7f8a700c9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f8a75914640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a700ca3c0 0x7f8a700ca7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f8a75914640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a70164f00 con 0x7f8a700740c0 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f8a75914640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8a70074d20 con 0x7f8a700ca3c0 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f8a75914640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8a700c4e00 con 0x7f8a7015ae30 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f8a6f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a700ca3c0 0x7f8a700ca7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f8a6f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a700ca3c0 0x7f8a700ca7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49346/0 (socket says 172.21.15.1:49346) 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f8a6f7fe640 1 -- 172.21.15.1:0/3315657100 learned_addr learned my addr 172.21.15.1:0/3315657100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f8a6e7fc640 1 --2- 172.21.15.1:0/3315657100 >> v2:172.21.15.196:3300/0 conn(0x7f8a7015ae30 0x7f8a700c9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f8a05f66640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a000c4d70 0x7f8a0014e870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f258bfff640 1 -- 172.21.15.1:0/1358030814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c033070 con 0x7f2594155cb0 2024-05-09T05:06:13.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f258bfff640 1 -- 172.21.15.1:0/1358030814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f258c02d630 con 0x7f2594155cb0 2024-05-09T05:06:13.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f258bfff640 1 -- 172.21.15.1:0/1358030814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c02da00 con 0x7f2594155cb0 2024-05-09T05:06:13.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f8a6f7fe640 1 -- 172.21.15.1:0/3315657100 >> v2:172.21.15.196:3300/0 conn(0x7f8a7015ae30 msgr2=0x7f8a700c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f8a6f7fe640 1 --2- 172.21.15.1:0/3315657100 >> v2:172.21.15.196:3300/0 conn(0x7f8a7015ae30 0x7f8a700c9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f8a6f7fe640 1 -- 172.21.15.1:0/3315657100 >> v1:172.21.15.1:6789/0 conn(0x7f8a700740c0 legacy=0x7f8a70074610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:13.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f8a05f66640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0015a6b0 0x7f8a000cc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f89feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0015a6b0 0x7f8a000cc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f89feffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0015a6b0 0x7f8a000cc8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49354/0 (socket says 172.21.15.1:49354) 2024-05-09T05:06:13.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f89feffd640 1 -- 172.21.15.1:0/984681529 learned_addr learned my addr 172.21.15.1:0/984681529 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f8a05f66640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a001612b0 0x7f8a0015b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7f1846ffd640 1 -- 172.21.15.1:0/3600545105 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f185006de60 con 0x7f1848045350 2024-05-09T05:06:13.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f1846ffd640 1 -- 172.21.15.1:0/3600545105 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1850084600 con 0x7f1848045350 2024-05-09T05:06:13.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f1846ffd640 1 -- 172.21.15.1:0/3600545105 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18500849f0 con 0x7f1848045350 2024-05-09T05:06:13.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f1857534640 1 -- 172.21.15.1:0/3600545105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18480fb450 con 0x7f1848045350 2024-05-09T05:06:13.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f1857534640 1 -- 172.21.15.1:0/3600545105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1848193e60 con 0x7f1848045350 2024-05-09T05:06:13.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.273+0000 7f8a6f7fe640 1 -- 172.21.15.1:0/3315657100 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a700c97d0 con 0x7f8a700ca3c0 2024-05-09T05:06:13.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f119ffff640 1 -- 172.21.15.1:0/3283414143 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f11b40c1020 con 0x7f11ac00e4b0 2024-05-09T05:06:13.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f8a05f66640 1 -- 172.21.15.1:0/984681529 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8a000c9a50 con 0x7f8a0015a6b0 2024-05-09T05:06:13.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7ff2d976c640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2c4005350 0x7ff2c400b9e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4052170 tx=0x7ff2c4005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72e95f6a234a90c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f8a05f66640 1 -- 172.21.15.1:0/984681529 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8a000c7180 con 0x7f8a000c4d70 2024-05-09T05:06:13.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f8a05f66640 1 -- 172.21.15.1:0/984681529 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8a000c9bd0 con 0x7f8a001612b0 2024-05-09T05:06:13.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f1844ff9640 1 -- 172.21.15.1:0/3600545105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f182c005290 con 0x7f1848045350 2024-05-09T05:06:13.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.272+0000 7fb232b4d640 1 -- 172.21.15.1:0/20053188 wait complete. 2024-05-09T05:06:13.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f8a6f7fe640 1 --2- 172.21.15.1:0/3315657100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a700ca3c0 0x7f8a700ca7d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8a70074050 tx=0x7f8a6802a750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d28264f2935cae04 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7fb232b4d640 1 Processor -- start 2024-05-09T05:06:13.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f80037fe640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 0x7f7fec003a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ff80098b0 tx=0x7f7fec005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2698471b146809b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d409d070 con 0x7ff2c4005350 2024-05-09T05:06:13.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2d4097be0 con 0x7ff2c4005350 2024-05-09T05:06:13.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7ff2d37fe640 1 -- 172.21.15.1:0/4171654277 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2d4064120 con 0x7ff2c4005350 2024-05-09T05:06:13.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f8a6dffb640 1 -- 172.21.15.1:0/3315657100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68036070 con 0x7f8a700ca3c0 2024-05-09T05:06:13.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f8a6dffb640 1 -- 172.21.15.1:0/3315657100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68031600 con 0x7f8a700ca3c0 2024-05-09T05:06:13.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f8a6dffb640 1 -- 172.21.15.1:0/3315657100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68031a10 con 0x7f8a700ca3c0 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f89fffff640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a001612b0 0x7f8a0015b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.274+0000 7f89ff7fe640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a000c4d70 0x7f8a0014e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f89fffff640 1 -- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a000c4d70 msgr2=0x7f8a0014e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f89fffff640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a000c4d70 0x7f8a0014e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f89fffff640 1 -- 172.21.15.1:0/984681529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0015a6b0 msgr2=0x7f8a000cc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f89fffff640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0015a6b0 0x7f8a000cc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f89fffff640 1 -- 172.21.15.1:0/984681529 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a0014edb0 con 0x7f8a001612b0 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f89feffd640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0015a6b0 0x7f8a000cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:13.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f89ff7fe640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a000c4d70 0x7f8a0014e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:13.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3283414143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11ac0054d0 con 0x7f11ac00e4b0 2024-05-09T05:06:13.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f119ffff640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11980763a0 0x7f11980788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f89fffff640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a001612b0 0x7f8a0015b920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89e800c3c0 tx=0x7f89e800c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8037070 con 0x7f80040c8990 2024-05-09T05:06:13.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff802ad50 con 0x7f80040c8990 2024-05-09T05:06:13.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f259a446640 1 -- 172.21.15.1:0/1358030814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2594155cb0 msgr2=0x7f259406ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.275+0000 7f11b2ffd640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11980763a0 0x7f11980788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7fb232b4d640 1 -- start start 2024-05-09T05:06:13.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f259a446640 1 --2- 172.21.15.1:0/1358030814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2594155cb0 0x7f259406ed30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f25940ca690 tx=0x7f258c02bae0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f80027fc640 1 -- 172.21.15.1:0/844761180 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff800f040 con 0x7f80040c8990 2024-05-09T05:06:13.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f119ffff640 1 -- 172.21.15.1:0/3283414143 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f11b4134f00 con 0x7f11ac00e4b0 2024-05-09T05:06:13.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f89f7fff640 1 -- 172.21.15.1:0/984681529 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8017070 con 0x7f8a001612b0 2024-05-09T05:06:13.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f89f7fff640 1 -- 172.21.15.1:0/984681529 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f89e80052d0 con 0x7f8a001612b0 2024-05-09T05:06:13.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f89f7fff640 1 -- 172.21.15.1:0/984681529 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89e8003830 con 0x7f8a001612b0 2024-05-09T05:06:13.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f259a446640 1 -- 172.21.15.1:0/1358030814 shutdown_connections 2024-05-09T05:06:13.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f259a446640 1 --2- 172.21.15.1:0/1358030814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2594155cb0 0x7f259406ed30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f259a446640 1 --2- 172.21.15.1:0/1358030814 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c2c80 0x7f259406fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f259a446640 1 -- 172.21.15.1:0/1358030814 >> 172.21.15.1:0/1358030814 conn(0x7f25940cdc60 msgr2=0x7f25940ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f11b2ffd640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11980763a0 0x7f11980788b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f11ac007b60 tx=0x7f11a8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2c4005350 msgr2=0x7ff2c400b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f1846ffd640 1 -- 172.21.15.1:0/3600545105 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1850084b90 con 0x7f1848045350 2024-05-09T05:06:13.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f259a446640 1 -- 172.21.15.1:0/1358030814 shutdown_connections 2024-05-09T05:06:13.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f8a75914640 1 -- 172.21.15.1:0/3315657100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a700ca3c0 msgr2=0x7f8a700ca7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f8a75914640 1 --2- 172.21.15.1:0/3315657100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a700ca3c0 0x7f8a700ca7d0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f8a70074050 tx=0x7f8a6802a750 comp rx=0 tx=0).stop 2024-05-09T05:06:13.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f8a75914640 1 -- 172.21.15.1:0/3315657100 shutdown_connections 2024-05-09T05:06:13.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f8a75914640 1 --2- 172.21.15.1:0/3315657100 >> v2:172.21.15.196:3300/0 conn(0x7f8a7015ae30 0x7f8a700c9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f8a75914640 1 --2- 172.21.15.1:0/3315657100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a700ca3c0 0x7f8a700ca7d0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f8a75914640 1 -- 172.21.15.1:0/3315657100 >> 172.21.15.1:0/3315657100 conn(0x7f8a70070f40 msgr2=0x7f8a7006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7ff2d9f6d640 1 --2- 172.21.15.1:0/4171654277 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2c4005350 0x7ff2c400b9e0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff2d4052170 tx=0x7ff2c4005830 comp rx=0 tx=0).stop 2024-05-09T05:06:13.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7ff2d976c640 1 -- 172.21.15.1:0/4171654277 reap_dead start 2024-05-09T05:06:13.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/4171654277 shutdown_connections 2024-05-09T05:06:13.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/4171654277 >> 172.21.15.1:0/4171654277 conn(0x7ff2cc0503a0 msgr2=0x7ff2cc0507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7f8009f00640 1 -- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 msgr2=0x7f7fec003a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f8a05f66640 1 -- 172.21.15.1:0/984681529 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a00150000 con 0x7f8a001612b0 2024-05-09T05:06:13.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f8a05f66640 1 -- 172.21.15.1:0/984681529 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a001505c0 con 0x7f8a001612b0 2024-05-09T05:06:13.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/4171654277 shutdown_connections 2024-05-09T05:06:13.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7f8009f00640 1 --2- 172.21.15.1:0/844761180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80040c8990 0x7f7fec003a00 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f7ff80098b0 tx=0x7f7fec005830 comp rx=0 tx=0).stop 2024-05-09T05:06:13.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7f8003fff640 1 -- 172.21.15.1:0/844761180 reap_dead start 2024-05-09T05:06:13.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7f1846ffd640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18340763a0 0x7f18340788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7f8009f00640 1 -- 172.21.15.1:0/844761180 shutdown_connections 2024-05-09T05:06:13.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7f8009f00640 1 -- 172.21.15.1:0/844761180 >> 172.21.15.1:0/844761180 conn(0x7f800406dca0 msgr2=0x7f800406e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7f1846ffd640 1 -- 172.21.15.1:0/3600545105 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f1850103630 con 0x7f1848045350 2024-05-09T05:06:13.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.277+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/4171654277 wait complete. 2024-05-09T05:06:13.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7f8009f00640 1 -- 172.21.15.1:0/844761180 shutdown_connections 2024-05-09T05:06:13.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7f8a05f66640 1 -- 172.21.15.1:0/984681529 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a00050870 con 0x7f8a001612b0 2024-05-09T05:06:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb232b4d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 0x7fb22c162810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb232b4d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb22c2138f0 0x7fb22c15c100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb232b4d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22c202a70 0x7fb22c202e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb232b4d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c0ca120 con 0x7fb22c202a70 2024-05-09T05:06:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb232b4d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c074230 con 0x7fb22c2138f0 2024-05-09T05:06:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb232b4d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb22c163ae0 con 0x7fb214008160 2024-05-09T05:06:13.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.276+0000 7f259a446640 1 -- 172.21.15.1:0/1358030814 wait complete. 2024-05-09T05:06:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7f8009f00640 1 -- 172.21.15.1:0/844761180 wait complete. 2024-05-09T05:06:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb22bfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb22c2138f0 0x7fb22c15c100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb22bfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb22c2138f0 0x7fb22c15c100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44046/0 (socket says 172.21.15.1:44046) 2024-05-09T05:06:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb22bfff640 1 -- 172.21.15.1:0/902583579 learned_addr learned my addr 172.21.15.1:0/902583579 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7fb2310ea640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22c202a70 0x7fb22c202e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7fb22bfff640 1 -- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 msgr2=0x7fb22c162810 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:06:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7fb22bfff640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 0x7fb22c162810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7fb22bfff640 1 -- 172.21.15.1:0/902583579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22c202a70 msgr2=0x7fb22c202e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7fb22bfff640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22c202a70 0x7fb22c202e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7fb22bfff640 1 -- 172.21.15.1:0/902583579 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c200770 con 0x7fb22c2138f0 2024-05-09T05:06:13.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7fb22bfff640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb22c2138f0 0x7fb22c15c100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb21c007670 tx=0x7fb21c005bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7f1856532640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18340763a0 0x7f18340788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7f259a446640 1 Processor -- start 2024-05-09T05:06:13.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.279+0000 7f259a446640 1 -- start start 2024-05-09T05:06:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7f8a75914640 1 -- 172.21.15.1:0/3315657100 shutdown_connections 2024-05-09T05:06:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.278+0000 7f8a75914640 1 -- 172.21.15.1:0/3315657100 wait complete. 2024-05-09T05:06:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.280+0000 7f89f7fff640 1 -- 172.21.15.1:0/984681529 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f89e801a070 con 0x7f8a001612b0 2024-05-09T05:06:13.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.280+0000 7f1856532640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18340763a0 0x7f18340788b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f184c044000 tx=0x7f184c008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.280+0000 7ff2d9f6d640 1 Processor -- start 2024-05-09T05:06:13.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.280+0000 7ff2d9f6d640 1 -- start start 2024-05-09T05:06:13.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.280+0000 7f8a75914640 1 Processor -- start 2024-05-09T05:06:13.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f89f7fff640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89e00763a0 0x7f89e00788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f8009f00640 1 Processor -- start 2024-05-09T05:06:13.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f8a75914640 1 -- start start 2024-05-09T05:06:13.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f89f7fff640 1 -- 172.21.15.1:0/984681529 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f89e8096900 con 0x7f8a001612b0 2024-05-09T05:06:13.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f259a446640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c2c80 0x7f2594202b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f8009f00640 1 -- start start 2024-05-09T05:06:13.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f89ff7fe640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89e00763a0 0x7f89e00788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f259a446640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594155cb0 0x7f25940cd230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f259a446640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25941587d0 0x7f2594159930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f259a446640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f25941560c0 con 0x7f25940c2c80 2024-05-09T05:06:13.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f259a446640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2594158be0 con 0x7f2594155cb0 2024-05-09T05:06:13.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f259a446640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2594156240 con 0x7f25941587d0 2024-05-09T05:06:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f2599444640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c2c80 0x7f2594202b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f2599444640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c2c80 0x7f2594202b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49380/0 (socket says 172.21.15.1:49380) 2024-05-09T05:06:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.281+0000 7f2599444640 1 -- 172.21.15.1:0/1930106763 learned_addr learned my addr 172.21.15.1:0/1930106763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7fb2297fa640 1 -- 172.21.15.1:0/902583579 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c00e740 con 0x7fb22c2138f0 2024-05-09T05:06:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7fb2297fa640 1 -- 172.21.15.1:0/902583579 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb21c018070 con 0x7fb22c2138f0 2024-05-09T05:06:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7fb2297fa640 1 -- 172.21.15.1:0/902583579 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21c00eb00 con 0x7fb22c2138f0 2024-05-09T05:06:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22c2033c0 con 0x7fb22c2138f0 2024-05-09T05:06:13.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22c163210 con 0x7fb22c2138f0 2024-05-09T05:06:13.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7f2599444640 1 -- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25941587d0 msgr2=0x7f2594159930 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:06:13.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7f2599444640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25941587d0 0x7f2594159930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7f2599444640 1 -- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594155cb0 msgr2=0x7f25940cd230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7f2599444640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594155cb0 0x7f25940cd230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7f2599444640 1 -- 172.21.15.1:0/1930106763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25940cce20 con 0x7f25940c2c80 2024-05-09T05:06:13.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7f119ffff640 1 -- 172.21.15.1:0/3283414143 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f11b40c8030 con 0x7f11ac00e4b0 2024-05-09T05:06:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.282+0000 7f2599444640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c2c80 0x7f2594202b40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f258400c7f0 tx=0x7f258400ccb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7ff2d9f6d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4005350 0x7ff2cc0f7530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7ff2d9f6d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc10ea10 0x7ff2cc0e7df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7f89ff7fe640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89e00763a0 0x7f89e00788b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89f0034000 tx=0x7f89f0009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7ff2d9f6d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2cc0f81a0 0x7ff2cc0f8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7ff2d3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc10ea10 0x7ff2cc0e7df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7ff2d3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc10ea10 0x7ff2cc0e7df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59730/0 (socket says 172.21.15.1:59730) 2024-05-09T05:06:13.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7f8009f00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8004161cf0 0x7f80040ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7f8a75914640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a700740c0 0x7f8a701553b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7f8009f00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f800415e030 0x7f80041638e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7f8009f00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8004163e20 0x7f80041627c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7ff2d8f6b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4005350 0x7ff2cc0f7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7f8a75914640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a700ca3c0 0x7f8a70203020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.283+0000 7ff2d8f6b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4005350 0x7ff2cc0f7530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44064/0 (socket says 172.21.15.1:44064) 2024-05-09T05:06:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a75914640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7015ae30 0x7f8a70157c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7ff2d3fff640 1 -- 172.21.15.1:0/2225763946 learned_addr learned my addr 172.21.15.1:0/2225763946 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f80037fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8004161cf0 0x7f80040ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f80037fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8004161cf0 0x7f80040ce790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44070/0 (socket says 172.21.15.1:44070) 2024-05-09T05:06:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8002ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f800415e030 0x7f80041638e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8002ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f800415e030 0x7f80041638e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59738/0 (socket says 172.21.15.1:59738) 2024-05-09T05:06:13.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7ff2d976c640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2cc0f81a0 0x7ff2cc0f8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8003fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8004163e20 0x7f80041627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8003fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8004163e20 0x7f80041627c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49388/0 (socket says 172.21.15.1:49388) 2024-05-09T05:06:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f80037fe640 1 -- 172.21.15.1:0/172970822 learned_addr learned my addr 172.21.15.1:0/172970822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f80040c7470 con 0x7f8004163e20 2024-05-09T05:06:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f80040cdc10 con 0x7f8004161cf0 2024-05-09T05:06:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f80040b4830 con 0x7f800415e030 2024-05-09T05:06:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/2225763946 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc045d80 con 0x7ff2cc0f81a0 2024-05-09T05:06:13.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/2225763946 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff2cc045c00 con 0x7ff2c4005350 2024-05-09T05:06:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/2225763946 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff2cc100e30 con 0x7ff2cc10ea10 2024-05-09T05:06:13.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb20c005290 con 0x7fb22c2138f0 2024-05-09T05:06:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a6effd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a700740c0 0x7f8a701553b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a6effd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a700740c0 0x7f8a701553b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59736/0 (socket says 172.21.15.1:59736) 2024-05-09T05:06:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a700ca3c0 0x7f8a70203020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a6e7fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a700ca3c0 0x7f8a70203020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44072/0 (socket says 172.21.15.1:44072) 2024-05-09T05:06:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a6effd640 1 -- 172.21.15.1:0/572273563 learned_addr learned my addr 172.21.15.1:0/572273563 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70073610 con 0x7f8a7015ae30 2024-05-09T05:06:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8a70164c10 con 0x7f8a700ca3c0 2024-05-09T05:06:13.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.284+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8a7015b240 con 0x7f8a700740c0 2024-05-09T05:06:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d8f6b640 1 -- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc10ea10 msgr2=0x7ff2cc0e7df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d8f6b640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc10ea10 0x7ff2cc0e7df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d8f6b640 1 -- 172.21.15.1:0/2225763946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2cc0f81a0 msgr2=0x7ff2cc0f8ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d8f6b640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2cc0f81a0 0x7ff2cc0f8ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d8f6b640 1 -- 172.21.15.1:0/2225763946 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2cc0f52b0 con 0x7ff2c4005350 2024-05-09T05:06:13.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d3fff640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc10ea10 0x7ff2cc0e7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f89f7fff640 1 -- 172.21.15.1:0/984681529 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f89e8021080 con 0x7f8a001612b0 2024-05-09T05:06:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7fb2297fa640 1 -- 172.21.15.1:0/902583579 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb21c00eca0 con 0x7fb22c2138f0 2024-05-09T05:06:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d8f6b640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4005350 0x7ff2cc0f7530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800f930 tx=0x7ff2c800fdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f80037fe640 1 -- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f800415e030 msgr2=0x7f80041638e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f80037fe640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f800415e030 0x7f80041638e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f80037fe640 1 -- 172.21.15.1:0/172970822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8004163e20 msgr2=0x7f80041627c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f80037fe640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8004163e20 0x7f80041627c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f80037fe640 1 -- 172.21.15.1:0/172970822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80041fa780 con 0x7f8004161cf0 2024-05-09T05:06:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f80037fe640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8004161cf0 0x7f80040ce790 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8000c00 tx=0x7f7ff8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f8a6f7fe640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7015ae30 0x7f8a70157c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f7febfff640 1 -- 172.21.15.1:0/172970822 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8042030 con 0x7f8004161cf0 2024-05-09T05:06:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7f7febfff640 1 -- 172.21.15.1:0/172970822 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7ff8006dd0 con 0x7f8004161cf0 2024-05-09T05:06:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7fb2297fa640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb218076320 0x7fb218078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80041fb5a0 con 0x7f8004161cf0 2024-05-09T05:06:13.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7fb2297fa640 1 -- 172.21.15.1:0/902583579 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb21c05cc00 con 0x7fb22c2138f0 2024-05-09T05:06:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7fb2308e9640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb218076320 0x7fb218078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80041fad00 con 0x7f8004161cf0 2024-05-09T05:06:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7fb2308e9640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb218076320 0x7fb218078830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb22403b000 tx=0x7fb224007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f7febfff640 1 -- 172.21.15.1:0/172970822 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ff8042030 con 0x7f8004161cf0 2024-05-09T05:06:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f1846ffd640 1 -- 172.21.15.1:0/3600545105 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1850080030 con 0x7f1848045350 2024-05-09T05:06:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8a6f7fe640 1 -- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a700740c0 msgr2=0x7f8a701553b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8a6f7fe640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a700740c0 0x7f8a701553b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8a6f7fe640 1 -- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a700ca3c0 msgr2=0x7f8a70203020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8a6f7fe640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a700ca3c0 0x7f8a70203020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8a6f7fe640 1 -- 172.21.15.1:0/572273563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a70203560 con 0x7f8a7015ae30 2024-05-09T05:06:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8a6f7fe640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7015ae30 0x7f8a70157c80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8a700cb290 tx=0x7f8a68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.286+0000 7f8a6effd640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a700740c0 0x7f8a701553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d17fa640 1 -- 172.21.15.1:0/2225763946 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c801a070 con 0x7ff2c4005350 2024-05-09T05:06:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d17fa640 1 -- 172.21.15.1:0/2225763946 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff2c80053e0 con 0x7ff2c4005350 2024-05-09T05:06:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.285+0000 7ff2d17fa640 1 -- 172.21.15.1:0/2225763946 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2c8007270 con 0x7ff2c4005350 2024-05-09T05:06:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/2225763946 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2cc0fdaf0 con 0x7ff2c4005350 2024-05-09T05:06:13.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f8a57fff640 1 -- 172.21.15.1:0/572273563 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68041020 con 0x7f8a7015ae30 2024-05-09T05:06:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f2589ffb640 1 -- 172.21.15.1:0/1930106763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584015070 con 0x7f25940c2c80 2024-05-09T05:06:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f2589ffb640 1 -- 172.21.15.1:0/1930106763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2584004220 con 0x7f25940c2c80 2024-05-09T05:06:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f2589ffb640 1 -- 172.21.15.1:0/1930106763 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584003610 con 0x7f25940c2c80 2024-05-09T05:06:13.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7015b740 con 0x7f8a7015ae30 2024-05-09T05:06:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7ff2d9f6d640 1 -- 172.21.15.1:0/2225763946 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2cc0fa2f0 con 0x7ff2c4005350 2024-05-09T05:06:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7015bd00 con 0x7f8a7015ae30 2024-05-09T05:06:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8004164230 con 0x7f8004161cf0 2024-05-09T05:06:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f8a57fff640 1 -- 172.21.15.1:0/572273563 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68031600 con 0x7f8a7015ae30 2024-05-09T05:06:13.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f8a57fff640 1 -- 172.21.15.1:0/572273563 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68036040 con 0x7f8a7015ae30 2024-05-09T05:06:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.287+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25941f1ee0 con 0x7f25940c2c80 2024-05-09T05:06:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.288+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25940cc620 con 0x7f25940c2c80 2024-05-09T05:06:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.288+0000 7ff2beffd640 1 -- 172.21.15.1:0/2225763946 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2cc000e90 con 0x7ff2c4005350 2024-05-09T05:06:13.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.289+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a50005290 con 0x7f8a7015ae30 2024-05-09T05:06:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.290+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2594050870 con 0x7f25940c2c80 2024-05-09T05:06:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.290+0000 7f8a57fff640 1 -- 172.21.15.1:0/572273563 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8a68044020 con 0x7f8a7015ae30 2024-05-09T05:06:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.290+0000 7f8a57fff640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a5c0762d0 0x7f8a5c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.291+0000 7f8a57fff640 1 -- 172.21.15.1:0/572273563 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8a680b7d90 con 0x7f8a7015ae30 2024-05-09T05:06:13.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.291+0000 7f8a6effd640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a5c0762d0 0x7f8a5c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.291+0000 7f2589ffb640 1 -- 172.21.15.1:0/1930106763 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f25840039c0 con 0x7f25940c2c80 2024-05-09T05:06:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.292+0000 7fb2297fa640 1 -- 172.21.15.1:0/902583579 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb21c060d10 con 0x7fb22c2138f0 2024-05-09T05:06:13.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.292+0000 7f8a6effd640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a5c0762d0 0x7f8a5c0787e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009d80 tx=0x7f8a60009110 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.292+0000 7f2589ffb640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2578076320 0x7f2578078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.292+0000 7f2589ffb640 1 -- 172.21.15.1:0/1930106763 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f25840979a0 con 0x7f25940c2c80 2024-05-09T05:06:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.293+0000 7f2598c43640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2578076320 0x7f2578078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.293+0000 7f2598c43640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2578076320 0x7f2578078830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2590006fd0 tx=0x7f2590006bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.294+0000 7f7febfff640 1 -- 172.21.15.1:0/172970822 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7ff8045070 con 0x7f8004161cf0 2024-05-09T05:06:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.294+0000 7f7febfff640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7ff00764d0 0x7f7ff00789e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.294+0000 7ff2d17fa640 1 -- 172.21.15.1:0/2225763946 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff2c801d070 con 0x7ff2c4005350 2024-05-09T05:06:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.295+0000 7f7febfff640 1 -- 172.21.15.1:0/172970822 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f7ff80b7990 con 0x7f8004161cf0 2024-05-09T05:06:13.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.295+0000 7f8002ffd640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7ff00764d0 0x7f7ff00789e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.295+0000 7ff2d17fa640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff2b80763d0 0x7ff2b80788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.295+0000 7f8002ffd640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7ff00764d0 0x7f7ff00789e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8004162100 tx=0x7f7ff4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.295+0000 7ff2d17fa640 1 -- 172.21.15.1:0/2225763946 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff2c80972e0 con 0x7ff2c4005350 2024-05-09T05:06:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.295+0000 7ff2d3fff640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff2b80763d0 0x7ff2b80788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.296+0000 7ff2d3fff640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff2b80763d0 0x7ff2b80788e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0fd720 tx=0x7ff2c00090e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.298+0000 7f7febfff640 1 -- 172.21.15.1:0/172970822 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7ff8080c90 con 0x7f8004161cf0 2024-05-09T05:06:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.300+0000 7f2589ffb640 1 -- 172.21.15.1:0/1930106763 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2584061470 con 0x7f25940c2c80 2024-05-09T05:06:13.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.305+0000 7ff2d17fa640 1 -- 172.21.15.1:0/2225763946 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff2c8024080 con 0x7ff2c4005350 2024-05-09T05:06:13.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.311+0000 7f8a57fff640 1 -- 172.21.15.1:0/572273563 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8a6804b070 con 0x7f8a7015ae30 2024-05-09T05:06:13.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.723+0000 7f8a05f66640 1 -- 172.21.15.1:0/984681529 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f8a000c9e90 con 0x7f8a001612b0 2024-05-09T05:06:13.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.724+0000 7f89f7fff640 1 -- 172.21.15.1:0/984681529 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f89e8068380 con 0x7f8a001612b0 2024-05-09T05:06:13.725 INFO:teuthology.orchestra.run.smithi001.stdout:64424509445 2024-05-09T05:06:13.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.728+0000 7f89f5ffb640 1 -- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89e00763a0 msgr2=0x7f89e00788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.728+0000 7f89f5ffb640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89e00763a0 0x7f89e00788b0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89f0034000 tx=0x7f89f0009000 comp rx=0 tx=0).stop 2024-05-09T05:06:13.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.728+0000 7f89f5ffb640 1 -- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a001612b0 msgr2=0x7f8a0015b920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.728+0000 7f89f5ffb640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a001612b0 0x7f8a0015b920 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f89e800c3c0 tx=0x7f89e800c880 comp rx=0 tx=0).stop 2024-05-09T05:06:13.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.728+0000 7f89f5ffb640 1 -- 172.21.15.1:0/984681529 shutdown_connections 2024-05-09T05:06:13.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.728+0000 7f89f5ffb640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f89e00763a0 0x7f89e00788b0 secure :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f89f0034000 tx=0x7f89f0009000 comp rx=0 tx=0).stop 2024-05-09T05:06:13.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.729+0000 7f89f5ffb640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a001612b0 0x7f8a0015b920 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.729+0000 7f89f5ffb640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a0015a6b0 0x7f8a000cc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.729+0000 7f89f5ffb640 1 --2- 172.21.15.1:0/984681529 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a000c4d70 0x7f8a0014e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.729+0000 7f89f5ffb640 1 -- 172.21.15.1:0/984681529 >> 172.21.15.1:0/984681529 conn(0x7f8a00070fa0 msgr2=0x7f8a0013ed80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.729+0000 7f89f5ffb640 1 -- 172.21.15.1:0/984681529 shutdown_connections 2024-05-09T05:06:13.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.729+0000 7f89f5ffb640 1 -- 172.21.15.1:0/984681529 wait complete. 2024-05-09T05:06:13.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.732+0000 7f1844ff9640 1 -- 172.21.15.1:0/3600545105 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f182c005580 con 0x7f1848045350 2024-05-09T05:06:13.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.733+0000 7f11b9c9f640 1 -- 172.21.15.1:0/3283414143 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f11ac04c750 con 0x7f11ac00e4b0 2024-05-09T05:06:13.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.733+0000 7f1846ffd640 1 -- 172.21.15.1:0/3600545105 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f18500ccf70 con 0x7f1848045350 2024-05-09T05:06:13.733 INFO:teuthology.orchestra.run.smithi001.stdout:60129542150 2024-05-09T05:06:13.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.733+0000 7f119ffff640 1 -- 172.21.15.1:0/3283414143 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f11b40fe160 con 0x7f11ac00e4b0 2024-05-09T05:06:13.734 INFO:teuthology.orchestra.run.smithi001.stdout:60129542150 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.736+0000 7f1857534640 1 -- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18340763a0 msgr2=0x7f18340788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.736+0000 7f1857534640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18340763a0 0x7f18340788b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f184c044000 tx=0x7f184c008030 comp rx=0 tx=0).stop 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.736+0000 7f1857534640 1 -- 172.21.15.1:0/3600545105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 msgr2=0x7f18480fd0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.736+0000 7f1857534640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480fd0c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f185006bc20 tx=0x7f1850069e50 comp rx=0 tx=0).stop 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f1857534640 1 -- 172.21.15.1:0/3600545105 shutdown_connections 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f1857534640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18340763a0 0x7f18340788b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f1857534640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1848045350 0x7f18480fd0c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f1857534640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1848036070 0x7f18480fc790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f1857534640 1 --2- 172.21.15.1:0/3600545105 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18480349e0 0x7f18480f81b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f1857534640 1 -- 172.21.15.1:0/3600545105 >> 172.21.15.1:0/3600545105 conn(0x7f18480503b0 msgr2=0x7f18480f62a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f1857534640 1 -- 172.21.15.1:0/3600545105 shutdown_connections 2024-05-09T05:06:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f1857534640 1 -- 172.21.15.1:0/3600545105 wait complete. 2024-05-09T05:06:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f119dffb640 1 -- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11980763a0 msgr2=0x7f11980788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f119dffb640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11980763a0 0x7f11980788b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f11ac007b60 tx=0x7f11a8008040 comp rx=0 tx=0).stop 2024-05-09T05:06:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f119dffb640 1 -- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac00e4b0 msgr2=0x7f11ac048ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f119dffb640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac00e4b0 0x7f11ac048ec0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f11b402e1c0 tx=0x7f11b4077ee0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f119dffb640 1 -- 172.21.15.1:0/3283414143 shutdown_connections 2024-05-09T05:06:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.737+0000 7f119dffb640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11980763a0 0x7f11980788b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.738+0000 7f119dffb640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11ac04c970 0x7f11ac047d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.738+0000 7f119dffb640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11ac0471e0 0x7f11ac0d6100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.738+0000 7f119dffb640 1 --2- 172.21.15.1:0/3283414143 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11ac00e4b0 0x7f11ac048ec0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.738+0000 7f119dffb640 1 -- 172.21.15.1:0/3283414143 >> 172.21.15.1:0/3283414143 conn(0x7f11ac0503e0 msgr2=0x7f11ac045e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.738+0000 7f119dffb640 1 -- 172.21.15.1:0/3283414143 shutdown_connections 2024-05-09T05:06:13.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.738+0000 7f119dffb640 1 -- 172.21.15.1:0/3283414143 wait complete. 2024-05-09T05:06:13.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.746+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb20c005850 con 0x7fb22c2138f0 2024-05-09T05:06:13.747 INFO:teuthology.orchestra.run.smithi001.stdout:60129542149 2024-05-09T05:06:13.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.747+0000 7fb2297fa640 1 -- 172.21.15.1:0/902583579 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb21c060610 con 0x7fb22c2138f0 2024-05-09T05:06:13.750 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.750+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f25940b3b40 con 0x7f25940c2c80 2024-05-09T05:06:13.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.750+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f8004050870 con 0x7f8004161cf0 2024-05-09T05:06:13.752 INFO:teuthology.orchestra.run.smithi001.stdout:60129542150 2024-05-09T05:06:13.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.751+0000 7f2589ffb640 1 -- 172.21.15.1:0/1930106763 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f2584060d70 con 0x7f25940c2c80 2024-05-09T05:06:13.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.751+0000 7f7febfff640 1 -- 172.21.15.1:0/172970822 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f7ff8084f90 con 0x7f8004161cf0 2024-05-09T05:06:13.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.751+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb218076320 msgr2=0x7fb218078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.751+0000 7fb232b4d640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb218076320 0x7fb218078830 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb22403b000 tx=0x7fb224007000 comp rx=0 tx=0).stop 2024-05-09T05:06:13.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.751+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb22c2138f0 msgr2=0x7fb22c15c100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.751+0000 7fb232b4d640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb22c2138f0 0x7fb22c15c100 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb21c007670 tx=0x7fb21c005bc0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.754 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509445 for osd.5 2024-05-09T05:06:13.754 INFO:teuthology.orchestra.run.smithi001.stdout:64424509444 2024-05-09T05:06:13.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7ff2beffd640 1 -- 172.21.15.1:0/2225763946 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7ff2cc0fa910 con 0x7ff2c4005350 2024-05-09T05:06:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 shutdown_connections 2024-05-09T05:06:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7fb232b4d640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb218076320 0x7fb218078830 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7fb232b4d640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb22c2138f0 0x7fb22c15c100 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7fb232b4d640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb22c202a70 0x7fb22c202e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7fb232b4d640 1 --2- 172.21.15.1:0/902583579 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb214008160 0x7fb22c162810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 >> 172.21.15.1:0/902583579 conn(0x7fb22c070fa0 msgr2=0x7fb22c0c88d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 shutdown_connections 2024-05-09T05:06:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7fb232b4d640 1 -- 172.21.15.1:0/902583579 wait complete. 2024-05-09T05:06:13.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.752+0000 7ff2d17fa640 1 -- 172.21.15.1:0/2225763946 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7ff2c80605d0 con 0x7ff2c4005350 2024-05-09T05:06:13.756 INFO:teuthology.orchestra.run.smithi001.stdout:60129542149 2024-05-09T05:06:13.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2578076320 msgr2=0x7f2578078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2578076320 0x7f2578078830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2590006fd0 tx=0x7f2590006bc0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c2c80 msgr2=0x7f2594202b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c2c80 0x7f2594202b40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f258400c7f0 tx=0x7f258400ccb0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 shutdown_connections 2024-05-09T05:06:13.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2578076320 0x7f2578078830 secure :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2590006fd0 tx=0x7f2590006bc0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594155cb0 0x7f25940cd230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25941587d0 0x7f2594159930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 --2- 172.21.15.1:0/1930106763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c2c80 0x7f2594202b40 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f258400c7f0 tx=0x7f258400ccb0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.754+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 >> 172.21.15.1:0/1930106763 conn(0x7f25940cdc60 msgr2=0x7f25941572b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 shutdown_connections 2024-05-09T05:06:13.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f259a446640 1 -- 172.21.15.1:0/1930106763 wait complete. 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7ff00764d0 msgr2=0x7f7ff00789e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7ff00764d0 0x7f7ff00789e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f8004162100 tx=0x7f7ff4008040 comp rx=0 tx=0).stop 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8004161cf0 msgr2=0x7f80040ce790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8004161cf0 0x7f80040ce790 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f7ff8000c00 tx=0x7f7ff8004730 comp rx=0 tx=0).stop 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 shutdown_connections 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7ff00764d0 0x7f7ff00789e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8004163e20 0x7f80041627c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f800415e030 0x7f80041638e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 --2- 172.21.15.1:0/172970822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8004161cf0 0x7f80040ce790 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.755+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 >> 172.21.15.1:0/172970822 conn(0x7f800406dca0 msgr2=0x7f80040b67f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.756+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 shutdown_connections 2024-05-09T05:06:13.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.756+0000 7f8009f00640 1 -- 172.21.15.1:0/172970822 wait complete. 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.756+0000 7ff2beffd640 1 -- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff2b80763d0 msgr2=0x7ff2b80788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.756+0000 7ff2beffd640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff2b80763d0 0x7ff2b80788e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7ff2cc0fd720 tx=0x7ff2c00090e0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.756+0000 7ff2beffd640 1 -- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4005350 msgr2=0x7ff2cc0f7530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.756+0000 7ff2beffd640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4005350 0x7ff2cc0f7530 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7ff2c800f930 tx=0x7ff2c800fdf0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.757+0000 7ff2beffd640 1 -- 172.21.15.1:0/2225763946 shutdown_connections 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.757+0000 7ff2beffd640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff2b80763d0 0x7ff2b80788e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.757+0000 7ff2beffd640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff2cc0f81a0 0x7ff2cc0f8ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.757+0000 7ff2beffd640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff2cc10ea10 0x7ff2cc0e7df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.757+0000 7ff2beffd640 1 --2- 172.21.15.1:0/2225763946 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff2c4005350 0x7ff2cc0f7530 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.757+0000 7ff2beffd640 1 -- 172.21.15.1:0/2225763946 >> 172.21.15.1:0/2225763946 conn(0x7ff2cc0503a0 msgr2=0x7ff2cc0f64a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.757+0000 7ff2beffd640 1 -- 172.21.15.1:0/2225763946 shutdown_connections 2024-05-09T05:06:13.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.757+0000 7ff2beffd640 1 -- 172.21.15.1:0/2225763946 wait complete. 2024-05-09T05:06:13.762 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.3 2024-05-09T05:06:13.763 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.4 2024-05-09T05:06:13.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.761+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f8a50005580 con 0x7f8a7015ae30 2024-05-09T05:06:13.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.761+0000 7f8a57fff640 1 -- 172.21.15.1:0/572273563 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8a70203560 con 0x7f8a7015ae30 2024-05-09T05:06:13.764 INFO:teuthology.orchestra.run.smithi001.stdout:60129542151 2024-05-09T05:06:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.765+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a5c0762d0 msgr2=0x7f8a5c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.765+0000 7f8a75914640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a5c0762d0 0x7f8a5c0787e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f8a60009d80 tx=0x7f8a60009110 comp rx=0 tx=0).stop 2024-05-09T05:06:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.765+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7015ae30 msgr2=0x7f8a70157c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.765+0000 7f8a75914640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7015ae30 0x7f8a70157c80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f8a700cb290 tx=0x7f8a68004a40 comp rx=0 tx=0).stop 2024-05-09T05:06:13.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.766+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 shutdown_connections 2024-05-09T05:06:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.766+0000 7f8a75914640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a5c0762d0 0x7f8a5c0787e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.766+0000 7f8a75914640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a700ca3c0 0x7f8a70203020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.766+0000 7f8a75914640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a7015ae30 0x7f8a70157c80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.766+0000 7f8a75914640 1 --2- 172.21.15.1:0/572273563 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a700740c0 0x7f8a701553b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.766+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 >> 172.21.15.1:0/572273563 conn(0x7f8a70070f40 msgr2=0x7f8a700cbd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.766+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 shutdown_connections 2024-05-09T05:06:13.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:13.766+0000 7f8a75914640 1 -- 172.21.15.1:0/572273563 wait complete. 2024-05-09T05:06:13.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.6 2024-05-09T05:06:13.772 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509444 for osd.7 2024-05-09T05:06:13.773 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542149 for osd.0 2024-05-09T05:06:13.776 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542150 for osd.1 2024-05-09T05:06:13.781 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542151 for osd.2 2024-05-09T05:06:14.755 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2024-05-09T05:06:14.763 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2024-05-09T05:06:14.764 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2024-05-09T05:06:14.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2024-05-09T05:06:14.772 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2024-05-09T05:06:14.774 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2024-05-09T05:06:14.776 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2024-05-09T05:06:14.781 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2024-05-09T05:06:14.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.956+0000 7efc668dd640 1 Processor -- start 2024-05-09T05:06:14.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.957+0000 7efc668dd640 1 -- start start 2024-05-09T05:06:14.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.960+0000 7efc668dd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc5804c560 0x7efc580e78a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:14.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.960+0000 7efc668dd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efc5804e870 0x7efc5800c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:14.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.960+0000 7efc668dd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efc5800cd60 0x7efc5804d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:14.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.960+0000 7efc668dd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7efc58037a50 con 0x7efc5804e870 2024-05-09T05:06:14.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.960+0000 7efc668dd640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7efc580e7de0 con 0x7efc5800cd60 2024-05-09T05:06:14.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.960+0000 7efc668dd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efc58035670 con 0x7efc5804c560 2024-05-09T05:06:14.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.961+0000 7efc660dc640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efc5800cd60 0x7efc5804d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:14.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.962+0000 7efc660dc640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efc5800cd60 0x7efc5804d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44086/0 (socket says 172.21.15.1:44086) 2024-05-09T05:06:14.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.962+0000 7efc660dc640 1 -- 172.21.15.1:0/2620094103 learned_addr learned my addr 172.21.15.1:0/2620094103 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:14.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.962+0000 7efc660dc640 1 -- 172.21.15.1:0/2620094103 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc5804c560 msgr2=0x7efc580e78a0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:06:14.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.963+0000 7efc660dc640 1 --2- 172.21.15.1:0/2620094103 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc5804c560 0x7efc580e78a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:14.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.963+0000 7efc660dc640 1 -- 172.21.15.1:0/2620094103 >> v2:172.21.15.1:3300/0 conn(0x7efc5804e870 msgr2=0x7efc5800c820 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:06:14.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.963+0000 7efc650da640 1 --2- 172.21.15.1:0/2620094103 >> v2:172.21.15.1:3300/0 conn(0x7efc5804e870 0x7efc5800c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:14.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.963+0000 7efc660dc640 1 --2- 172.21.15.1:0/2620094103 >> v2:172.21.15.1:3300/0 conn(0x7efc5804e870 0x7efc5800c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:14.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.963+0000 7efc660dc640 1 -- 172.21.15.1:0/2620094103 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc58037490 con 0x7efc5800cd60 2024-05-09T05:06:14.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.965+0000 7efc650da640 1 --2- 172.21.15.1:0/2620094103 >> v2:172.21.15.1:3300/0 conn(0x7efc5804e870 0x7efc5800c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:14.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.965+0000 7efc660dc640 1 --2- 172.21.15.1:0/2620094103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5800cd60 0x7efc5804d180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efc60065ff0 tx=0x7efc600a78b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6fde7df1b0c4b74d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:14.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.965+0000 7efc648d9640 1 -- 172.21.15.1:0/2620094103 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600ae440 con 0x7efc5800cd60 2024-05-09T05:06:14.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.965+0000 7efc648d9640 1 -- 172.21.15.1:0/2620094103 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc600ae5e0 con 0x7efc5800cd60 2024-05-09T05:06:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.967+0000 7efc668dd640 1 -- 172.21.15.1:0/2620094103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5800cd60 msgr2=0x7efc5804d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.967+0000 7efc668dd640 1 --2- 172.21.15.1:0/2620094103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5800cd60 0x7efc5804d180 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7efc60065ff0 tx=0x7efc600a78b0 comp rx=0 tx=0).stop 2024-05-09T05:06:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.967+0000 7efc668dd640 1 -- 172.21.15.1:0/2620094103 shutdown_connections 2024-05-09T05:06:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.967+0000 7efc668dd640 1 --2- 172.21.15.1:0/2620094103 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5800cd60 0x7efc5804d180 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.967+0000 7efc668dd640 1 --2- 172.21.15.1:0/2620094103 >> v2:172.21.15.1:3300/0 conn(0x7efc5804e870 0x7efc5800c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.967+0000 7efc668dd640 1 --2- 172.21.15.1:0/2620094103 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc5804c560 0x7efc580e78a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.967+0000 7efc668dd640 1 -- 172.21.15.1:0/2620094103 >> 172.21.15.1:0/2620094103 conn(0x7efc580527a0 msgr2=0x7efc58052bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:14.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.967+0000 7efc668dd640 1 -- 172.21.15.1:0/2620094103 shutdown_connections 2024-05-09T05:06:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.968+0000 7efc668dd640 1 -- 172.21.15.1:0/2620094103 wait complete. 2024-05-09T05:06:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.971+0000 7efc668dd640 1 Processor -- start 2024-05-09T05:06:14.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.971+0000 7efc668dd640 1 -- start start 2024-05-09T05:06:14.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.973+0000 7efc668dd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc5804c560 0x7efc581847b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:14.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.974+0000 7efc668dd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5804e870 0x7efc581855f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:14.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.974+0000 7efc668dd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc580d0720 0x7efc580e3710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:14.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.974+0000 7efc650da640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5804e870 0x7efc581855f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:14.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.974+0000 7efc650da640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5804e870 0x7efc581855f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44098/0 (socket says 172.21.15.1:44098) 2024-05-09T05:06:14.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.974+0000 7efc660dc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc580d0720 0x7efc580e3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:14.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.974+0000 7efc660dc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc580d0720 0x7efc580e3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49414/0 (socket says 172.21.15.1:49414) 2024-05-09T05:06:14.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.975+0000 7efc650da640 1 -- 172.21.15.1:0/1313842553 learned_addr learned my addr 172.21.15.1:0/1313842553 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:14.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.975+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efc58035bf0 con 0x7efc580d0720 2024-05-09T05:06:14.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.975+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efc58035910 con 0x7efc5804e870 2024-05-09T05:06:14.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.975+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efc5804ec80 con 0x7efc5804c560 2024-05-09T05:06:14.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.976+0000 7efc660dc640 1 -- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc5804c560 msgr2=0x7efc581847b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:06:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.976+0000 7efc660dc640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc5804c560 0x7efc581847b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.976+0000 7efc660dc640 1 -- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5804e870 msgr2=0x7efc581855f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.976+0000 7efc660dc640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5804e870 0x7efc581855f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.976+0000 7efc660dc640 1 -- 172.21.15.1:0/1313842553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc58184cf0 con 0x7efc580d0720 2024-05-09T05:06:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.976+0000 7efc660dc640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc580d0720 0x7efc580e3710 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc6006b7d0 tx=0x7efc6006a820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.977+0000 7efc527fc640 1 -- 172.21.15.1:0/1313842553 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600adc50 con 0x7efc580d0720 2024-05-09T05:06:14.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.977+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc58185b30 con 0x7efc580d0720 2024-05-09T05:06:14.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.977+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc580e4660 con 0x7efc580d0720 2024-05-09T05:06:14.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.982+0000 7efc527fc640 1 -- 172.21.15.1:0/1313842553 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efc600addf0 con 0x7efc580d0720 2024-05-09T05:06:14.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.982+0000 7efc527fc640 1 -- 172.21.15.1:0/1313842553 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc600b8640 con 0x7efc580d0720 2024-05-09T05:06:14.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.983+0000 7efc527fc640 1 -- 172.21.15.1:0/1313842553 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efc600b6040 con 0x7efc580d0720 2024-05-09T05:06:14.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.984+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efc48005290 con 0x7efc580d0720 2024-05-09T05:06:14.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.995+0000 7efc527fc640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc44076400 0x7efc44078910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.995+0000 7efc527fc640 1 -- 172.21.15.1:0/1313842553 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7efc60134fe0 con 0x7efc580d0720 2024-05-09T05:06:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:14.996+0000 7efc527fc640 1 -- 172.21.15.1:0/1313842553 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efc600c7030 con 0x7efc580d0720 2024-05-09T05:06:15.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.003+0000 7efc658db640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc44076400 0x7efc44078910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.018+0000 7efc658db640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc44076400 0x7efc44078910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efc580369a0 tx=0x7efc54006b30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.021+0000 7f70c89d9640 1 Processor -- start 2024-05-09T05:06:15.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.022+0000 7f70c89d9640 1 -- start start 2024-05-09T05:06:15.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c89d9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f70c4073bd0 0x7f70c40cddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c89d9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c416d0f0 0x7f70c416d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c89d9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70c40c3460 0x7f70c40c3870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c89d9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f70c40c2fd0 con 0x7f70c40c3460 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c89d9640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f70c4074d40 con 0x7f70c4073bd0 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c89d9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f70c40747c0 con 0x7f70c416d0f0 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70c40c3460 0x7f70c40c3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f70c40c3460 0x7f70c40c3870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49424/0 (socket says 172.21.15.1:49424) 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.024+0000 7f70c3fff640 1 -- 172.21.15.1:0/4163223604 learned_addr learned my addr 172.21.15.1:0/4163223604 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.025+0000 7f70c2ffd640 1 --2- 172.21.15.1:0/4163223604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c416d0f0 0x7f70c416d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.025+0000 7f70c3fff640 1 -- 172.21.15.1:0/4163223604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c416d0f0 msgr2=0x7f70c416d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.025+0000 7f70c3fff640 1 --2- 172.21.15.1:0/4163223604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c416d0f0 0x7f70c416d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.025+0000 7f70c3fff640 1 -- 172.21.15.1:0/4163223604 >> v2:172.21.15.196:3300/0 conn(0x7f70c4073bd0 msgr2=0x7f70c40cddd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.025+0000 7f70c3fff640 1 --2- 172.21.15.1:0/4163223604 >> v2:172.21.15.196:3300/0 conn(0x7f70c4073bd0 0x7f70c40cddd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.025+0000 7f70c3fff640 1 -- 172.21.15.1:0/4163223604 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c4074a60 con 0x7f70c40c3460 2024-05-09T05:06:15.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.026+0000 7f70c3fff640 1 --2- 172.21.15.1:0/4163223604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c40c3460 0x7f70c40c3870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f70c40c4ba0 tx=0x7f70bc02c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb52483bc0c36946 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.025+0000 7fbd8d72b640 1 Processor -- start 2024-05-09T05:06:15.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.025+0000 7fbd8d72b640 1 -- start start 2024-05-09T05:06:15.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.027+0000 7fb45bb86640 1 Processor -- start 2024-05-09T05:06:15.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.027+0000 7f70c27fc640 1 -- 172.21.15.1:0/4163223604 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc003990 con 0x7f70c40c3460 2024-05-09T05:06:15.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.027+0000 7f8dccd9f640 1 Processor -- start 2024-05-09T05:06:15.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fbd8d72b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd8004da40 0x7fbd800d6330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7f8dccd9f640 1 -- start start 2024-05-09T05:06:15.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fbd8d72b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd800480b0 0x7fbd800dfc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fbd87fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd8004da40 0x7fbd800d6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fbd8d72b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd800464f0 con 0x7fbd80047ca0 2024-05-09T05:06:15.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fbd87fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd8004da40 0x7fbd800d6330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49434/0 (socket says 172.21.15.1:49434) 2024-05-09T05:06:15.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fbd87fff640 1 -- 172.21.15.1:0/3437212832 learned_addr learned my addr 172.21.15.1:0/3437212832 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fbd8d72b640 1 -- 172.21.15.1:0/3437212832 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbd800467b0 con 0x7fbd8004da40 2024-05-09T05:06:15.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fbd8d72b640 1 -- 172.21.15.1:0/3437212832 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8004cea0 con 0x7fbd800480b0 2024-05-09T05:06:15.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.029+0000 7fbd8cf2a640 1 --2- 172.21.15.1:0/3437212832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd800480b0 0x7fbd800dfc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.029+0000 7f70c27fc640 1 -- 172.21.15.1:0/4163223604 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70bc003b30 con 0x7f70c40c3460 2024-05-09T05:06:15.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.029+0000 7fbd87fff640 1 -- 172.21.15.1:0/3437212832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd800480b0 msgr2=0x7fbd800dfc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.029+0000 7fbd87fff640 1 --2- 172.21.15.1:0/3437212832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd800480b0 0x7fbd800dfc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.029+0000 7fbd87fff640 1 -- 172.21.15.1:0/3437212832 >> v1:172.21.15.1:6789/0 conn(0x7fbd80047ca0 legacy=0x7fbd800df570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.029+0000 7fbd87fff640 1 -- 172.21.15.1:0/3437212832 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd8004d180 con 0x7fbd8004da40 2024-05-09T05:06:15.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f70c27fc640 1 -- 172.21.15.1:0/4163223604 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70bc004030 con 0x7f70c40c3460 2024-05-09T05:06:15.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f8dccd9f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc0046870 0x7f8dc0046270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7fbd8cf2a640 1 --2- 172.21.15.1:0/3437212832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd800480b0 0x7fbd800dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:15.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.029+0000 7f326e138640 1 Processor -- start 2024-05-09T05:06:15.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.029+0000 7f326e138640 1 -- start start 2024-05-09T05:06:15.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f8dccd9f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc0006930 con 0x7f8dc00dda30 2024-05-09T05:06:15.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f8dccd9f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8dc00dde40 con 0x7f8dc0006520 2024-05-09T05:06:15.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f8dccd9f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc00e3ef0 con 0x7f8dc0046870 2024-05-09T05:06:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.028+0000 7fb45bb86640 1 -- start start 2024-05-09T05:06:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7fb45bb86640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4540740c0 0x7fb45415ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7fb45bb86640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb454164f00 con 0x7fb454074610 2024-05-09T05:06:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7fb45bb86640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4540c97d0 con 0x7fb454074a20 2024-05-09T05:06:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7fb45bb86640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb4540c4e00 con 0x7fb4540740c0 2024-05-09T05:06:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7fb459922640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4540740c0 0x7fb45415ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7fb459922640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb4540740c0 0x7fb45415ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49438/0 (socket says 172.21.15.1:49438) 2024-05-09T05:06:15.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f8dc7fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8dc00dda30 0x7f8dc004bb30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35608/0 (socket says 172.21.15.1:35608) 2024-05-09T05:06:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7fb459922640 1 -- 172.21.15.1:0/4234666407 learned_addr learned my addr 172.21.15.1:0/4234666407 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f70c89d9640 1 -- 172.21.15.1:0/4163223604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c40c3460 msgr2=0x7f70c40c3870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f70c89d9640 1 --2- 172.21.15.1:0/4163223604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c40c3460 0x7f70c40c3870 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f70c40c4ba0 tx=0x7f70bc02c890 comp rx=0 tx=0).stop 2024-05-09T05:06:15.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7fbd87fff640 1 --2- 172.21.15.1:0/3437212832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd800d6330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbd88065ff0 tx=0x7fbd880a9a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a85cbda220026b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f8dc7fff640 1 -- 172.21.15.1:0/3423480448 learned_addr learned my addr 172.21.15.1:0/3423480448 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:15.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f8dc6ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f8dc0006520 0x7f8dc00e36d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:57678/0 (socket says 172.21.15.1:57678) 2024-05-09T05:06:15.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.030+0000 7f8dc77fe640 1 --2- 172.21.15.1:0/3423480448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc0046870 0x7f8dc0046270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f8dc67fc640 1 -- 172.21.15.1:0/3423480448 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1589623803 0 0) 0x7f8dc00dde40 con 0x7f8dc0006520 2024-05-09T05:06:15.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f8dc67fc640 1 -- 172.21.15.1:0/3423480448 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8db8003520 con 0x7f8dc0006520 2024-05-09T05:06:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f70c89d9640 1 -- 172.21.15.1:0/4163223604 shutdown_connections 2024-05-09T05:06:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f8dc77fe640 1 -- 172.21.15.1:0/3423480448 >> v1:172.21.15.196:6789/0 conn(0x7f8dc0006520 legacy=0x7f8dc00e36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f70c89d9640 1 --2- 172.21.15.1:0/4163223604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c40c3460 0x7f70c40c3870 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f70c89d9640 1 --2- 172.21.15.1:0/4163223604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c416d0f0 0x7f70c416d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f70c89d9640 1 --2- 172.21.15.1:0/4163223604 >> v2:172.21.15.196:3300/0 conn(0x7f70c4073bd0 0x7f70c40cddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f70c89d9640 1 -- 172.21.15.1:0/4163223604 >> 172.21.15.1:0/4163223604 conn(0x7f70c406cb10 msgr2=0x7f70c406cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f8dc67fc640 1 -- 172.21.15.1:0/3423480448 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 105379934 0 0) 0x7f8db8003520 con 0x7f8dc0006520 2024-05-09T05:06:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7fb459922640 1 -- 172.21.15.1:0/4234666407 >> v1:172.21.15.196:6789/0 conn(0x7fb454074a20 legacy=0x7fb4540ca590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7fb459922640 1 -- 172.21.15.1:0/4234666407 >> v1:172.21.15.1:6789/0 conn(0x7fb454074610 legacy=0x7fb4540c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7fbd86ffd640 1 -- 172.21.15.1:0/3437212832 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd880b2070 con 0x7fbd8004da40 2024-05-09T05:06:15.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7fb459922640 1 -- 172.21.15.1:0/4234666407 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4540c9ab0 con 0x7fb4540740c0 2024-05-09T05:06:15.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7fe9b68f9640 1 Processor -- start 2024-05-09T05:06:15.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7f326e138640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3268152af0 0x7f32681592a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7fbd86ffd640 1 -- 172.21.15.1:0/3437212832 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd880ae920 con 0x7fbd8004da40 2024-05-09T05:06:15.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f8dc77fe640 1 -- 172.21.15.1:0/3423480448 >> v1:172.21.15.1:6789/0 conn(0x7f8dc00dda30 legacy=0x7f8dc004bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.031+0000 7f8dc77fe640 1 -- 172.21.15.1:0/3423480448 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc00e41d0 con 0x7f8dc0046870 2024-05-09T05:06:15.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7fbd86ffd640 1 -- 172.21.15.1:0/3437212832 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd880aed30 con 0x7fbd8004da40 2024-05-09T05:06:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7f32677fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3268069a80 0x7f32680c3ed0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:57702/0 (socket says 172.21.15.1:57702) 2024-05-09T05:06:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7f32677fe640 1 -- 172.21.15.1:0/2179041725 learned_addr learned my addr 172.21.15.1:0/2179041725 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7f326e138640 1 -- 172.21.15.1:0/2179041725 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3268152f00 con 0x7f326806c6f0 2024-05-09T05:06:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fe9b68f9640 1 -- start start 2024-05-09T05:06:15.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7f70c89d9640 1 -- 172.21.15.1:0/4163223604 shutdown_connections 2024-05-09T05:06:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f8dc77fe640 1 --2- 172.21.15.1:0/3423480448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc0046870 0x7f8dc0046270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009740 tx=0x7f8db402a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bbfc6c920288e638 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fb943fff640 1 Processor -- start 2024-05-09T05:06:15.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fb943fff640 1 -- start start 2024-05-09T05:06:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f8dc67fc640 1 -- 172.21.15.1:0/3423480448 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4037070 con 0x7f8dc0046870 2024-05-09T05:06:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f8dc67fc640 1 -- 172.21.15.1:0/3423480448 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db402acd0 con 0x7f8dc0046870 2024-05-09T05:06:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f8dc67fc640 1 -- 172.21.15.1:0/3423480448 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8dbc0016f0 con 0x7f8dc0046870 2024-05-09T05:06:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.032+0000 7f3266ffd640 1 --2- 172.21.15.1:0/2179041725 >> v2:172.21.15.1:3300/0 conn(0x7f3268152af0 0x7f32681592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f326e138640 1 -- 172.21.15.1:0/2179041725 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3268073a00 con 0x7f3268069a80 2024-05-09T05:06:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f326e138640 1 -- 172.21.15.1:0/2179041725 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f326806f010 con 0x7f3268152af0 2024-05-09T05:06:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 911251821 0 0) 0x7f3268073a00 con 0x7f3268069a80 2024-05-09T05:06:15.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3250007930 con 0x7f3268069a80 2024-05-09T05:06:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fbd8d72b640 1 -- 172.21.15.1:0/3437212832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 msgr2=0x7fbd800d6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/3437212832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd800d6330 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbd88065ff0 tx=0x7fbd880a9a00 comp rx=0 tx=0).stop 2024-05-09T05:06:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fbd8d72b640 1 -- 172.21.15.1:0/3437212832 shutdown_connections 2024-05-09T05:06:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/3437212832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd800480b0 0x7fbd800dfc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/3437212832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd800d6330 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fbd8d72b640 1 -- 172.21.15.1:0/3437212832 >> 172.21.15.1:0/3437212832 conn(0x7fbd800514b0 msgr2=0x7fbd800518c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 507826555 0 0) 0x7f3250007930 con 0x7f3268069a80 2024-05-09T05:06:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 >> v2:172.21.15.1:3300/0 conn(0x7f3268152af0 msgr2=0x7f32681592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7fbd8d72b640 1 -- 172.21.15.1:0/3437212832 shutdown_connections 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 --2- 172.21.15.1:0/2179041725 >> v2:172.21.15.1:3300/0 conn(0x7f3268152af0 0x7f32681592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 >> v1:172.21.15.1:6789/0 conn(0x7f326806c6f0 legacy=0x7f32681642b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326806f260 con 0x7f3268069a80 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f32580016f0 con 0x7f3268069a80 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 >> v1:172.21.15.196:6789/0 conn(0x7f3268069a80 legacy=0x7f32680c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3250007e30 0x7f32500082c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7f32667fc640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f3250009020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f32667fc640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3250009560 0x7f32500145c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f326806f010 con 0x7f3250009560 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3250014b00 con 0x7f3250007e30 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3250014c80 con 0x7f3250008c10 2024-05-09T05:06:15.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3266ffd640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f3250009020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3267fff640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3250009560 0x7f32500145c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3266ffd640 1 -- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3250007e30 msgr2=0x7f32500082c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3266ffd640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3250007e30 0x7f32500082c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3266ffd640 1 -- 172.21.15.1:0/2179041725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3250009560 msgr2=0x7f32500145c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3266ffd640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3250009560 0x7f32500145c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3266ffd640 1 -- 172.21.15.1:0/2179041725 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326806f260 con 0x7f3250008c10 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3267fff640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3250009560 0x7f32500145c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f3267fff640 1 -- 172.21.15.1:0/2179041725 reap_dead start 2024-05-09T05:06:15.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7fbd8d72b640 1 -- 172.21.15.1:0/3437212832 wait complete. 2024-05-09T05:06:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f8dccd9f640 1 -- 172.21.15.1:0/3423480448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc0046870 msgr2=0x7f8dc0046270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f8dccd9f640 1 --2- 172.21.15.1:0/3423480448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc0046870 0x7f8dc0046270 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f8db4009740 tx=0x7f8db402a1c0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f8dccd9f640 1 -- 172.21.15.1:0/3423480448 shutdown_connections 2024-05-09T05:06:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f8dccd9f640 1 --2- 172.21.15.1:0/3423480448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc0046870 0x7f8dc0046270 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f8dccd9f640 1 -- 172.21.15.1:0/3423480448 >> 172.21.15.1:0/3423480448 conn(0x7f8dc00507f0 msgr2=0x7f8dc0050c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f8dccd9f640 1 -- 172.21.15.1:0/3423480448 shutdown_connections 2024-05-09T05:06:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7f8dccd9f640 1 -- 172.21.15.1:0/3423480448 wait complete. 2024-05-09T05:06:15.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7f3266ffd640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f3250009020 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f325c009aa0 tx=0x7f3250009c40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e158d8761a76ad65 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7fb943fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb93c0463a0 0x7fb93c051830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.033+0000 7fb459922640 1 --2- 172.21.15.1:0/4234666407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540740c0 0x7fb45415ae30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb450009870 tx=0x7fb45002a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8cdb5e8afb2fea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.034+0000 7fb458920640 1 -- 172.21.15.1:0/4234666407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb450037070 con 0x7fb4540740c0 2024-05-09T05:06:15.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7fb458920640 1 -- 172.21.15.1:0/4234666407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb450032640 con 0x7fb4540740c0 2024-05-09T05:06:15.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7fb458920640 1 -- 172.21.15.1:0/4234666407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb450032a10 con 0x7fb4540740c0 2024-05-09T05:06:15.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7fb943fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c034050 0x7fb93c051d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7fe9b68f9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b01592a0 0x7fe9b006c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7f70c89d9640 1 -- 172.21.15.1:0/4163223604 wait complete. 2024-05-09T05:06:15.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7fb943fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb93c0dfb70 0x7fb93c0dff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fe9b68f9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9b006cc30 0x7fe9b01631a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b01592a0 0x7fe9b006c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fe9af7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b01592a0 0x7fe9b006c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59826/0 (socket says 172.21.15.1:59826) 2024-05-09T05:06:15.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7fb942ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb93c0463a0 0x7fb93c051830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fb942ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb93c0463a0 0x7fb93c051830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44110/0 (socket says 172.21.15.1:44110) 2024-05-09T05:06:15.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fb942ffd640 1 -- 172.21.15.1:0/354724051 learned_addr learned my addr 172.21.15.1:0/354724051 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fe9af7fe640 1 -- 172.21.15.1:0/1406806760 learned_addr learned my addr 172.21.15.1:0/1406806760 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.035+0000 7fb9427fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c034050 0x7fb93c051d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fb9437fe640 1 --2- 172.21.15.1:0/354724051 >> v2:172.21.15.1:3300/0 conn(0x7fb93c0dfb70 0x7fb93c0dff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c036070 con 0x7f3250008c10 2024-05-09T05:06:15.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325c031400 con 0x7f3250008c10 2024-05-09T05:06:15.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7f32667fc640 1 -- 172.21.15.1:0/2179041725 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c009670 con 0x7f3250008c10 2024-05-09T05:06:15.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fb943fff640 1 -- 172.21.15.1:0/354724051 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb93c04ca00 con 0x7fb93c0dfb70 2024-05-09T05:06:15.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fe9b4e96640 1 --2- 172.21.15.1:0/1406806760 >> v2:172.21.15.1:3300/0 conn(0x7fe9b006cc30 0x7fe9b01631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fb943fff640 1 -- 172.21.15.1:0/354724051 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb93c0dcae0 con 0x7fb93c0463a0 2024-05-09T05:06:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fb943fff640 1 -- 172.21.15.1:0/354724051 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb93c0dcdc0 con 0x7fb93c034050 2024-05-09T05:06:15.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1406806760 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9b0152f00 con 0x7fe9b0152af0 2024-05-09T05:06:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1406806760 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe9b006f010 con 0x7fe9b006cc30 2024-05-09T05:06:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.036+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1406806760 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b006fb90 con 0x7fe9b01592a0 2024-05-09T05:06:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fb9427fc640 1 -- 172.21.15.1:0/354724051 >> v2:172.21.15.196:3300/0 conn(0x7fb93c0463a0 msgr2=0x7fb93c051830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fb9427fc640 1 --2- 172.21.15.1:0/354724051 >> v2:172.21.15.196:3300/0 conn(0x7fb93c0463a0 0x7fb93c051830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fb9427fc640 1 -- 172.21.15.1:0/354724051 >> v2:172.21.15.1:3300/0 conn(0x7fb93c0dfb70 msgr2=0x7fb93c0dff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fb9427fc640 1 --2- 172.21.15.1:0/354724051 >> v2:172.21.15.1:3300/0 conn(0x7fb93c0dfb70 0x7fb93c0dff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fb9427fc640 1 -- 172.21.15.1:0/354724051 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93c04cc50 con 0x7fb93c034050 2024-05-09T05:06:15.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7f326e138640 1 -- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 msgr2=0x7f3250009020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7f326e138640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f3250009020 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f325c009aa0 tx=0x7f3250009c40 comp rx=0 tx=0).stop 2024-05-09T05:06:15.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fe9b4e96640 1 -- 172.21.15.1:0/1406806760 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b01592a0 msgr2=0x7fe9b006c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fe9b4e96640 1 --2- 172.21.15.1:0/1406806760 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b01592a0 0x7fe9b006c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fe9b4e96640 1 -- 172.21.15.1:0/1406806760 >> v1:172.21.15.196:6789/0 conn(0x7fe9b0152af0 legacy=0x7fe9b00c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fe9b4e96640 1 -- 172.21.15.1:0/1406806760 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b01596b0 con 0x7fe9b006cc30 2024-05-09T05:06:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fbd8d72b640 1 Processor -- start 2024-05-09T05:06:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7f326e138640 1 -- 172.21.15.1:0/2179041725 shutdown_connections 2024-05-09T05:06:15.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7f326e138640 1 --2- 172.21.15.1:0/2179041725 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f3250009020 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7f326e138640 1 -- 172.21.15.1:0/2179041725 >> 172.21.15.1:0/2179041725 conn(0x7f32680cdcd0 msgr2=0x7f32680ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7f326e138640 1 -- 172.21.15.1:0/2179041725 shutdown_connections 2024-05-09T05:06:15.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7f326e138640 1 -- 172.21.15.1:0/2179041725 wait complete. 2024-05-09T05:06:15.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fbd8d72b640 1 -- start start 2024-05-09T05:06:15.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7f8dccd9f640 1 Processor -- start 2024-05-09T05:06:15.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb45bb86640 1 -- 172.21.15.1:0/4234666407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540740c0 msgr2=0x7fb45415ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb45bb86640 1 --2- 172.21.15.1:0/4234666407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540740c0 0x7fb45415ae30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb450009870 tx=0x7fb45002a3c0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.037+0000 7fb9437fe640 1 --2- 172.21.15.1:0/354724051 >> v2:172.21.15.1:3300/0 conn(0x7fb93c0dfb70 0x7fb93c0dff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb45bb86640 1 -- 172.21.15.1:0/4234666407 shutdown_connections 2024-05-09T05:06:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb45bb86640 1 --2- 172.21.15.1:0/4234666407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb4540740c0 0x7fb45415ae30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb45bb86640 1 -- 172.21.15.1:0/4234666407 >> 172.21.15.1:0/4234666407 conn(0x7fb454070f40 msgr2=0x7fb45406c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb9427fc640 1 --2- 172.21.15.1:0/354724051 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c034050 0x7fb93c051d70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0524c0 tx=0x7fb93802e870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a11adcf7d0d9387c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb45bb86640 1 -- 172.21.15.1:0/4234666407 shutdown_connections 2024-05-09T05:06:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fe9b4e96640 1 --2- 172.21.15.1:0/1406806760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9b006cc30 0x7fe9b01631a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800b050 tx=0x7fe9a802a840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=803a3033d77e9875 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7f8dccd9f640 1 -- start start 2024-05-09T05:06:15.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb45bb86640 1 -- 172.21.15.1:0/4234666407 wait complete. 2024-05-09T05:06:15.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.038+0000 7fb941ffb640 1 -- 172.21.15.1:0/354724051 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938034770 con 0x7fb93c034050 2024-05-09T05:06:15.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.039+0000 7fb941ffb640 1 -- 172.21.15.1:0/354724051 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb93803a070 con 0x7fb93c034050 2024-05-09T05:06:15.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.039+0000 7fb941ffb640 1 -- 172.21.15.1:0/354724051 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938034b30 con 0x7fb93c034050 2024-05-09T05:06:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.039+0000 7fe9aeffd640 1 -- 172.21.15.1:0/1406806760 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a802e050 con 0x7fe9b006cc30 2024-05-09T05:06:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.039+0000 7fe9aeffd640 1 -- 172.21.15.1:0/1406806760 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9a8032630 con 0x7fe9b006cc30 2024-05-09T05:06:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.039+0000 7fe9aeffd640 1 -- 172.21.15.1:0/1406806760 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a8032a40 con 0x7fe9b006cc30 2024-05-09T05:06:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.039+0000 7f70c89d9640 1 Processor -- start 2024-05-09T05:06:15.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7f70c89d9640 1 -- start start 2024-05-09T05:06:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fbd8d72b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd80047ca0 0x7fbd800d6ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fbd8d72b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd800480b0 0x7fbd80171900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fbd8d72b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd80035d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7f326e138640 1 Processor -- start 2024-05-09T05:06:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fbd87fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd80047ca0 0x7fbd800d6ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fbd87fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd80047ca0 0x7fbd800d6ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59830/0 (socket says 172.21.15.1:59830) 2024-05-09T05:06:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fbd877fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd800480b0 0x7fbd80171900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fbd877fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd800480b0 0x7fbd80171900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44116/0 (socket says 172.21.15.1:44116) 2024-05-09T05:06:15.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1406806760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9b006cc30 msgr2=0x7fe9b01631a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7f326e138640 1 -- start start 2024-05-09T05:06:15.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1406806760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9b006cc30 0x7fe9b01631a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe9a800b050 tx=0x7fe9a802a840 comp rx=0 tx=0).stop 2024-05-09T05:06:15.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7f8dccd9f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc0006520 0x7f8dc004e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fb45bb86640 1 Processor -- start 2024-05-09T05:06:15.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1406806760 shutdown_connections 2024-05-09T05:06:15.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fbd8cf2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd80035d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc0006520 0x7f8dc004e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1406806760 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9b006cc30 0x7fe9b01631a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fbd8cf2a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd80035d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49488/0 (socket says 172.21.15.1:49488) 2024-05-09T05:06:15.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7f8dc77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc0006520 0x7f8dc004e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49502/0 (socket says 172.21.15.1:49502) 2024-05-09T05:06:15.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1406806760 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b01592a0 0x7fe9b006c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fbd87fff640 1 -- 172.21.15.1:0/235034894 learned_addr learned my addr 172.21.15.1:0/235034894 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7f8dc77fe640 1 -- 172.21.15.1:0/1299993978 learned_addr learned my addr 172.21.15.1:0/1299993978 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1406806760 >> 172.21.15.1:0/1406806760 conn(0x7fe9b00cdcd0 msgr2=0x7fe9b00ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbd800484c0 con 0x7fbd8004da40 2024-05-09T05:06:15.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7f8dccd9f640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc0046870 0x7f8dc00e16d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd80047730 con 0x7fbd800480b0 2024-05-09T05:06:15.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbd8004d460 con 0x7fbd80047ca0 2024-05-09T05:06:15.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.041+0000 7f8dccd9f640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc00dda30 0x7f8dc004f410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dc6ffd640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc0046870 0x7f8dc00e16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7fb45bb86640 1 -- start start 2024-05-09T05:06:15.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dc7fff640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc00dda30 0x7f8dc004f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dccd9f640 1 -- 172.21.15.1:0/1299993978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc0045a70 con 0x7f8dc0006520 2024-05-09T05:06:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dccd9f640 1 -- 172.21.15.1:0/1299993978 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8dc0045790 con 0x7f8dc0046870 2024-05-09T05:06:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dccd9f640 1 -- 172.21.15.1:0/1299993978 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8dc0045d50 con 0x7f8dc00dda30 2024-05-09T05:06:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7fbd87fff640 1 -- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd800480b0 msgr2=0x7fbd80171900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7fbd87fff640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd800480b0 0x7fbd80171900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7fbd87fff640 1 -- 172.21.15.1:0/235034894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 msgr2=0x7fbd80035d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f70c89d9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c4073bd0 0x7f70c4207790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 -- 172.21.15.1:0/354724051 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c034050 msgr2=0x7fb93c051d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 --2- 172.21.15.1:0/354724051 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c034050 0x7fb93c051d70 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb93c0524c0 tx=0x7fb93802e870 comp rx=0 tx=0).stop 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 -- 172.21.15.1:0/354724051 shutdown_connections 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 --2- 172.21.15.1:0/354724051 >> v2:172.21.15.1:3300/0 conn(0x7fb93c0dfb70 0x7fb93c0dff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 --2- 172.21.15.1:0/354724051 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c034050 0x7fb93c051d70 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 --2- 172.21.15.1:0/354724051 >> v2:172.21.15.196:3300/0 conn(0x7fb93c0463a0 0x7fb93c051830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 -- 172.21.15.1:0/354724051 >> 172.21.15.1:0/354724051 conn(0x7fb93c008070 msgr2=0x7fb93c008480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 -- 172.21.15.1:0/354724051 shutdown_connections 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.040+0000 7fb943fff640 1 -- 172.21.15.1:0/354724051 wait complete. 2024-05-09T05:06:15.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7fb943fff640 1 Processor -- start 2024-05-09T05:06:15.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c89d9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c40c3460 0x7f70c40cbe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7fbd87fff640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd80035d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7fbd87fff640 1 -- 172.21.15.1:0/235034894 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd800088a0 con 0x7fbd80047ca0 2024-05-09T05:06:15.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7fbd8cf2a640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd80035d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:15.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dc77fe640 1 -- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc00dda30 msgr2=0x7f8dc004f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c4073bd0 0x7f70c4207790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c4073bd0 0x7f70c4207790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49514/0 (socket says 172.21.15.1:49514) 2024-05-09T05:06:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1406806760 shutdown_connections 2024-05-09T05:06:15.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c37fe640 1 -- 172.21.15.1:0/1391496679 learned_addr learned my addr 172.21.15.1:0/1391496679 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7fb943fff640 1 -- start start 2024-05-09T05:06:15.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1406806760 wait complete. 2024-05-09T05:06:15.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c89d9640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f70c416d0f0 0x7f70c4086690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f70c408d9a0 con 0x7f70c4073bd0 2024-05-09T05:06:15.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7fbd87fff640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd80047ca0 0x7fbd800d6ba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbd8802e1c0 tx=0x7fbd8806a820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dc77fe640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc00dda30 0x7f8dc004f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dc77fe640 1 -- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc0046870 msgr2=0x7f8dc00e16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dc77fe640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc0046870 0x7f8dc00e16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.042+0000 7f8dc77fe640 1 -- 172.21.15.1:0/1299993978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8dc004eba0 con 0x7f8dc0006520 2024-05-09T05:06:15.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f70c40c3db0 con 0x7f70c416d0f0 2024-05-09T05:06:15.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c3fff640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f70c416d0f0 0x7f70c4086690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7fbd84ff9640 1 -- 172.21.15.1:0/235034894 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd880b2040 con 0x7fbd80047ca0 2024-05-09T05:06:15.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f326e138640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f32681f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c2ffd640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c40c3460 0x7f70c40cbe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7fbd84ff9640 1 -- 172.21.15.1:0/235034894 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd880ad8c0 con 0x7fbd80047ca0 2024-05-09T05:06:15.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f326e138640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3268159630 0x7f3268154450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd80170e40 con 0x7fbd80047ca0 2024-05-09T05:06:15.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f32677fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f32681f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f70c4072cd0 con 0x7f70c40c3460 2024-05-09T05:06:15.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f32677fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f32681f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59872/0 (socket says 172.21.15.1:59872) 2024-05-09T05:06:15.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd800d7810 con 0x7fbd80047ca0 2024-05-09T05:06:15.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3266ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3268159630 0x7f3268154450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3266ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3268159630 0x7f3268154450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44138/0 (socket says 172.21.15.1:44138) 2024-05-09T05:06:15.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3266ffd640 1 -- 172.21.15.1:0/3083348531 learned_addr learned my addr 172.21.15.1:0/3083348531 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f326e138640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326814dd00 0x7f3268148770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f326806f820 con 0x7f326814dd00 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f326806fb90 con 0x7f3268159630 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3268073ea0 con 0x7f3250008c10 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3267fff640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326814dd00 0x7f3268148770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3266ffd640 1 -- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 msgr2=0x7f32681f2690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3266ffd640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f32681f2690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3266ffd640 1 -- 172.21.15.1:0/3083348531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326814dd00 msgr2=0x7f3268148770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3266ffd640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326814dd00 0x7f3268148770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7fb45bb86640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4540740c0 0x7fb454156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7fb45bb86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb454074610 0x7fb45415dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7fb45bb86640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb454074a20 0x7fb45415e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7fb45bb86640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb454073610 con 0x7fb454074610 2024-05-09T05:06:15.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3266ffd640 1 -- 172.21.15.1:0/3083348531 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f326814e8d0 con 0x7f3268159630 2024-05-09T05:06:15.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f3267fff640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326814dd00 0x7f3268148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:15.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7fb45bb86640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb45415b370 con 0x7fb454074a20 2024-05-09T05:06:15.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7fb45bb86640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb4540c5290 con 0x7fb4540740c0 2024-05-09T05:06:15.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7fb459121640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb454074610 0x7fb45415dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.043+0000 7f8dc77fe640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc0006520 0x7f8dc004e660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8db402a7b0 tx=0x7f8db40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f8db3fff640 1 -- 172.21.15.1:0/1299993978 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4042020 con 0x7f8dc0006520 2024-05-09T05:06:15.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f8db3fff640 1 -- 172.21.15.1:0/1299993978 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db403d670 con 0x7f8dc0006520 2024-05-09T05:06:15.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7f8db3fff640 1 -- 172.21.15.1:0/1299993978 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db4037040 con 0x7f8dc0006520 2024-05-09T05:06:15.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f70c37fe640 1 -- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c40c3460 msgr2=0x7f70c40cbe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7f3266ffd640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3268159630 0x7f3268154450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f325c009bc0 tx=0x7f325c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f70c37fe640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c40c3460 0x7f70c40cbe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f70c37fe640 1 -- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f70c416d0f0 msgr2=0x7f70c4086690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7fb459121640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb454074610 0x7fb45415dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49526/0 (socket says 172.21.15.1:49526) 2024-05-09T05:06:15.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f70c37fe640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f70c416d0f0 0x7f70c4086690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.044+0000 7f70c37fe640 1 -- 172.21.15.1:0/1391496679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70c4207cd0 con 0x7f70c4073bd0 2024-05-09T05:06:15.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7f324ffff640 1 -- 172.21.15.1:0/3083348531 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c041070 con 0x7f3268159630 2024-05-09T05:06:15.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7fb459121640 1 -- 172.21.15.1:0/3827792635 learned_addr learned my addr 172.21.15.1:0/3827792635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7fb45a123640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb454074a20 0x7fb45415e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7fb459922640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4540740c0 0x7fb454156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb45a123640 1 -- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4540740c0 msgr2=0x7fb454156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb45a123640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4540740c0 0x7fb454156050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb45a123640 1 -- 172.21.15.1:0/3827792635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb454074610 msgr2=0x7fb45415dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb45a123640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb454074610 0x7fb45415dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb45a123640 1 -- 172.21.15.1:0/3827792635 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb454156590 con 0x7fb454074a20 2024-05-09T05:06:15.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7f70c37fe640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c4073bd0 0x7f70c4207790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f70b800b410 tx=0x7f70b800b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb943fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93c034050 0x7fb93c0d2710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb943fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c0463a0 0x7fb93c0da810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb943fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb93c0dfb70 0x7fb93c0dec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb943fff640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb93c04cf30 con 0x7fb93c034050 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb943fff640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb93c0e04c0 con 0x7fb93c0dfb70 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb943fff640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb93c0d6340 con 0x7fb93c0463a0 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb9427fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c0463a0 0x7fb93c0da810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb9427fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c0463a0 0x7fb93c0da810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59896/0 (socket says 172.21.15.1:59896) 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fb9427fc640 1 -- 172.21.15.1:0/671197023 learned_addr learned my addr 172.21.15.1:0/671197023 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb9437fe640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb93c0dfb70 0x7fb93c0dec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb9427fc640 1 -- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb93c0dfb70 msgr2=0x7fb93c0dec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb9427fc640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb93c0dfb70 0x7fb93c0dec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb9427fc640 1 -- 172.21.15.1:0/671197023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93c034050 msgr2=0x7fb93c0d2710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb9427fc640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93c034050 0x7fb93c0d2710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb9427fc640 1 -- 172.21.15.1:0/671197023 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb93c0db160 con 0x7fb93c0463a0 2024-05-09T05:06:15.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb9437fe640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb93c0dfb70 0x7fb93c0dec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:15.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb9427fc640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c0463a0 0x7fb93c0da810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb93800ba40 tx=0x7fb9380046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32681533b0 con 0x7f3268159630 2024-05-09T05:06:15.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.045+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3268153970 con 0x7f3268159630 2024-05-09T05:06:15.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7f324ffff640 1 -- 172.21.15.1:0/3083348531 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f325c031400 con 0x7f3268159630 2024-05-09T05:06:15.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7f324ffff640 1 -- 172.21.15.1:0/3083348531 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f325c036040 con 0x7f3268159630 2024-05-09T05:06:15.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fbd84ff9640 1 -- 172.21.15.1:0/235034894 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd880adcb0 con 0x7fbd80047ca0 2024-05-09T05:06:15.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd800058e0 con 0x7fbd80047ca0 2024-05-09T05:06:15.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fbd84ff9640 1 -- 172.21.15.1:0/235034894 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbd880bf020 con 0x7fbd80047ca0 2024-05-09T05:06:15.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7f8dccd9f640 1 -- 172.21.15.1:0/1299993978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8dc0035a10 con 0x7f8dc0006520 2024-05-09T05:06:15.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7f8dccd9f640 1 -- 172.21.15.1:0/1299993978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8dc0035fd0 con 0x7f8dc0006520 2024-05-09T05:06:15.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.046+0000 7fe9b68f9640 1 Processor -- start 2024-05-09T05:06:15.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fe9b68f9640 1 -- start start 2024-05-09T05:06:15.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.048+0000 7f8db1ffb640 1 -- 172.21.15.1:0/1299993978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8dc00087a0 con 0x7f8dc0006520 2024-05-09T05:06:15.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.048+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3248005290 con 0x7f3268159630 2024-05-09T05:06:15.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.048+0000 7fbd84ff9640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd6c0763a0 0x7fbd6c0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb45a123640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb454074a20 0x7fb45415e2a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00c700 tx=0x7fb44c00cbc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb44a7fc640 1 -- 172.21.15.1:0/3827792635 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c003b70 con 0x7fb454074a20 2024-05-09T05:06:15.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb44a7fc640 1 -- 172.21.15.1:0/3827792635 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb44c003d10 con 0x7fb454074a20 2024-05-09T05:06:15.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.047+0000 7fb44a7fc640 1 -- 172.21.15.1:0/3827792635 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44c003b70 con 0x7fb454074a20 2024-05-09T05:06:15.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.048+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4541f74a0 con 0x7fb454074a20 2024-05-09T05:06:15.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.048+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4541578d0 con 0x7fb454074a20 2024-05-09T05:06:15.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.048+0000 7f70abfff640 1 -- 172.21.15.1:0/1391496679 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8015020 con 0x7f70c4073bd0 2024-05-09T05:06:15.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7f70abfff640 1 -- 172.21.15.1:0/1391496679 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f70b8011830 con 0x7f70c4073bd0 2024-05-09T05:06:15.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7f70abfff640 1 -- 172.21.15.1:0/1391496679 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70b8011c20 con 0x7f70c4073bd0 2024-05-09T05:06:15.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70c40880e0 con 0x7f70c4073bd0 2024-05-09T05:06:15.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b68f9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b006cc30 0x7fe9b01555c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70c420ad00 con 0x7f70c4073bd0 2024-05-09T05:06:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b68f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9b0152af0 0x7fe9b0153ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7f324ffff640 1 -- 172.21.15.1:0/3083348531 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f325c039040 con 0x7f3268159630 2024-05-09T05:06:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b68f9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9b01592a0 0x7fe9b00c5270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b68f9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b006f540 con 0x7fe9b0152af0 2024-05-09T05:06:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b68f9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe9b006f260 con 0x7fe9b01592a0 2024-05-09T05:06:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b4e96640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9b01592a0 0x7fe9b00c5270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b4e96640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9b01592a0 0x7fe9b00c5270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44152/0 (socket says 172.21.15.1:44152) 2024-05-09T05:06:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b4e96640 1 -- 172.21.15.1:0/1861253132 learned_addr learned my addr 172.21.15.1:0/1861253132 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9affff640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b006cc30 0x7fe9b01555c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe9b006f820 con 0x7fe9b006cc30 2024-05-09T05:06:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7f324ffff640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32540762d0 0x7f32540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7fe9affff640 1 -- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9b01592a0 msgr2=0x7fe9b00c5270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7fe9affff640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9b01592a0 0x7fe9b00c5270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7fe9affff640 1 -- 172.21.15.1:0/1861253132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9b0152af0 msgr2=0x7fe9b0153ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7fe9affff640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9b0152af0 0x7fe9b0153ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb454155c70 con 0x7fb454074a20 2024-05-09T05:06:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7fe9affff640 1 -- 172.21.15.1:0/1861253132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9b0155b00 con 0x7fe9b006cc30 2024-05-09T05:06:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.049+0000 7f8db3fff640 1 -- 172.21.15.1:0/1299993978 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8db4045020 con 0x7f8dc0006520 2024-05-09T05:06:15.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7f32677fe640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32540762d0 0x7f32540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7f8db3fff640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8dac0763a0 0x7f8dac0788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7f324ffff640 1 -- 172.21.15.1:0/3083348531 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f325c0b7b40 con 0x7f3268159630 2024-05-09T05:06:15.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7fe9affff640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b006cc30 0x7fe9b01555c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000b3e0 tx=0x7fe9a000b8a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7f8db3fff640 1 -- 172.21.15.1:0/1299993978 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8db40b8980 con 0x7f8dc0006520 2024-05-09T05:06:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.051+0000 7fbd877fe640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd6c0763a0 0x7fbd6c0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.051+0000 7fbd84ff9640 1 -- 172.21.15.1:0/235034894 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbd88134f10 con 0x7fbd80047ca0 2024-05-09T05:06:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.050+0000 7f8dc6ffd640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8dac0763a0 0x7f8dac0788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.051+0000 7f32677fe640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32540762d0 0x7f32540787e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3258001860 tx=0x7f3258001ab0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.051+0000 7fb44a7fc640 1 -- 172.21.15.1:0/3827792635 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb44c0050d0 con 0x7fb454074a20 2024-05-09T05:06:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.051+0000 7fbd877fe640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd6c0763a0 0x7fbd6c0788b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd800dd0b0 tx=0x7fbd7c016040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.051+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70c4050870 con 0x7f70c4073bd0 2024-05-09T05:06:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.052+0000 7f8dc6ffd640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8dac0763a0 0x7f8dac0788b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00e3e20 tx=0x7f8dbc001dd0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.052+0000 7fb44a7fc640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb438076700 0x7fb438078c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.052+0000 7fb44a7fc640 1 -- 172.21.15.1:0/3827792635 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb44c003970 con 0x7fb454074a20 2024-05-09T05:06:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fb459922640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb438076700 0x7fb438078c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fe9acff9640 1 -- 172.21.15.1:0/1861253132 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0015020 con 0x7fe9b006cc30 2024-05-09T05:06:15.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fb92bfff640 1 -- 172.21.15.1:0/671197023 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb938044070 con 0x7fb93c0463a0 2024-05-09T05:06:15.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9b00c6a50 con 0x7fe9b006cc30 2024-05-09T05:06:15.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9b0154dc0 con 0x7fe9b006cc30 2024-05-09T05:06:15.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fe9acff9640 1 -- 172.21.15.1:0/1861253132 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9a0011830 con 0x7fe9b006cc30 2024-05-09T05:06:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb93c0d08f0 con 0x7fb93c0463a0 2024-05-09T05:06:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fb92bfff640 1 -- 172.21.15.1:0/671197023 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb938010440 con 0x7fb93c0463a0 2024-05-09T05:06:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fbd84ff9640 1 -- 172.21.15.1:0/235034894 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbd880ae8f0 con 0x7fbd80047ca0 2024-05-09T05:06:15.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb93c0d0b90 con 0x7fb93c0463a0 2024-05-09T05:06:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.053+0000 7fb92bfff640 1 -- 172.21.15.1:0/671197023 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb9380107f0 con 0x7fb93c0463a0 2024-05-09T05:06:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.054+0000 7fb459922640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb438076700 0x7fb438078c10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb450034000 tx=0x7fb45002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.055+0000 7fe9acff9640 1 -- 172.21.15.1:0/1861253132 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9a0011be0 con 0x7fe9b006cc30 2024-05-09T05:06:15.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.055+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb93c0054a0 con 0x7fb93c0463a0 2024-05-09T05:06:15.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.055+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe990005290 con 0x7fe9b006cc30 2024-05-09T05:06:15.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.055+0000 7f324ffff640 1 -- 172.21.15.1:0/3083348531 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f325c080e40 con 0x7f3268159630 2024-05-09T05:06:15.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.055+0000 7fe9acff9640 1 -- 172.21.15.1:0/1861253132 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe9a0018020 con 0x7fe9b006cc30 2024-05-09T05:06:15.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.056+0000 7fb92bfff640 1 -- 172.21.15.1:0/671197023 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb938047070 con 0x7fb93c0463a0 2024-05-09T05:06:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.056+0000 7fe9acff9640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe99c076300 0x7fe99c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.056+0000 7fe9acff9640 1 -- 172.21.15.1:0/1861253132 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe9a00970c0 con 0x7fe9b006cc30 2024-05-09T05:06:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.057+0000 7fe9af7fe640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe99c076300 0x7fe99c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.057+0000 7fb92bfff640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb9300763d0 0x7fb9300788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.057+0000 7f8db3fff640 1 -- 172.21.15.1:0/1299993978 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8dc00087a0 con 0x7f8dc0006520 2024-05-09T05:06:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.057+0000 7fb92bfff640 1 -- 172.21.15.1:0/671197023 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb9380b8760 con 0x7fb93c0463a0 2024-05-09T05:06:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.057+0000 7fb942ffd640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb9300763d0 0x7fb9300788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.058+0000 7fe9af7fe640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe99c076300 0x7fe99c078810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0156310 tx=0x7fe9a4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.058+0000 7fb942ffd640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb9300763d0 0x7fb9300788e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb934006a90 tx=0x7fb934006a20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.060+0000 7f70abfff640 1 -- 172.21.15.1:0/1391496679 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f70b8018020 con 0x7f70c4073bd0 2024-05-09T05:06:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.060+0000 7f70abfff640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f70b00763a0 0x7f70b00788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.060+0000 7f70abfff640 1 -- 172.21.15.1:0/1391496679 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f70b8097b10 con 0x7f70c4073bd0 2024-05-09T05:06:15.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.061+0000 7fb44a7fc640 1 -- 172.21.15.1:0/3827792635 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb44c020020 con 0x7fb454074a20 2024-05-09T05:06:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.061+0000 7f70c2ffd640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f70b00763a0 0x7f70b00788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.061+0000 7fe9acff9640 1 -- 172.21.15.1:0/1861253132 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe9a001f030 con 0x7fe9b006cc30 2024-05-09T05:06:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.061+0000 7f70c2ffd640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f70b00763a0 0x7f70b00788b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f70b4006fd0 tx=0x7f70b4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.064+0000 7f70abfff640 1 -- 172.21.15.1:0/1391496679 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f70b801f020 con 0x7f70c4073bd0 2024-05-09T05:06:15.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.066+0000 7fb92bfff640 1 -- 172.21.15.1:0/671197023 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb938082130 con 0x7fb93c0463a0 2024-05-09T05:06:15.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.434+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7efc48005580 con 0x7efc580d0720 2024-05-09T05:06:15.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.435+0000 7efc527fc640 1 -- 172.21.15.1:0/1313842553 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7efc58184cf0 con 0x7efc580d0720 2024-05-09T05:06:15.444 INFO:teuthology.orchestra.run.smithi001.stdout:64424509447 2024-05-09T05:06:15.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.444+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc44076400 msgr2=0x7efc44078910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.444+0000 7efc668dd640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc44076400 0x7efc44078910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7efc580369a0 tx=0x7efc54006b30 comp rx=0 tx=0).stop 2024-05-09T05:06:15.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.444+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc580d0720 msgr2=0x7efc580e3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.444+0000 7efc668dd640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc580d0720 0x7efc580e3710 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7efc6006b7d0 tx=0x7efc6006a820 comp rx=0 tx=0).stop 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.445+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 shutdown_connections 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.445+0000 7efc668dd640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efc44076400 0x7efc44078910 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.445+0000 7efc668dd640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efc5804e870 0x7efc581855f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.445+0000 7efc668dd640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efc580d0720 0x7efc580e3710 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.445+0000 7efc668dd640 1 --2- 172.21.15.1:0/1313842553 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efc5804c560 0x7efc581847b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.445+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 >> 172.21.15.1:0/1313842553 conn(0x7efc580527a0 msgr2=0x7efc580e5ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.445+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 shutdown_connections 2024-05-09T05:06:15.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.445+0000 7efc668dd640 1 -- 172.21.15.1:0/1313842553 wait complete. 2024-05-09T05:06:15.465 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509447 got 64424509447 for osd.5 2024-05-09T05:06:15.465 DEBUG:teuthology.parallel:result is None 2024-05-09T05:06:15.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.476+0000 7f8db1ffb640 1 -- 172.21.15.1:0/1299993978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f8dc00058e0 con 0x7f8dc0006520 2024-05-09T05:06:15.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.476+0000 7f8db3fff640 1 -- 172.21.15.1:0/1299993978 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f8db404c020 con 0x7f8dc0006520 2024-05-09T05:06:15.477 INFO:teuthology.orchestra.run.smithi001.stdout:60129542152 2024-05-09T05:06:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 -- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8dac0763a0 msgr2=0x7f8dac0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8dac0763a0 0x7f8dac0788b0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f8dc00e3e20 tx=0x7f8dbc001dd0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 -- 172.21.15.1:0/1299993978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc0006520 msgr2=0x7f8dc004e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc0006520 0x7f8dc004e660 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f8db402a7b0 tx=0x7f8db40046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 -- 172.21.15.1:0/1299993978 shutdown_connections 2024-05-09T05:06:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8dac0763a0 0x7f8dac0788b0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8dc0046870 0x7f8dc00e16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8dc00dda30 0x7f8dc004f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 --2- 172.21.15.1:0/1299993978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8dc0006520 0x7f8dc004e660 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 -- 172.21.15.1:0/1299993978 >> 172.21.15.1:0/1299993978 conn(0x7f8dc00507f0 msgr2=0x7f8dc00dc200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 -- 172.21.15.1:0/1299993978 shutdown_connections 2024-05-09T05:06:15.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.479+0000 7f8db1ffb640 1 -- 172.21.15.1:0/1299993978 wait complete. 2024-05-09T05:06:15.491 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.1 2024-05-09T05:06:15.491 DEBUG:teuthology.parallel:result is None 2024-05-09T05:06:15.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.506+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f3248005580 con 0x7f3268159630 2024-05-09T05:06:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.506+0000 7f324ffff640 1 -- 172.21.15.1:0/3083348531 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f325c085140 con 0x7f3268159630 2024-05-09T05:06:15.507 INFO:teuthology.orchestra.run.smithi001.stdout:64424509446 2024-05-09T05:06:15.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.509+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32540762d0 msgr2=0x7f32540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.509+0000 7f326e138640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32540762d0 0x7f32540787e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f3258001860 tx=0x7f3258001ab0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.509+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3268159630 msgr2=0x7f3268154450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3268159630 0x7f3268154450 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f325c009bc0 tx=0x7f325c004730 comp rx=0 tx=0).stop 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 shutdown_connections 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32540762d0 0x7f32540787e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3268159630 0x7f3268154450 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f326814dd00 0x7f3268148770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 --2- 172.21.15.1:0/3083348531 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3250008c10 0x7f32681f2690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 >> 172.21.15.1:0/3083348531 conn(0x7f32680cdcd0 msgr2=0x7f3268150160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 shutdown_connections 2024-05-09T05:06:15.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.510+0000 7f326e138640 1 -- 172.21.15.1:0/3083348531 wait complete. 2024-05-09T05:06:15.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.513+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7fbd8004d820 con 0x7fbd80047ca0 2024-05-09T05:06:15.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.513+0000 7fbd84ff9640 1 -- 172.21.15.1:0/235034894 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fbd880fe200 con 0x7fbd80047ca0 2024-05-09T05:06:15.514 INFO:teuthology.orchestra.run.smithi001.stdout:60129542151 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd6c0763a0 msgr2=0x7fbd6c0788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd6c0763a0 0x7fbd6c0788b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fbd800dd0b0 tx=0x7fbd7c016040 comp rx=0 tx=0).stop 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd80047ca0 msgr2=0x7fbd800d6ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd80047ca0 0x7fbd800d6ba0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fbd8802e1c0 tx=0x7fbd8806a820 comp rx=0 tx=0).stop 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 shutdown_connections 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd6c0763a0 0x7fbd6c0788b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd800480b0 0x7fbd80171900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd8004da40 0x7fbd80035d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 --2- 172.21.15.1:0/235034894 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd80047ca0 0x7fbd800d6ba0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 >> 172.21.15.1:0/235034894 conn(0x7fbd800514b0 msgr2=0x7fbd80050cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 shutdown_connections 2024-05-09T05:06:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.516+0000 7fbd8d72b640 1 -- 172.21.15.1:0/235034894 wait complete. 2024-05-09T05:06:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.517+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fe990005850 con 0x7fe9b006cc30 2024-05-09T05:06:15.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.517+0000 7fe9acff9640 1 -- 172.21.15.1:0/1861253132 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fe9a0060b20 con 0x7fe9b006cc30 2024-05-09T05:06:15.518 INFO:teuthology.orchestra.run.smithi001.stdout:60129542153 2024-05-09T05:06:15.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.517+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7fb45406d470 con 0x7fb454074a20 2024-05-09T05:06:15.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.518+0000 7fb44a7fc640 1 -- 172.21.15.1:0/3827792635 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb44c020020 con 0x7fb454074a20 2024-05-09T05:06:15.519 INFO:teuthology.orchestra.run.smithi001.stdout:60129542152 2024-05-09T05:06:15.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.518+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fb93c04d5d0 con 0x7fb93c0463a0 2024-05-09T05:06:15.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.518+0000 7fb92bfff640 1 -- 172.21.15.1:0/671197023 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fb938081a30 con 0x7fb93c0463a0 2024-05-09T05:06:15.520 INFO:teuthology.orchestra.run.smithi001.stdout:60129542151 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe99c076300 msgr2=0x7fe99c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe99c076300 0x7fe99c078810 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe9b0156310 tx=0x7fe9a4008040 comp rx=0 tx=0).stop 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b006cc30 msgr2=0x7fe9b01555c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b006cc30 0x7fe9b01555c0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fe9a000b3e0 tx=0x7fe9a000b8a0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 shutdown_connections 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe99c076300 0x7fe99c078810 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9b01592a0 0x7fe9b00c5270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9b0152af0 0x7fe9b0153ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 --2- 172.21.15.1:0/1861253132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9b006cc30 0x7fe9b01555c0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 >> 172.21.15.1:0/1861253132 conn(0x7fe9b00cdcd0 msgr2=0x7fe9b00caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 shutdown_connections 2024-05-09T05:06:15.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb438076700 msgr2=0x7fb438078c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fb45bb86640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb438076700 0x7fb438078c10 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb450034000 tx=0x7fb45002e000 comp rx=0 tx=0).stop 2024-05-09T05:06:15.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb454074a20 msgr2=0x7fb45415e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fb45bb86640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb454074a20 0x7fb45415e2a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00c700 tx=0x7fb44c00cbc0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.521+0000 7fe9b68f9640 1 -- 172.21.15.1:0/1861253132 wait complete. 2024-05-09T05:06:15.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 shutdown_connections 2024-05-09T05:06:15.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb45bb86640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb438076700 0x7fb438078c10 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb45bb86640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb454074a20 0x7fb45415e2a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb45bb86640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb454074610 0x7fb45415dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb45bb86640 1 --2- 172.21.15.1:0/3827792635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4540740c0 0x7fb454156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 >> 172.21.15.1:0/3827792635 conn(0x7fb454070f40 msgr2=0x7fb4540c7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 shutdown_connections 2024-05-09T05:06:15.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb45bb86640 1 -- 172.21.15.1:0/3827792635 wait complete. 2024-05-09T05:06:15.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb9300763d0 msgr2=0x7fb9300788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb9300763d0 0x7fb9300788e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fb934006a90 tx=0x7fb934006a20 comp rx=0 tx=0).stop 2024-05-09T05:06:15.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c0463a0 msgr2=0x7fb93c0da810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c0463a0 0x7fb93c0da810 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb93800ba40 tx=0x7fb9380046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 shutdown_connections 2024-05-09T05:06:15.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb9300763d0 0x7fb9300788e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb93c0dfb70 0x7fb93c0dec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb93c0463a0 0x7fb93c0da810 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb93800ba40 tx=0x7fb9380046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 --2- 172.21.15.1:0/671197023 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb93c034050 0x7fb93c0d2710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 >> 172.21.15.1:0/671197023 conn(0x7fb93c008070 msgr2=0x7fb93c04d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.522+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 shutdown_connections 2024-05-09T05:06:15.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.523+0000 7fb943fff640 1 -- 172.21.15.1:0/671197023 wait complete. 2024-05-09T05:06:15.529 INFO:tasks.ceph.ceph_manager.ceph:need seq 64424509446 got 64424509446 for osd.7 2024-05-09T05:06:15.529 DEBUG:teuthology.parallel:result is None 2024-05-09T05:06:15.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.530+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f70c40c9bd0 con 0x7f70c4073bd0 2024-05-09T05:06:15.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.530+0000 7f70abfff640 1 -- 172.21.15.1:0/1391496679 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f70b8061450 con 0x7f70c4073bd0 2024-05-09T05:06:15.531 INFO:teuthology.orchestra.run.smithi001.stdout:60129542152 2024-05-09T05:06:15.533 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.0 2024-05-09T05:06:15.533 DEBUG:teuthology.parallel:result is None 2024-05-09T05:06:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f70b00763a0 msgr2=0x7f70b00788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f70b00763a0 0x7f70b00788b0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f70b4006fd0 tx=0x7f70b4008040 comp rx=0 tx=0).stop 2024-05-09T05:06:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c4073bd0 msgr2=0x7f70c4207790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c4073bd0 0x7f70c4207790 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f70b800b410 tx=0x7f70b800b8d0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 shutdown_connections 2024-05-09T05:06:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f70b00763a0 0x7f70b00788b0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f70c416d0f0 0x7f70c4086690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f70c40c3460 0x7f70c40cbe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 --2- 172.21.15.1:0/1391496679 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f70c4073bd0 0x7f70c4207790 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 >> 172.21.15.1:0/1391496679 conn(0x7f70c406cb10 msgr2=0x7f70c406cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.534+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 shutdown_connections 2024-05-09T05:06:15.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.535+0000 7f70c89d9640 1 -- 172.21.15.1:0/1391496679 wait complete. 2024-05-09T05:06:15.537 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542153 got 60129542153 for osd.2 2024-05-09T05:06:15.537 DEBUG:teuthology.parallel:result is None 2024-05-09T05:06:15.540 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542151 got 60129542151 for osd.6 2024-05-09T05:06:15.540 DEBUG:teuthology.parallel:result is None 2024-05-09T05:06:15.541 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.3 2024-05-09T05:06:15.541 DEBUG:teuthology.parallel:result is None 2024-05-09T05:06:15.548 INFO:tasks.ceph.ceph_manager.ceph:need seq 60129542152 got 60129542152 for osd.4 2024-05-09T05:06:15.548 DEBUG:teuthology.parallel:result is None 2024-05-09T05:06:15.548 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T05:06:15.548 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:06:15.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.702+0000 7f2f9b35f640 1 Processor -- start 2024-05-09T05:06:15.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.702+0000 7f2f9b35f640 1 -- start start 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f9b35f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2f940c64b0 0x7f2f94156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f9b35f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f2f9406e1f0 0x7f2f94068650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f9b35f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2f940c6bc0 con 0x7f2f9406e600 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f9b35f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2f9415c8e0 con 0x7f2f940c64b0 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f9b35f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f2f9406eaf0 con 0x7f2f9406e1f0 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f990fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2f940c64b0 0x7f2f94156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f990fb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2f940c64b0 0x7f2f94156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49554/0 (socket says 172.21.15.1:49554) 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f990fb640 1 -- 172.21.15.1:0/1074012966 learned_addr learned my addr 172.21.15.1:0/1074012966 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f988fa640 1 --2- 172.21.15.1:0/1074012966 >> v2:172.21.15.196:3300/0 conn(0x7f2f9406e1f0 0x7f2f94068650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f990fb640 1 -- 172.21.15.1:0/1074012966 >> v2:172.21.15.196:3300/0 conn(0x7f2f9406e1f0 msgr2=0x7f2f94068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f990fb640 1 --2- 172.21.15.1:0/1074012966 >> v2:172.21.15.196:3300/0 conn(0x7f2f9406e1f0 0x7f2f94068650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f990fb640 1 -- 172.21.15.1:0/1074012966 >> v1:172.21.15.196:6789/0 conn(0x7f2f9406e600 legacy=0x7f2f94070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.704+0000 7f2f990fb640 1 -- 172.21.15.1:0/1074012966 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9415cba0 con 0x7f2f940c64b0 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.705+0000 7f2f990fb640 1 --2- 172.21.15.1:0/1074012966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94156c90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2f94069350 tx=0x7f2f8c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=577902a5b557164f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.705+0000 7f2f93fff640 1 -- 172.21.15.1:0/1074012966 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c037070 con 0x7f2f940c64b0 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.705+0000 7f2f93fff640 1 -- 172.21.15.1:0/1074012966 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f8c032440 con 0x7f2f940c64b0 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.705+0000 7f2f93fff640 1 -- 172.21.15.1:0/1074012966 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c032810 con 0x7f2f940c64b0 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.706+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1074012966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 msgr2=0x7f2f94156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.706+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1074012966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94156c90 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f2f94069350 tx=0x7f2f8c02a270 comp rx=0 tx=0).stop 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.706+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1074012966 shutdown_connections 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.706+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1074012966 >> v2:172.21.15.196:3300/0 conn(0x7f2f9406e1f0 0x7f2f94068650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.706+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1074012966 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94156c90 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.706+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1074012966 >> 172.21.15.1:0/1074012966 conn(0x7f2f940cc6a0 msgr2=0x7f2f940ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.706+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1074012966 shutdown_connections 2024-05-09T05:06:15.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.706+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1074012966 wait complete. 2024-05-09T05:06:15.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.708+0000 7f2f9b35f640 1 Processor -- start 2024-05-09T05:06:15.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.708+0000 7f2f9b35f640 1 -- start start 2024-05-09T05:06:15.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f9b35f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2f9406e1f0 0x7f2f94154ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f9b35f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f9406e600 0x7f2f941545e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f9b35f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f9b35f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9415d440 con 0x7f2f940c64b0 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f9b35f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9415d160 con 0x7f2f9406e600 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f9b35f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2f940712e0 con 0x7f2f9406e1f0 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94150340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49568/0 (socket says 172.21.15.1:49568) 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 -- 172.21.15.1:0/1101654088 learned_addr learned my addr 172.21.15.1:0/1101654088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f990fb640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2f9406e1f0 0x7f2f94154ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f988fa640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f9406e600 0x7f2f941545e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 -- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2f9406e1f0 msgr2=0x7f2f94154ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2f9406e1f0 0x7f2f94154ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 -- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f9406e600 msgr2=0x7f2f941545e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f9406e600 0x7f2f941545e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 -- 172.21.15.1:0/1101654088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f941567f0 con 0x7f2f940c64b0 2024-05-09T05:06:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.709+0000 7f2f998fc640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94150340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400c700 tx=0x7f2f8400cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.710+0000 7f2f91ffb640 1 -- 172.21.15.1:0/1101654088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f84003b70 con 0x7f2f940c64b0 2024-05-09T05:06:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.710+0000 7f2f91ffb640 1 -- 172.21.15.1:0/1101654088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2f84003d10 con 0x7f2f940c64b0 2024-05-09T05:06:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.710+0000 7f2f91ffb640 1 -- 172.21.15.1:0/1101654088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8400aa10 con 0x7f2f940c64b0 2024-05-09T05:06:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.710+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f940b4430 con 0x7f2f940c64b0 2024-05-09T05:06:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.710+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9415fb60 con 0x7f2f940c64b0 2024-05-09T05:06:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.711+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2f9415fdc0 con 0x7f2f940c64b0 2024-05-09T05:06:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.711+0000 7f2f91ffb640 1 -- 172.21.15.1:0/1101654088 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2f8400abb0 con 0x7f2f940c64b0 2024-05-09T05:06:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.712+0000 7f2f91ffb640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2f800762d0 0x7f2f800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.712+0000 7f2f91ffb640 1 -- 172.21.15.1:0/1101654088 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2f84097480 con 0x7f2f940c64b0 2024-05-09T05:06:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.712+0000 7f2f990fb640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2f800762d0 0x7f2f800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:15.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.713+0000 7f2f990fb640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2f800762d0 0x7f2f800787e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c034000 tx=0x7f2f8c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.717+0000 7f2f91ffb640 1 -- 172.21.15.1:0/1101654088 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2f84060dd0 con 0x7f2f940c64b0 2024-05-09T05:06:15.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.907+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f2f940b41e0 con 0x7f2f800762d0 2024-05-09T05:06:15.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.915+0000 7f2f91ffb640 1 -- 172.21.15.1:0/1101654088 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7f2f940b41e0 con 0x7f2f800762d0 2024-05-09T05:06:15.917 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:15.918 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2f800762d0 msgr2=0x7f2f800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2f800762d0 0x7f2f800787e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c034000 tx=0x7f2f8c02e000 comp rx=0 tx=0).stop 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 msgr2=0x7f2f94150340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94150340 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2f8400c700 tx=0x7f2f8400cbc0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 shutdown_connections 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2f800762d0 0x7f2f800787e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2f9406e600 0x7f2f941545e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2f940c64b0 0x7f2f94150340 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 --2- 172.21.15.1:0/1101654088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2f9406e1f0 0x7f2f94154ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:15.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 >> 172.21.15.1:0/1101654088 conn(0x7f2f940cc6a0 msgr2=0x7f2f940d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:15.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.917+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 shutdown_connections 2024-05-09T05:06:15.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:15.918+0000 7f2f9b35f640 1 -- 172.21.15.1:0/1101654088 wait complete. 2024-05-09T05:06:15.929 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-05-09T05:06:15.546213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239784,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734936,"statfs":{"total":773094113280,"available":772848574464,"internally_reserved":0,"allocated":2924544,"data_stored":1773788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002645"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737187+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:06:06.737187+0000","last_peered":"2024-05-09T05:06:06.737187+0000","last_clean":"2024-05-09T05:06:06.737187+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:06:06.737187+0000","last_undegraded":"2024-05-09T05:06:06.737187+0000","last_fullsized":"2024-05-09T05:06:06.737187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058883+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:06:06.058883+0000","last_peered":"2024-05-09T05:06:06.058883+0000","last_clean":"2024-05-09T05:06:06.058883+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:06:06.058883+0000","last_undegraded":"2024-05-09T05:06:06.058883+0000","last_fullsized":"2024-05-09T05:06:06.058883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736013+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:06:06.736013+0000","last_peered":"2024-05-09T05:06:06.736013+0000","last_clean":"2024-05-09T05:06:06.736013+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:06:06.736013+0000","last_undegraded":"2024-05-09T05:06:06.736013+0000","last_fullsized":"2024-05-09T05:06:06.736013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:48.770365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062625+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:06:06.062625+0000","last_peered":"2024-05-09T05:06:06.062625+0000","last_clean":"2024-05-09T05:06:06.062625+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:06:06.062625+0000","last_undegraded":"2024-05-09T05:06:06.062625+0000","last_fullsized":"2024-05-09T05:06:06.062625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737255+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:06:06.737255+0000","last_peered":"2024-05-09T05:06:06.737255+0000","last_clean":"2024-05-09T05:06:06.737255+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:06:06.737255+0000","last_undegraded":"2024-05-09T05:06:06.737255+0000","last_fullsized":"2024-05-09T05:06:06.737255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062081+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:06:06.062081+0000","last_peered":"2024-05-09T05:06:06.062081+0000","last_clean":"2024-05-09T05:06:06.062081+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:06:06.062081+0000","last_undegraded":"2024-05-09T05:06:06.062081+0000","last_fullsized":"2024-05-09T05:06:06.062081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736570+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:06:06.736570+0000","last_peered":"2024-05-09T05:06:06.736570+0000","last_clean":"2024-05-09T05:06:06.736570+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:06:06.736570+0000","last_undegraded":"2024-05-09T05:06:06.736570+0000","last_fullsized":"2024-05-09T05:06:06.736570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735634+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:06:06.735634+0000","last_peered":"2024-05-09T05:06:06.735634+0000","last_clean":"2024-05-09T05:06:06.735634+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:06:06.735634+0000","last_undegraded":"2024-05-09T05:06:06.735634+0000","last_fullsized":"2024-05-09T05:06:06.735634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737253+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:06:06.737253+0000","last_peered":"2024-05-09T05:06:06.737253+0000","last_clean":"2024-05-09T05:06:06.737253+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:06:06.737253+0000","last_undegraded":"2024-05-09T05:06:06.737253+0000","last_fullsized":"2024-05-09T05:06:06.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736506+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:06:06.736506+0000","last_peered":"2024-05-09T05:06:06.736506+0000","last_clean":"2024-05-09T05:06:06.736506+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:06:06.736506+0000","last_undegraded":"2024-05-09T05:06:06.736506+0000","last_fullsized":"2024-05-09T05:06:06.736506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736352+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:06:06.736352+0000","last_peered":"2024-05-09T05:06:06.736352+0000","last_clean":"2024-05-09T05:06:06.736352+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:06:06.736352+0000","last_undegraded":"2024-05-09T05:06:06.736352+0000","last_fullsized":"2024-05-09T05:06:06.736352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722788+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:06:06.722788+0000","last_peered":"2024-05-09T05:06:06.722788+0000","last_clean":"2024-05-09T05:06:06.722788+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:06:06.722788+0000","last_undegraded":"2024-05-09T05:06:06.722788+0000","last_fullsized":"2024-05-09T05:06:06.722788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736075+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:06:06.736075+0000","last_peered":"2024-05-09T05:06:06.736075+0000","last_clean":"2024-05-09T05:06:06.736075+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:06:06.736075+0000","last_undegraded":"2024-05-09T05:06:06.736075+0000","last_fullsized":"2024-05-09T05:06:06.736075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736071+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:06:06.736071+0000","last_peered":"2024-05-09T05:06:06.736071+0000","last_clean":"2024-05-09T05:06:06.736071+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:06:06.736071+0000","last_undegraded":"2024-05-09T05:06:06.736071+0000","last_fullsized":"2024-05-09T05:06:06.736071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:37.258923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058421+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:06:06.058421+0000","last_peered":"2024-05-09T05:06:06.058421+0000","last_clean":"2024-05-09T05:06:06.058421+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:06:06.058421+0000","last_undegraded":"2024-05-09T05:06:06.058421+0000","last_fullsized":"2024-05-09T05:06:06.058421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062588+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:06:06.062588+0000","last_peered":"2024-05-09T05:06:06.062588+0000","last_clean":"2024-05-09T05:06:06.062588+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:06:06.062588+0000","last_undegraded":"2024-05-09T05:06:06.062588+0000","last_fullsized":"2024-05-09T05:06:06.062588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062864+0000","last_change":"2024-05-09T05:06:05.071898+0000","last_active":"2024-05-09T05:06:06.062864+0000","last_peered":"2024-05-09T05:06:06.062864+0000","last_clean":"2024-05-09T05:06:06.062864+0000","last_became_active":"2024-05-09T05:06:05.071618+0000","last_became_peered":"2024-05-09T05:06:05.071618+0000","last_unstale":"2024-05-09T05:06:06.062864+0000","last_undegraded":"2024-05-09T05:06:06.062864+0000","last_fullsized":"2024-05-09T05:06:06.062864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060053+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:06:06.060053+0000","last_peered":"2024-05-09T05:06:06.060053+0000","last_clean":"2024-05-09T05:06:06.060053+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:06:06.060053+0000","last_undegraded":"2024-05-09T05:06:06.060053+0000","last_fullsized":"2024-05-09T05:06:06.060053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737104+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:06:06.737104+0000","last_peered":"2024-05-09T05:06:06.737104+0000","last_clean":"2024-05-09T05:06:06.737104+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:06:06.737104+0000","last_undegraded":"2024-05-09T05:06:06.737104+0000","last_fullsized":"2024-05-09T05:06:06.737104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736090+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:06:06.736090+0000","last_peered":"2024-05-09T05:06:06.736090+0000","last_clean":"2024-05-09T05:06:06.736090+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:06:06.736090+0000","last_undegraded":"2024-05-09T05:06:06.736090+0000","last_fullsized":"2024-05-09T05:06:06.736090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736595+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:06:06.736595+0000","last_peered":"2024-05-09T05:06:06.736595+0000","last_clean":"2024-05-09T05:06:06.736595+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:06:06.736595+0000","last_undegraded":"2024-05-09T05:06:06.736595+0000","last_fullsized":"2024-05-09T05:06:06.736595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:02.665344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062276+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:06:06.062276+0000","last_peered":"2024-05-09T05:06:06.062276+0000","last_clean":"2024-05-09T05:06:06.062276+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:06:06.062276+0000","last_undegraded":"2024-05-09T05:06:06.062276+0000","last_fullsized":"2024-05-09T05:06:06.062276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735814+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:06:06.735814+0000","last_peered":"2024-05-09T05:06:06.735814+0000","last_clean":"2024-05-09T05:06:06.735814+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:06:06.735814+0000","last_undegraded":"2024-05-09T05:06:06.735814+0000","last_fullsized":"2024-05-09T05:06:06.735814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062254+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062254+0000","last_peered":"2024-05-09T05:06:06.062254+0000","last_clean":"2024-05-09T05:06:06.062254+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:06:06.062254+0000","last_undegraded":"2024-05-09T05:06:06.062254+0000","last_fullsized":"2024-05-09T05:06:06.062254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736447+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:06:06.736447+0000","last_peered":"2024-05-09T05:06:06.736447+0000","last_clean":"2024-05-09T05:06:06.736447+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:06:06.736447+0000","last_undegraded":"2024-05-09T05:06:06.736447+0000","last_fullsized":"2024-05-09T05:06:06.736447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:25:28.166958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736699+0000","last_change":"2024-05-09T05:06:05.067878+0000","last_active":"2024-05-09T05:06:06.736699+0000","last_peered":"2024-05-09T05:06:06.736699+0000","last_clean":"2024-05-09T05:06:06.736699+0000","last_became_active":"2024-05-09T05:06:05.067426+0000","last_became_peered":"2024-05-09T05:06:05.067426+0000","last_unstale":"2024-05-09T05:06:06.736699+0000","last_undegraded":"2024-05-09T05:06:06.736699+0000","last_fullsized":"2024-05-09T05:06:06.736699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058528+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:06:06.058528+0000","last_peered":"2024-05-09T05:06:06.058528+0000","last_clean":"2024-05-09T05:06:06.058528+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:06:06.058528+0000","last_undegraded":"2024-05-09T05:06:06.058528+0000","last_fullsized":"2024-05-09T05:06:06.058528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736532+0000","last_change":"2024-05-09T05:06:05.063359+0000","last_active":"2024-05-09T05:06:06.736532+0000","last_peered":"2024-05-09T05:06:06.736532+0000","last_clean":"2024-05-09T05:06:06.736532+0000","last_became_active":"2024-05-09T05:06:05.063026+0000","last_became_peered":"2024-05-09T05:06:05.063026+0000","last_unstale":"2024-05-09T05:06:06.736532+0000","last_undegraded":"2024-05-09T05:06:06.736532+0000","last_fullsized":"2024-05-09T05:06:06.736532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736550+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:06:06.736550+0000","last_peered":"2024-05-09T05:06:06.736550+0000","last_clean":"2024-05-09T05:06:06.736550+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:06:06.736550+0000","last_undegraded":"2024-05-09T05:06:06.736550+0000","last_fullsized":"2024-05-09T05:06:06.736550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062682+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:06:06.062682+0000","last_peered":"2024-05-09T05:06:06.062682+0000","last_clean":"2024-05-09T05:06:06.062682+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:06:06.062682+0000","last_undegraded":"2024-05-09T05:06:06.062682+0000","last_fullsized":"2024-05-09T05:06:06.062682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059766+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:06:06.059766+0000","last_peered":"2024-05-09T05:06:06.059766+0000","last_clean":"2024-05-09T05:06:06.059766+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:06:06.059766+0000","last_undegraded":"2024-05-09T05:06:06.059766+0000","last_fullsized":"2024-05-09T05:06:06.059766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736152+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:06:06.736152+0000","last_peered":"2024-05-09T05:06:06.736152+0000","last_clean":"2024-05-09T05:06:06.736152+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:06:06.736152+0000","last_undegraded":"2024-05-09T05:06:06.736152+0000","last_fullsized":"2024-05-09T05:06:06.736152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722480+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:06:06.722480+0000","last_peered":"2024-05-09T05:06:06.722480+0000","last_clean":"2024-05-09T05:06:06.722480+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:06:06.722480+0000","last_undegraded":"2024-05-09T05:06:06.722480+0000","last_fullsized":"2024-05-09T05:06:06.722480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736730+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:06:06.736730+0000","last_peered":"2024-05-09T05:06:06.736730+0000","last_clean":"2024-05-09T05:06:06.736730+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:06:06.736730+0000","last_undegraded":"2024-05-09T05:06:06.736730+0000","last_fullsized":"2024-05-09T05:06:06.736730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736786+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:06:06.736786+0000","last_peered":"2024-05-09T05:06:06.736786+0000","last_clean":"2024-05-09T05:06:06.736786+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:06:06.736786+0000","last_undegraded":"2024-05-09T05:06:06.736786+0000","last_fullsized":"2024-05-09T05:06:06.736786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736064+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:06:06.736064+0000","last_peered":"2024-05-09T05:06:06.736064+0000","last_clean":"2024-05-09T05:06:06.736064+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:06:06.736064+0000","last_undegraded":"2024-05-09T05:06:06.736064+0000","last_fullsized":"2024-05-09T05:06:06.736064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736318+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:06:06.736318+0000","last_peered":"2024-05-09T05:06:06.736318+0000","last_clean":"2024-05-09T05:06:06.736318+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:06:06.736318+0000","last_undegraded":"2024-05-09T05:06:06.736318+0000","last_fullsized":"2024-05-09T05:06:06.736318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735754+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:06:06.735754+0000","last_peered":"2024-05-09T05:06:06.735754+0000","last_clean":"2024-05-09T05:06:06.735754+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:06:06.735754+0000","last_undegraded":"2024-05-09T05:06:06.735754+0000","last_fullsized":"2024-05-09T05:06:06.735754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062362+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:06:06.062362+0000","last_peered":"2024-05-09T05:06:06.062362+0000","last_clean":"2024-05-09T05:06:06.062362+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:06:06.062362+0000","last_undegraded":"2024-05-09T05:06:06.062362+0000","last_fullsized":"2024-05-09T05:06:06.062362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058418+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:06:06.058418+0000","last_peered":"2024-05-09T05:06:06.058418+0000","last_clean":"2024-05-09T05:06:06.058418+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:06:06.058418+0000","last_undegraded":"2024-05-09T05:06:06.058418+0000","last_fullsized":"2024-05-09T05:06:06.058418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736176+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:06:06.736176+0000","last_peered":"2024-05-09T05:06:06.736176+0000","last_clean":"2024-05-09T05:06:06.736176+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:06:06.736176+0000","last_undegraded":"2024-05-09T05:06:06.736176+0000","last_fullsized":"2024-05-09T05:06:06.736176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062383+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:06:06.062383+0000","last_peered":"2024-05-09T05:06:06.062383+0000","last_clean":"2024-05-09T05:06:06.062383+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:06:06.062383+0000","last_undegraded":"2024-05-09T05:06:06.062383+0000","last_fullsized":"2024-05-09T05:06:06.062383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062550+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:06:06.062550+0000","last_peered":"2024-05-09T05:06:06.062550+0000","last_clean":"2024-05-09T05:06:06.062550+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:06:06.062550+0000","last_undegraded":"2024-05-09T05:06:06.062550+0000","last_fullsized":"2024-05-09T05:06:06.062550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736605+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:06:06.736605+0000","last_peered":"2024-05-09T05:06:06.736605+0000","last_clean":"2024-05-09T05:06:06.736605+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:06:06.736605+0000","last_undegraded":"2024-05-09T05:06:06.736605+0000","last_fullsized":"2024-05-09T05:06:06.736605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:26.607004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053831+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:06:10.053831+0000","last_peered":"2024-05-09T05:06:10.053831+0000","last_clean":"2024-05-09T05:06:10.053831+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:06:10.053831+0000","last_undegraded":"2024-05-09T05:06:10.053831+0000","last_fullsized":"2024-05-09T05:06:10.053831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722806+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:06:06.722806+0000","last_peered":"2024-05-09T05:06:06.722806+0000","last_clean":"2024-05-09T05:06:06.722806+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:06:06.722806+0000","last_undegraded":"2024-05-09T05:06:06.722806+0000","last_fullsized":"2024-05-09T05:06:06.722806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.055062+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:06:10.055062+0000","last_peered":"2024-05-09T05:06:10.055062+0000","last_clean":"2024-05-09T05:06:10.055062+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:06:10.055062+0000","last_undegraded":"2024-05-09T05:06:10.055062+0000","last_fullsized":"2024-05-09T05:06:10.055062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736736+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:06:06.736736+0000","last_peered":"2024-05-09T05:06:06.736736+0000","last_clean":"2024-05-09T05:06:06.736736+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:06:06.736736+0000","last_undegraded":"2024-05-09T05:06:06.736736+0000","last_fullsized":"2024-05-09T05:06:06.736736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053672+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:06:10.053672+0000","last_peered":"2024-05-09T05:06:10.053672+0000","last_clean":"2024-05-09T05:06:10.053672+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:06:10.053672+0000","last_undegraded":"2024-05-09T05:06:10.053672+0000","last_fullsized":"2024-05-09T05:06:10.053672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736396+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:06:06.736396+0000","last_peered":"2024-05-09T05:06:06.736396+0000","last_clean":"2024-05-09T05:06:06.736396+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:06:06.736396+0000","last_undegraded":"2024-05-09T05:06:06.736396+0000","last_fullsized":"2024-05-09T05:06:06.736396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:41.283927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062613+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:06:06.062613+0000","last_peered":"2024-05-09T05:06:06.062613+0000","last_clean":"2024-05-09T05:06:06.062613+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:06:06.062613+0000","last_undegraded":"2024-05-09T05:06:06.062613+0000","last_fullsized":"2024-05-09T05:06:06.062613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737118+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:06:06.737118+0000","last_peered":"2024-05-09T05:06:06.737118+0000","last_clean":"2024-05-09T05:06:06.737118+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:06:06.737118+0000","last_undegraded":"2024-05-09T05:06:06.737118+0000","last_fullsized":"2024-05-09T05:06:06.737118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736071+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:06:06.736071+0000","last_peered":"2024-05-09T05:06:06.736071+0000","last_clean":"2024-05-09T05:06:06.736071+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:06:06.736071+0000","last_undegraded":"2024-05-09T05:06:06.736071+0000","last_fullsized":"2024-05-09T05:06:06.736071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736621+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:06:06.736621+0000","last_peered":"2024-05-09T05:06:06.736621+0000","last_clean":"2024-05-09T05:06:06.736621+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:06:06.736621+0000","last_undegraded":"2024-05-09T05:06:06.736621+0000","last_fullsized":"2024-05-09T05:06:06.736621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.056327+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:06:10.056327+0000","last_peered":"2024-05-09T05:06:10.056327+0000","last_clean":"2024-05-09T05:06:10.056327+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:06:10.056327+0000","last_undegraded":"2024-05-09T05:06:10.056327+0000","last_fullsized":"2024-05-09T05:06:10.056327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736921+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:06:06.736921+0000","last_peered":"2024-05-09T05:06:06.736921+0000","last_clean":"2024-05-09T05:06:06.736921+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:06:06.736921+0000","last_undegraded":"2024-05-09T05:06:06.736921+0000","last_fullsized":"2024-05-09T05:06:06.736921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722789+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:06:06.722789+0000","last_peered":"2024-05-09T05:06:06.722789+0000","last_clean":"2024-05-09T05:06:06.722789+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:06:06.722789+0000","last_undegraded":"2024-05-09T05:06:06.722789+0000","last_fullsized":"2024-05-09T05:06:06.722789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737109+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:06:06.737109+0000","last_peered":"2024-05-09T05:06:06.737109+0000","last_clean":"2024-05-09T05:06:06.737109+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:06:06.737109+0000","last_undegraded":"2024-05-09T05:06:06.737109+0000","last_fullsized":"2024-05-09T05:06:06.737109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053514+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:06:10.053514+0000","last_peered":"2024-05-09T05:06:10.053514+0000","last_clean":"2024-05-09T05:06:10.053514+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:06:10.053514+0000","last_undegraded":"2024-05-09T05:06:10.053514+0000","last_fullsized":"2024-05-09T05:06:10.053514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737391+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:06:06.737391+0000","last_peered":"2024-05-09T05:06:06.737391+0000","last_clean":"2024-05-09T05:06:06.737391+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:06:06.737391+0000","last_undegraded":"2024-05-09T05:06:06.737391+0000","last_fullsized":"2024-05-09T05:06:06.737391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.555563+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:06:06.555563+0000","last_peered":"2024-05-09T05:06:06.555563+0000","last_clean":"2024-05-09T05:06:06.555563+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:06:06.555563+0000","last_undegraded":"2024-05-09T05:06:06.555563+0000","last_fullsized":"2024-05-09T05:06:06.555563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:06:06.062865+0000","last_peered":"2024-05-09T05:06:06.062865+0000","last_clean":"2024-05-09T05:06:06.062865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:06:06.062865+0000","last_undegraded":"2024-05-09T05:06:06.062865+0000","last_fullsized":"2024-05-09T05:06:06.062865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060038+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:06:06.060038+0000","last_peered":"2024-05-09T05:06:06.060038+0000","last_clean":"2024-05-09T05:06:06.060038+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:06:06.060038+0000","last_undegraded":"2024-05-09T05:06:06.060038+0000","last_fullsized":"2024-05-09T05:06:06.060038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736885+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:06:06.736885+0000","last_peered":"2024-05-09T05:06:06.736885+0000","last_clean":"2024-05-09T05:06:06.736885+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:06:06.736885+0000","last_undegraded":"2024-05-09T05:06:06.736885+0000","last_fullsized":"2024-05-09T05:06:06.736885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736326+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:06:06.736326+0000","last_peered":"2024-05-09T05:06:06.736326+0000","last_clean":"2024-05-09T05:06:06.736326+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:06:06.736326+0000","last_undegraded":"2024-05-09T05:06:06.736326+0000","last_fullsized":"2024-05-09T05:06:06.736326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062158+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:06:06.062158+0000","last_peered":"2024-05-09T05:06:06.062158+0000","last_clean":"2024-05-09T05:06:06.062158+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:06:06.062158+0000","last_undegraded":"2024-05-09T05:06:06.062158+0000","last_fullsized":"2024-05-09T05:06:06.062158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735899+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:06:06.735899+0000","last_peered":"2024-05-09T05:06:06.735899+0000","last_clean":"2024-05-09T05:06:06.735899+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:06:06.735899+0000","last_undegraded":"2024-05-09T05:06:06.735899+0000","last_fullsized":"2024-05-09T05:06:06.735899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736248+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:06:06.736248+0000","last_peered":"2024-05-09T05:06:06.736248+0000","last_clean":"2024-05-09T05:06:06.736248+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:06:06.736248+0000","last_undegraded":"2024-05-09T05:06:06.736248+0000","last_fullsized":"2024-05-09T05:06:06.736248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736254+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:06:06.736254+0000","last_peered":"2024-05-09T05:06:06.736254+0000","last_clean":"2024-05-09T05:06:06.736254+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:06:06.736254+0000","last_undegraded":"2024-05-09T05:06:06.736254+0000","last_fullsized":"2024-05-09T05:06:06.736254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:23.362012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059887+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:06:06.059887+0000","last_peered":"2024-05-09T05:06:06.059887+0000","last_clean":"2024-05-09T05:06:06.059887+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:06:06.059887+0000","last_undegraded":"2024-05-09T05:06:06.059887+0000","last_fullsized":"2024-05-09T05:06:06.059887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058422+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:06:06.058422+0000","last_peered":"2024-05-09T05:06:06.058422+0000","last_clean":"2024-05-09T05:06:06.058422+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:06:06.058422+0000","last_undegraded":"2024-05-09T05:06:06.058422+0000","last_fullsized":"2024-05-09T05:06:06.058422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062842+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:06:06.062842+0000","last_peered":"2024-05-09T05:06:06.062842+0000","last_clean":"2024-05-09T05:06:06.062842+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:06:06.062842+0000","last_undegraded":"2024-05-09T05:06:06.062842+0000","last_fullsized":"2024-05-09T05:06:06.062842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722592+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:06:06.722592+0000","last_peered":"2024-05-09T05:06:06.722592+0000","last_clean":"2024-05-09T05:06:06.722592+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:06:06.722592+0000","last_undegraded":"2024-05-09T05:06:06.722592+0000","last_fullsized":"2024-05-09T05:06:06.722592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058680+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:06:06.058680+0000","last_peered":"2024-05-09T05:06:06.058680+0000","last_clean":"2024-05-09T05:06:06.058680+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:06:06.058680+0000","last_undegraded":"2024-05-09T05:06:06.058680+0000","last_fullsized":"2024-05-09T05:06:06.058680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062852+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:06:06.062852+0000","last_peered":"2024-05-09T05:06:06.062852+0000","last_clean":"2024-05-09T05:06:06.062852+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:06:06.062852+0000","last_undegraded":"2024-05-09T05:06:06.062852+0000","last_fullsized":"2024-05-09T05:06:06.062852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737087+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:06:06.737087+0000","last_peered":"2024-05-09T05:06:06.737087+0000","last_clean":"2024-05-09T05:06:06.737087+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:06:06.737087+0000","last_undegraded":"2024-05-09T05:06:06.737087+0000","last_fullsized":"2024-05-09T05:06:06.737087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058622+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:06:06.058622+0000","last_peered":"2024-05-09T05:06:06.058622+0000","last_clean":"2024-05-09T05:06:06.058622+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:06:06.058622+0000","last_undegraded":"2024-05-09T05:06:06.058622+0000","last_fullsized":"2024-05-09T05:06:06.058622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058810+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:06:06.058810+0000","last_peered":"2024-05-09T05:06:06.058810+0000","last_clean":"2024-05-09T05:06:06.058810+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:06:06.058810+0000","last_undegraded":"2024-05-09T05:06:06.058810+0000","last_fullsized":"2024-05-09T05:06:06.058810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736837+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:06:06.736837+0000","last_peered":"2024-05-09T05:06:06.736837+0000","last_clean":"2024-05-09T05:06:06.736837+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:06:06.736837+0000","last_undegraded":"2024-05-09T05:06:06.736837+0000","last_fullsized":"2024-05-09T05:06:06.736837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:33.840344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053220+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:06:10.053220+0000","last_peered":"2024-05-09T05:06:10.053220+0000","last_clean":"2024-05-09T05:06:10.053220+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:06:10.053220+0000","last_undegraded":"2024-05-09T05:06:10.053220+0000","last_fullsized":"2024-05-09T05:06:10.053220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736183+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:06:06.736183+0000","last_peered":"2024-05-09T05:06:06.736183+0000","last_clean":"2024-05-09T05:06:06.736183+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:06:06.736183+0000","last_undegraded":"2024-05-09T05:06:06.736183+0000","last_fullsized":"2024-05-09T05:06:06.736183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:50.236344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736055+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:06:06.736055+0000","last_peered":"2024-05-09T05:06:06.736055+0000","last_clean":"2024-05-09T05:06:06.736055+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:06:06.736055+0000","last_undegraded":"2024-05-09T05:06:06.736055+0000","last_fullsized":"2024-05-09T05:06:06.736055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722580+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:06:06.722580+0000","last_peered":"2024-05-09T05:06:06.722580+0000","last_clean":"2024-05-09T05:06:06.722580+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:06:06.722580+0000","last_undegraded":"2024-05-09T05:06:06.722580+0000","last_fullsized":"2024-05-09T05:06:06.722580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058525+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:06:06.058525+0000","last_peered":"2024-05-09T05:06:06.058525+0000","last_clean":"2024-05-09T05:06:06.058525+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:06:06.058525+0000","last_undegraded":"2024-05-09T05:06:06.058525+0000","last_fullsized":"2024-05-09T05:06:06.058525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736386+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:06:06.736386+0000","last_peered":"2024-05-09T05:06:06.736386+0000","last_clean":"2024-05-09T05:06:06.736386+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:06:06.736386+0000","last_undegraded":"2024-05-09T05:06:06.736386+0000","last_fullsized":"2024-05-09T05:06:06.736386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058700+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:06:06.058700+0000","last_peered":"2024-05-09T05:06:06.058700+0000","last_clean":"2024-05-09T05:06:06.058700+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:06:06.058700+0000","last_undegraded":"2024-05-09T05:06:06.058700+0000","last_fullsized":"2024-05-09T05:06:06.058700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736096+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:06:06.736096+0000","last_peered":"2024-05-09T05:06:06.736096+0000","last_clean":"2024-05-09T05:06:06.736096+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:06:06.736096+0000","last_undegraded":"2024-05-09T05:06:06.736096+0000","last_fullsized":"2024-05-09T05:06:06.736096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:58:35.651549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053430+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:06:10.053430+0000","last_peered":"2024-05-09T05:06:10.053430+0000","last_clean":"2024-05-09T05:06:10.053430+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:06:10.053430+0000","last_undegraded":"2024-05-09T05:06:10.053430+0000","last_fullsized":"2024-05-09T05:06:10.053430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736285+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:06:06.736285+0000","last_peered":"2024-05-09T05:06:06.736285+0000","last_clean":"2024-05-09T05:06:06.736285+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:06:06.736285+0000","last_undegraded":"2024-05-09T05:06:06.736285+0000","last_fullsized":"2024-05-09T05:06:06.736285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062736+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062736+0000","last_peered":"2024-05-09T05:06:06.062736+0000","last_clean":"2024-05-09T05:06:06.062736+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:06:06.062736+0000","last_undegraded":"2024-05-09T05:06:06.062736+0000","last_fullsized":"2024-05-09T05:06:06.062736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053716+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:06:10.053716+0000","last_peered":"2024-05-09T05:06:10.053716+0000","last_clean":"2024-05-09T05:06:10.053716+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:06:10.053716+0000","last_undegraded":"2024-05-09T05:06:10.053716+0000","last_fullsized":"2024-05-09T05:06:10.053716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062769+0000","last_change":"2024-05-09T05:06:05.063337+0000","last_active":"2024-05-09T05:06:06.062769+0000","last_peered":"2024-05-09T05:06:06.062769+0000","last_clean":"2024-05-09T05:06:06.062769+0000","last_became_active":"2024-05-09T05:06:05.063016+0000","last_became_peered":"2024-05-09T05:06:05.063016+0000","last_unstale":"2024-05-09T05:06:06.062769+0000","last_undegraded":"2024-05-09T05:06:06.062769+0000","last_fullsized":"2024-05-09T05:06:06.062769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.052083+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:06:10.052083+0000","last_peered":"2024-05-09T05:06:10.052083+0000","last_clean":"2024-05-09T05:06:10.052083+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:06:10.052083+0000","last_undegraded":"2024-05-09T05:06:10.052083+0000","last_fullsized":"2024-05-09T05:06:10.052083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736394+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:06:06.736394+0000","last_peered":"2024-05-09T05:06:06.736394+0000","last_clean":"2024-05-09T05:06:06.736394+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:06:06.736394+0000","last_undegraded":"2024-05-09T05:06:06.736394+0000","last_fullsized":"2024-05-09T05:06:06.736394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:42.457145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736800+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:06:06.736800+0000","last_peered":"2024-05-09T05:06:06.736800+0000","last_clean":"2024-05-09T05:06:06.736800+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:06:06.736800+0000","last_undegraded":"2024-05-09T05:06:06.736800+0000","last_fullsized":"2024-05-09T05:06:06.736800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736091+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:06:06.736091+0000","last_peered":"2024-05-09T05:06:06.736091+0000","last_clean":"2024-05-09T05:06:06.736091+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:06:06.736091+0000","last_undegraded":"2024-05-09T05:06:06.736091+0000","last_fullsized":"2024-05-09T05:06:06.736091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:32:29.331565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.054793+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:06:10.054793+0000","last_peered":"2024-05-09T05:06:10.054793+0000","last_clean":"2024-05-09T05:06:10.054793+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:06:10.054793+0000","last_undegraded":"2024-05-09T05:06:10.054793+0000","last_fullsized":"2024-05-09T05:06:10.054793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":806912,"data_stored":664479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":73619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":73655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":221184,"data_stored":75141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":208896,"data_stored":73529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":217088,"data_stored":73689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36132,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335708,"statfs":{"total":96636764160,"available":96599764992,"internally_reserved":0,"allocated":815104,"data_stored":664535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:06:15.933 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:06:16.030 INFO:tasks.ceph.mds.f.smithi196.stderr:2024-05-09T05:06:16.029+0000 7f147bbd9640 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T05:06:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.083+0000 7fdcf2a8b640 1 Processor -- start 2024-05-09T05:06:16.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.083+0000 7fdcf2a8b640 1 -- start start 2024-05-09T05:06:16.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf2a8b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdcec06efa0 0x7fdcec156900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf2a8b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fdcec0cc560 0x7fdcec0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf2a8b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdcec06de20 con 0x7fdcec15fdf0 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf2a8b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdcec15c9a0 con 0x7fdcec06efa0 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf2a8b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fdcec0b51a0 con 0x7fdcec0cc560 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf0827640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdcec06efa0 0x7fdcec156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf1028640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fdcec15fdf0 0x7fdcec160200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35636/0 (socket says 172.21.15.1:35636) 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf1028640 1 -- 172.21.15.1:0/507583564 learned_addr learned my addr 172.21.15.1:0/507583564 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcebfff640 1 --2- 172.21.15.1:0/507583564 >> v2:172.21.15.196:3300/0 conn(0x7fdcec0cc560 0x7fdcec0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf0827640 1 -- 172.21.15.1:0/507583564 >> v2:172.21.15.196:3300/0 conn(0x7fdcec0cc560 msgr2=0x7fdcec0cc970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf0827640 1 --2- 172.21.15.1:0/507583564 >> v2:172.21.15.196:3300/0 conn(0x7fdcec0cc560 0x7fdcec0cc970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf0827640 1 -- 172.21.15.1:0/507583564 >> v1:172.21.15.1:6789/0 conn(0x7fdcec15fdf0 legacy=0x7fdcec160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdcf0827640 1 -- 172.21.15.1:0/507583564 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcec15d720 con 0x7fdcec06efa0 2024-05-09T05:06:16.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.085+0000 7fdceb7fe640 1 -- 172.21.15.1:0/507583564 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4031727248 0 0) 0x7fdcec06de20 con 0x7fdcec15fdf0 2024-05-09T05:06:16.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.086+0000 7fdcf0827640 1 --2- 172.21.15.1:0/507583564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec06efa0 0x7fdcec156900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdcec08ef30 tx=0x7fdce402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4c82fcb52560c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:16.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.086+0000 7fdceb7fe640 1 -- 172.21.15.1:0/507583564 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce4037070 con 0x7fdcec06efa0 2024-05-09T05:06:16.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.086+0000 7fdceb7fe640 1 -- 172.21.15.1:0/507583564 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdce4032440 con 0x7fdcec06efa0 2024-05-09T05:06:16.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.086+0000 7fdceb7fe640 1 -- 172.21.15.1:0/507583564 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdce4032810 con 0x7fdcec06efa0 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.087+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/507583564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec06efa0 msgr2=0x7fdcec156900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.087+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/507583564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec06efa0 0x7fdcec156900 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fdcec08ef30 tx=0x7fdce402a230 comp rx=0 tx=0).stop 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.087+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/507583564 shutdown_connections 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.087+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/507583564 >> v2:172.21.15.196:3300/0 conn(0x7fdcec0cc560 0x7fdcec0cc970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.087+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/507583564 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec06efa0 0x7fdcec156900 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.087+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/507583564 >> 172.21.15.1:0/507583564 conn(0x7fdcec068d90 msgr2=0x7fdcec0d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.087+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/507583564 shutdown_connections 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.087+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/507583564 wait complete. 2024-05-09T05:06:16.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.088+0000 7fdcf2a8b640 1 Processor -- start 2024-05-09T05:06:16.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.088+0000 7fdcf2a8b640 1 -- start start 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf2a8b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdcec06efa0 0x7fdcec1f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf2a8b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec0cc560 0x7fdcec1538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf2a8b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdcec15fdf0 0x7fdcec154d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf2a8b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdcec15cf20 con 0x7fdcec0cc560 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf2a8b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdcec15cc40 con 0x7fdcec06efa0 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf2a8b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdcec15d200 con 0x7fdcec15fdf0 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcebfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec0cc560 0x7fdcec1538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcebfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec0cc560 0x7fdcec1538a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49586/0 (socket says 172.21.15.1:49586) 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcebfff640 1 -- 172.21.15.1:0/3468907128 learned_addr learned my addr 172.21.15.1:0/3468907128 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf0827640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdcec06efa0 0x7fdcec1f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf1028640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdcec15fdf0 0x7fdcec154d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcebfff640 1 -- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdcec15fdf0 msgr2=0x7fdcec154d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcebfff640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdcec15fdf0 0x7fdcec154d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcebfff640 1 -- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdcec06efa0 msgr2=0x7fdcec1f6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcebfff640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdcec06efa0 0x7fdcec1f6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcebfff640 1 -- 172.21.15.1:0/3468907128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdcec1f73b0 con 0x7fdcec0cc560 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.090+0000 7fdcf1028640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdcec15fdf0 0x7fdcec154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.091+0000 7fdcebfff640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec0cc560 0x7fdcec1538a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400e870 tx=0x7fdcd400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.091+0000 7fdce97fa640 1 -- 172.21.15.1:0/3468907128 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd4017070 con 0x7fdcec0cc560 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.091+0000 7fdce97fa640 1 -- 172.21.15.1:0/3468907128 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdcd400ee60 con 0x7fdcec0cc560 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.091+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdcec0cae20 con 0x7fdcec0cc560 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.091+0000 7fdce97fa640 1 -- 172.21.15.1:0/3468907128 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdcd400cb60 con 0x7fdcec0cc560 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.091+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdcec1f3960 con 0x7fdcec0cc560 2024-05-09T05:06:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.092+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdcec1f36d0 con 0x7fdcec0cc560 2024-05-09T05:06:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.092+0000 7fdce97fa640 1 -- 172.21.15.1:0/3468907128 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdcd401a070 con 0x7fdcec0cc560 2024-05-09T05:06:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.093+0000 7fdce97fa640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdccc0762d0 0x7fdccc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.093+0000 7fdce97fa640 1 -- 172.21.15.1:0/3468907128 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fdcd40973a0 con 0x7fdcec0cc560 2024-05-09T05:06:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.093+0000 7fdcf0827640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdccc0762d0 0x7fdccc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.094+0000 7fdcf0827640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdccc0762d0 0x7fdccc0787e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdce4034000 tx=0x7fdce402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.098+0000 7fdce97fa640 1 -- 172.21.15.1:0/3468907128 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdcd40606b0 con 0x7fdcec0cc560 2024-05-09T05:06:16.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.287+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fdcec050870 con 0x7fdccc0762d0 2024-05-09T05:06:16.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.294+0000 7fdce97fa640 1 -- 172.21.15.1:0/3468907128 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+251825 (secure 0 0 0) 0x7fdcec050870 con 0x7fdccc0762d0 2024-05-09T05:06:16.297 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:16.298 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:06:16.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.296+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdccc0762d0 msgr2=0x7fdccc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.296+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdccc0762d0 0x7fdccc0787e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fdce4034000 tx=0x7fdce402e000 comp rx=0 tx=0).stop 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.296+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec0cc560 msgr2=0x7fdcec1538a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.296+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec0cc560 0x7fdcec1538a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fdcd400e870 tx=0x7fdcd400ed30 comp rx=0 tx=0).stop 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.297+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 shutdown_connections 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.297+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdccc0762d0 0x7fdccc0787e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.297+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdcec15fdf0 0x7fdcec154d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.297+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdcec0cc560 0x7fdcec1538a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.297+0000 7fdcf2a8b640 1 --2- 172.21.15.1:0/3468907128 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdcec06efa0 0x7fdcec1f6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.297+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 >> 172.21.15.1:0/3468907128 conn(0x7fdcec068d90 msgr2=0x7fdcec15d9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.297+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 shutdown_connections 2024-05-09T05:06:16.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.297+0000 7fdcf2a8b640 1 -- 172.21.15.1:0/3468907128 wait complete. 2024-05-09T05:06:16.308 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":26,"stamp":"2024-05-09T05:06:15.546213+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":592778,"num_objects":23,"num_object_clones":0,"num_object_copies":46,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":23,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":145,"num_write_kb":1642,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":110,"ondisk_log_size":110,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239784,"kb_used_data":2856,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734936,"statfs":{"total":773094113280,"available":772848574464,"internally_reserved":0,"allocated":2924544,"data_stored":1773788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002645"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737187+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:06:06.737187+0000","last_peered":"2024-05-09T05:06:06.737187+0000","last_clean":"2024-05-09T05:06:06.737187+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:06:06.737187+0000","last_undegraded":"2024-05-09T05:06:06.737187+0000","last_fullsized":"2024-05-09T05:06:06.737187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058883+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:06:06.058883+0000","last_peered":"2024-05-09T05:06:06.058883+0000","last_clean":"2024-05-09T05:06:06.058883+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:06:06.058883+0000","last_undegraded":"2024-05-09T05:06:06.058883+0000","last_fullsized":"2024-05-09T05:06:06.058883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736013+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:06:06.736013+0000","last_peered":"2024-05-09T05:06:06.736013+0000","last_clean":"2024-05-09T05:06:06.736013+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:06:06.736013+0000","last_undegraded":"2024-05-09T05:06:06.736013+0000","last_fullsized":"2024-05-09T05:06:06.736013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:48.770365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062625+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:06:06.062625+0000","last_peered":"2024-05-09T05:06:06.062625+0000","last_clean":"2024-05-09T05:06:06.062625+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:06:06.062625+0000","last_undegraded":"2024-05-09T05:06:06.062625+0000","last_fullsized":"2024-05-09T05:06:06.062625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737255+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:06:06.737255+0000","last_peered":"2024-05-09T05:06:06.737255+0000","last_clean":"2024-05-09T05:06:06.737255+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:06:06.737255+0000","last_undegraded":"2024-05-09T05:06:06.737255+0000","last_fullsized":"2024-05-09T05:06:06.737255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062081+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:06:06.062081+0000","last_peered":"2024-05-09T05:06:06.062081+0000","last_clean":"2024-05-09T05:06:06.062081+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:06:06.062081+0000","last_undegraded":"2024-05-09T05:06:06.062081+0000","last_fullsized":"2024-05-09T05:06:06.062081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736570+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:06:06.736570+0000","last_peered":"2024-05-09T05:06:06.736570+0000","last_clean":"2024-05-09T05:06:06.736570+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:06:06.736570+0000","last_undegraded":"2024-05-09T05:06:06.736570+0000","last_fullsized":"2024-05-09T05:06:06.736570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735634+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:06:06.735634+0000","last_peered":"2024-05-09T05:06:06.735634+0000","last_clean":"2024-05-09T05:06:06.735634+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:06:06.735634+0000","last_undegraded":"2024-05-09T05:06:06.735634+0000","last_fullsized":"2024-05-09T05:06:06.735634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737253+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:06:06.737253+0000","last_peered":"2024-05-09T05:06:06.737253+0000","last_clean":"2024-05-09T05:06:06.737253+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:06:06.737253+0000","last_undegraded":"2024-05-09T05:06:06.737253+0000","last_fullsized":"2024-05-09T05:06:06.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736506+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:06:06.736506+0000","last_peered":"2024-05-09T05:06:06.736506+0000","last_clean":"2024-05-09T05:06:06.736506+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:06:06.736506+0000","last_undegraded":"2024-05-09T05:06:06.736506+0000","last_fullsized":"2024-05-09T05:06:06.736506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736352+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:06:06.736352+0000","last_peered":"2024-05-09T05:06:06.736352+0000","last_clean":"2024-05-09T05:06:06.736352+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:06:06.736352+0000","last_undegraded":"2024-05-09T05:06:06.736352+0000","last_fullsized":"2024-05-09T05:06:06.736352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722788+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:06:06.722788+0000","last_peered":"2024-05-09T05:06:06.722788+0000","last_clean":"2024-05-09T05:06:06.722788+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:06:06.722788+0000","last_undegraded":"2024-05-09T05:06:06.722788+0000","last_fullsized":"2024-05-09T05:06:06.722788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736075+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:06:06.736075+0000","last_peered":"2024-05-09T05:06:06.736075+0000","last_clean":"2024-05-09T05:06:06.736075+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:06:06.736075+0000","last_undegraded":"2024-05-09T05:06:06.736075+0000","last_fullsized":"2024-05-09T05:06:06.736075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736071+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:06:06.736071+0000","last_peered":"2024-05-09T05:06:06.736071+0000","last_clean":"2024-05-09T05:06:06.736071+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:06:06.736071+0000","last_undegraded":"2024-05-09T05:06:06.736071+0000","last_fullsized":"2024-05-09T05:06:06.736071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:37.258923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058421+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:06:06.058421+0000","last_peered":"2024-05-09T05:06:06.058421+0000","last_clean":"2024-05-09T05:06:06.058421+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:06:06.058421+0000","last_undegraded":"2024-05-09T05:06:06.058421+0000","last_fullsized":"2024-05-09T05:06:06.058421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062588+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:06:06.062588+0000","last_peered":"2024-05-09T05:06:06.062588+0000","last_clean":"2024-05-09T05:06:06.062588+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:06:06.062588+0000","last_undegraded":"2024-05-09T05:06:06.062588+0000","last_fullsized":"2024-05-09T05:06:06.062588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062864+0000","last_change":"2024-05-09T05:06:05.071898+0000","last_active":"2024-05-09T05:06:06.062864+0000","last_peered":"2024-05-09T05:06:06.062864+0000","last_clean":"2024-05-09T05:06:06.062864+0000","last_became_active":"2024-05-09T05:06:05.071618+0000","last_became_peered":"2024-05-09T05:06:05.071618+0000","last_unstale":"2024-05-09T05:06:06.062864+0000","last_undegraded":"2024-05-09T05:06:06.062864+0000","last_fullsized":"2024-05-09T05:06:06.062864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060053+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:06:06.060053+0000","last_peered":"2024-05-09T05:06:06.060053+0000","last_clean":"2024-05-09T05:06:06.060053+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:06:06.060053+0000","last_undegraded":"2024-05-09T05:06:06.060053+0000","last_fullsized":"2024-05-09T05:06:06.060053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737104+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:06:06.737104+0000","last_peered":"2024-05-09T05:06:06.737104+0000","last_clean":"2024-05-09T05:06:06.737104+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:06:06.737104+0000","last_undegraded":"2024-05-09T05:06:06.737104+0000","last_fullsized":"2024-05-09T05:06:06.737104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736090+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:06:06.736090+0000","last_peered":"2024-05-09T05:06:06.736090+0000","last_clean":"2024-05-09T05:06:06.736090+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:06:06.736090+0000","last_undegraded":"2024-05-09T05:06:06.736090+0000","last_fullsized":"2024-05-09T05:06:06.736090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736595+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:06:06.736595+0000","last_peered":"2024-05-09T05:06:06.736595+0000","last_clean":"2024-05-09T05:06:06.736595+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:06:06.736595+0000","last_undegraded":"2024-05-09T05:06:06.736595+0000","last_fullsized":"2024-05-09T05:06:06.736595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:02.665344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062276+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:06:06.062276+0000","last_peered":"2024-05-09T05:06:06.062276+0000","last_clean":"2024-05-09T05:06:06.062276+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:06:06.062276+0000","last_undegraded":"2024-05-09T05:06:06.062276+0000","last_fullsized":"2024-05-09T05:06:06.062276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735814+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:06:06.735814+0000","last_peered":"2024-05-09T05:06:06.735814+0000","last_clean":"2024-05-09T05:06:06.735814+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:06:06.735814+0000","last_undegraded":"2024-05-09T05:06:06.735814+0000","last_fullsized":"2024-05-09T05:06:06.735814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062254+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062254+0000","last_peered":"2024-05-09T05:06:06.062254+0000","last_clean":"2024-05-09T05:06:06.062254+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:06:06.062254+0000","last_undegraded":"2024-05-09T05:06:06.062254+0000","last_fullsized":"2024-05-09T05:06:06.062254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736447+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:06:06.736447+0000","last_peered":"2024-05-09T05:06:06.736447+0000","last_clean":"2024-05-09T05:06:06.736447+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:06:06.736447+0000","last_undegraded":"2024-05-09T05:06:06.736447+0000","last_fullsized":"2024-05-09T05:06:06.736447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:25:28.166958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736699+0000","last_change":"2024-05-09T05:06:05.067878+0000","last_active":"2024-05-09T05:06:06.736699+0000","last_peered":"2024-05-09T05:06:06.736699+0000","last_clean":"2024-05-09T05:06:06.736699+0000","last_became_active":"2024-05-09T05:06:05.067426+0000","last_became_peered":"2024-05-09T05:06:05.067426+0000","last_unstale":"2024-05-09T05:06:06.736699+0000","last_undegraded":"2024-05-09T05:06:06.736699+0000","last_fullsized":"2024-05-09T05:06:06.736699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058528+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:06:06.058528+0000","last_peered":"2024-05-09T05:06:06.058528+0000","last_clean":"2024-05-09T05:06:06.058528+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:06:06.058528+0000","last_undegraded":"2024-05-09T05:06:06.058528+0000","last_fullsized":"2024-05-09T05:06:06.058528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736532+0000","last_change":"2024-05-09T05:06:05.063359+0000","last_active":"2024-05-09T05:06:06.736532+0000","last_peered":"2024-05-09T05:06:06.736532+0000","last_clean":"2024-05-09T05:06:06.736532+0000","last_became_active":"2024-05-09T05:06:05.063026+0000","last_became_peered":"2024-05-09T05:06:05.063026+0000","last_unstale":"2024-05-09T05:06:06.736532+0000","last_undegraded":"2024-05-09T05:06:06.736532+0000","last_fullsized":"2024-05-09T05:06:06.736532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736550+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:06:06.736550+0000","last_peered":"2024-05-09T05:06:06.736550+0000","last_clean":"2024-05-09T05:06:06.736550+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:06:06.736550+0000","last_undegraded":"2024-05-09T05:06:06.736550+0000","last_fullsized":"2024-05-09T05:06:06.736550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062682+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:06:06.062682+0000","last_peered":"2024-05-09T05:06:06.062682+0000","last_clean":"2024-05-09T05:06:06.062682+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:06:06.062682+0000","last_undegraded":"2024-05-09T05:06:06.062682+0000","last_fullsized":"2024-05-09T05:06:06.062682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059766+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:06:06.059766+0000","last_peered":"2024-05-09T05:06:06.059766+0000","last_clean":"2024-05-09T05:06:06.059766+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:06:06.059766+0000","last_undegraded":"2024-05-09T05:06:06.059766+0000","last_fullsized":"2024-05-09T05:06:06.059766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736152+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:06:06.736152+0000","last_peered":"2024-05-09T05:06:06.736152+0000","last_clean":"2024-05-09T05:06:06.736152+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:06:06.736152+0000","last_undegraded":"2024-05-09T05:06:06.736152+0000","last_fullsized":"2024-05-09T05:06:06.736152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722480+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:06:06.722480+0000","last_peered":"2024-05-09T05:06:06.722480+0000","last_clean":"2024-05-09T05:06:06.722480+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:06:06.722480+0000","last_undegraded":"2024-05-09T05:06:06.722480+0000","last_fullsized":"2024-05-09T05:06:06.722480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736730+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:06:06.736730+0000","last_peered":"2024-05-09T05:06:06.736730+0000","last_clean":"2024-05-09T05:06:06.736730+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:06:06.736730+0000","last_undegraded":"2024-05-09T05:06:06.736730+0000","last_fullsized":"2024-05-09T05:06:06.736730+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736786+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:06:06.736786+0000","last_peered":"2024-05-09T05:06:06.736786+0000","last_clean":"2024-05-09T05:06:06.736786+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:06:06.736786+0000","last_undegraded":"2024-05-09T05:06:06.736786+0000","last_fullsized":"2024-05-09T05:06:06.736786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736064+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:06:06.736064+0000","last_peered":"2024-05-09T05:06:06.736064+0000","last_clean":"2024-05-09T05:06:06.736064+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:06:06.736064+0000","last_undegraded":"2024-05-09T05:06:06.736064+0000","last_fullsized":"2024-05-09T05:06:06.736064+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736318+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:06:06.736318+0000","last_peered":"2024-05-09T05:06:06.736318+0000","last_clean":"2024-05-09T05:06:06.736318+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:06:06.736318+0000","last_undegraded":"2024-05-09T05:06:06.736318+0000","last_fullsized":"2024-05-09T05:06:06.736318+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735754+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:06:06.735754+0000","last_peered":"2024-05-09T05:06:06.735754+0000","last_clean":"2024-05-09T05:06:06.735754+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:06:06.735754+0000","last_undegraded":"2024-05-09T05:06:06.735754+0000","last_fullsized":"2024-05-09T05:06:06.735754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062362+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:06:06.062362+0000","last_peered":"2024-05-09T05:06:06.062362+0000","last_clean":"2024-05-09T05:06:06.062362+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:06:06.062362+0000","last_undegraded":"2024-05-09T05:06:06.062362+0000","last_fullsized":"2024-05-09T05:06:06.062362+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058418+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:06:06.058418+0000","last_peered":"2024-05-09T05:06:06.058418+0000","last_clean":"2024-05-09T05:06:06.058418+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:06:06.058418+0000","last_undegraded":"2024-05-09T05:06:06.058418+0000","last_fullsized":"2024-05-09T05:06:06.058418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736176+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:06:06.736176+0000","last_peered":"2024-05-09T05:06:06.736176+0000","last_clean":"2024-05-09T05:06:06.736176+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:06:06.736176+0000","last_undegraded":"2024-05-09T05:06:06.736176+0000","last_fullsized":"2024-05-09T05:06:06.736176+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062383+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:06:06.062383+0000","last_peered":"2024-05-09T05:06:06.062383+0000","last_clean":"2024-05-09T05:06:06.062383+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:06:06.062383+0000","last_undegraded":"2024-05-09T05:06:06.062383+0000","last_fullsized":"2024-05-09T05:06:06.062383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062550+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:06:06.062550+0000","last_peered":"2024-05-09T05:06:06.062550+0000","last_clean":"2024-05-09T05:06:06.062550+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:06:06.062550+0000","last_undegraded":"2024-05-09T05:06:06.062550+0000","last_fullsized":"2024-05-09T05:06:06.062550+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736605+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:06:06.736605+0000","last_peered":"2024-05-09T05:06:06.736605+0000","last_clean":"2024-05-09T05:06:06.736605+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:06:06.736605+0000","last_undegraded":"2024-05-09T05:06:06.736605+0000","last_fullsized":"2024-05-09T05:06:06.736605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:26.607004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053831+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:06:10.053831+0000","last_peered":"2024-05-09T05:06:10.053831+0000","last_clean":"2024-05-09T05:06:10.053831+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:06:10.053831+0000","last_undegraded":"2024-05-09T05:06:10.053831+0000","last_fullsized":"2024-05-09T05:06:10.053831+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722806+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:06:06.722806+0000","last_peered":"2024-05-09T05:06:06.722806+0000","last_clean":"2024-05-09T05:06:06.722806+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:06:06.722806+0000","last_undegraded":"2024-05-09T05:06:06.722806+0000","last_fullsized":"2024-05-09T05:06:06.722806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.055062+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:06:10.055062+0000","last_peered":"2024-05-09T05:06:10.055062+0000","last_clean":"2024-05-09T05:06:10.055062+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:06:10.055062+0000","last_undegraded":"2024-05-09T05:06:10.055062+0000","last_fullsized":"2024-05-09T05:06:10.055062+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736736+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:06:06.736736+0000","last_peered":"2024-05-09T05:06:06.736736+0000","last_clean":"2024-05-09T05:06:06.736736+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:06:06.736736+0000","last_undegraded":"2024-05-09T05:06:06.736736+0000","last_fullsized":"2024-05-09T05:06:06.736736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053672+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:06:10.053672+0000","last_peered":"2024-05-09T05:06:10.053672+0000","last_clean":"2024-05-09T05:06:10.053672+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:06:10.053672+0000","last_undegraded":"2024-05-09T05:06:10.053672+0000","last_fullsized":"2024-05-09T05:06:10.053672+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736396+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:06:06.736396+0000","last_peered":"2024-05-09T05:06:06.736396+0000","last_clean":"2024-05-09T05:06:06.736396+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:06:06.736396+0000","last_undegraded":"2024-05-09T05:06:06.736396+0000","last_fullsized":"2024-05-09T05:06:06.736396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:41.283927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062613+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:06:06.062613+0000","last_peered":"2024-05-09T05:06:06.062613+0000","last_clean":"2024-05-09T05:06:06.062613+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:06:06.062613+0000","last_undegraded":"2024-05-09T05:06:06.062613+0000","last_fullsized":"2024-05-09T05:06:06.062613+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737118+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:06:06.737118+0000","last_peered":"2024-05-09T05:06:06.737118+0000","last_clean":"2024-05-09T05:06:06.737118+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:06:06.737118+0000","last_undegraded":"2024-05-09T05:06:06.737118+0000","last_fullsized":"2024-05-09T05:06:06.737118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736071+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:06:06.736071+0000","last_peered":"2024-05-09T05:06:06.736071+0000","last_clean":"2024-05-09T05:06:06.736071+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:06:06.736071+0000","last_undegraded":"2024-05-09T05:06:06.736071+0000","last_fullsized":"2024-05-09T05:06:06.736071+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736621+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:06:06.736621+0000","last_peered":"2024-05-09T05:06:06.736621+0000","last_clean":"2024-05-09T05:06:06.736621+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:06:06.736621+0000","last_undegraded":"2024-05-09T05:06:06.736621+0000","last_fullsized":"2024-05-09T05:06:06.736621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'8","reported_seq":22,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.056327+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:06:10.056327+0000","last_peered":"2024-05-09T05:06:10.056327+0000","last_clean":"2024-05-09T05:06:10.056327+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:06:10.056327+0000","last_undegraded":"2024-05-09T05:06:10.056327+0000","last_fullsized":"2024-05-09T05:06:10.056327+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736921+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:06:06.736921+0000","last_peered":"2024-05-09T05:06:06.736921+0000","last_clean":"2024-05-09T05:06:06.736921+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:06:06.736921+0000","last_undegraded":"2024-05-09T05:06:06.736921+0000","last_fullsized":"2024-05-09T05:06:06.736921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722789+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:06:06.722789+0000","last_peered":"2024-05-09T05:06:06.722789+0000","last_clean":"2024-05-09T05:06:06.722789+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:06:06.722789+0000","last_undegraded":"2024-05-09T05:06:06.722789+0000","last_fullsized":"2024-05-09T05:06:06.722789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737109+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:06:06.737109+0000","last_peered":"2024-05-09T05:06:06.737109+0000","last_clean":"2024-05-09T05:06:06.737109+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:06:06.737109+0000","last_undegraded":"2024-05-09T05:06:06.737109+0000","last_fullsized":"2024-05-09T05:06:06.737109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053514+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:06:10.053514+0000","last_peered":"2024-05-09T05:06:10.053514+0000","last_clean":"2024-05-09T05:06:10.053514+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:06:10.053514+0000","last_undegraded":"2024-05-09T05:06:10.053514+0000","last_fullsized":"2024-05-09T05:06:10.053514+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737391+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:06:06.737391+0000","last_peered":"2024-05-09T05:06:06.737391+0000","last_clean":"2024-05-09T05:06:06.737391+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:06:06.737391+0000","last_undegraded":"2024-05-09T05:06:06.737391+0000","last_fullsized":"2024-05-09T05:06:06.737391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.555563+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:06:06.555563+0000","last_peered":"2024-05-09T05:06:06.555563+0000","last_clean":"2024-05-09T05:06:06.555563+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:06:06.555563+0000","last_undegraded":"2024-05-09T05:06:06.555563+0000","last_fullsized":"2024-05-09T05:06:06.555563+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:06:06.062865+0000","last_peered":"2024-05-09T05:06:06.062865+0000","last_clean":"2024-05-09T05:06:06.062865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:06:06.062865+0000","last_undegraded":"2024-05-09T05:06:06.062865+0000","last_fullsized":"2024-05-09T05:06:06.062865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060038+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:06:06.060038+0000","last_peered":"2024-05-09T05:06:06.060038+0000","last_clean":"2024-05-09T05:06:06.060038+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:06:06.060038+0000","last_undegraded":"2024-05-09T05:06:06.060038+0000","last_fullsized":"2024-05-09T05:06:06.060038+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736885+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:06:06.736885+0000","last_peered":"2024-05-09T05:06:06.736885+0000","last_clean":"2024-05-09T05:06:06.736885+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:06:06.736885+0000","last_undegraded":"2024-05-09T05:06:06.736885+0000","last_fullsized":"2024-05-09T05:06:06.736885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736326+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:06:06.736326+0000","last_peered":"2024-05-09T05:06:06.736326+0000","last_clean":"2024-05-09T05:06:06.736326+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:06:06.736326+0000","last_undegraded":"2024-05-09T05:06:06.736326+0000","last_fullsized":"2024-05-09T05:06:06.736326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062158+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:06:06.062158+0000","last_peered":"2024-05-09T05:06:06.062158+0000","last_clean":"2024-05-09T05:06:06.062158+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:06:06.062158+0000","last_undegraded":"2024-05-09T05:06:06.062158+0000","last_fullsized":"2024-05-09T05:06:06.062158+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735899+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:06:06.735899+0000","last_peered":"2024-05-09T05:06:06.735899+0000","last_clean":"2024-05-09T05:06:06.735899+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:06:06.735899+0000","last_undegraded":"2024-05-09T05:06:06.735899+0000","last_fullsized":"2024-05-09T05:06:06.735899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736248+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:06:06.736248+0000","last_peered":"2024-05-09T05:06:06.736248+0000","last_clean":"2024-05-09T05:06:06.736248+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:06:06.736248+0000","last_undegraded":"2024-05-09T05:06:06.736248+0000","last_fullsized":"2024-05-09T05:06:06.736248+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736254+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:06:06.736254+0000","last_peered":"2024-05-09T05:06:06.736254+0000","last_clean":"2024-05-09T05:06:06.736254+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:06:06.736254+0000","last_undegraded":"2024-05-09T05:06:06.736254+0000","last_fullsized":"2024-05-09T05:06:06.736254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:23.362012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059887+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:06:06.059887+0000","last_peered":"2024-05-09T05:06:06.059887+0000","last_clean":"2024-05-09T05:06:06.059887+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:06:06.059887+0000","last_undegraded":"2024-05-09T05:06:06.059887+0000","last_fullsized":"2024-05-09T05:06:06.059887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058422+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:06:06.058422+0000","last_peered":"2024-05-09T05:06:06.058422+0000","last_clean":"2024-05-09T05:06:06.058422+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:06:06.058422+0000","last_undegraded":"2024-05-09T05:06:06.058422+0000","last_fullsized":"2024-05-09T05:06:06.058422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062842+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:06:06.062842+0000","last_peered":"2024-05-09T05:06:06.062842+0000","last_clean":"2024-05-09T05:06:06.062842+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:06:06.062842+0000","last_undegraded":"2024-05-09T05:06:06.062842+0000","last_fullsized":"2024-05-09T05:06:06.062842+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722592+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:06:06.722592+0000","last_peered":"2024-05-09T05:06:06.722592+0000","last_clean":"2024-05-09T05:06:06.722592+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:06:06.722592+0000","last_undegraded":"2024-05-09T05:06:06.722592+0000","last_fullsized":"2024-05-09T05:06:06.722592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058680+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:06:06.058680+0000","last_peered":"2024-05-09T05:06:06.058680+0000","last_clean":"2024-05-09T05:06:06.058680+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:06:06.058680+0000","last_undegraded":"2024-05-09T05:06:06.058680+0000","last_fullsized":"2024-05-09T05:06:06.058680+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062852+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:06:06.062852+0000","last_peered":"2024-05-09T05:06:06.062852+0000","last_clean":"2024-05-09T05:06:06.062852+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:06:06.062852+0000","last_undegraded":"2024-05-09T05:06:06.062852+0000","last_fullsized":"2024-05-09T05:06:06.062852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737087+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:06:06.737087+0000","last_peered":"2024-05-09T05:06:06.737087+0000","last_clean":"2024-05-09T05:06:06.737087+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:06:06.737087+0000","last_undegraded":"2024-05-09T05:06:06.737087+0000","last_fullsized":"2024-05-09T05:06:06.737087+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058622+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:06:06.058622+0000","last_peered":"2024-05-09T05:06:06.058622+0000","last_clean":"2024-05-09T05:06:06.058622+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:06:06.058622+0000","last_undegraded":"2024-05-09T05:06:06.058622+0000","last_fullsized":"2024-05-09T05:06:06.058622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058810+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:06:06.058810+0000","last_peered":"2024-05-09T05:06:06.058810+0000","last_clean":"2024-05-09T05:06:06.058810+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:06:06.058810+0000","last_undegraded":"2024-05-09T05:06:06.058810+0000","last_fullsized":"2024-05-09T05:06:06.058810+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736837+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:06:06.736837+0000","last_peered":"2024-05-09T05:06:06.736837+0000","last_clean":"2024-05-09T05:06:06.736837+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:06:06.736837+0000","last_undegraded":"2024-05-09T05:06:06.736837+0000","last_fullsized":"2024-05-09T05:06:06.736837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:33.840344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053220+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:06:10.053220+0000","last_peered":"2024-05-09T05:06:10.053220+0000","last_clean":"2024-05-09T05:06:10.053220+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:06:10.053220+0000","last_undegraded":"2024-05-09T05:06:10.053220+0000","last_fullsized":"2024-05-09T05:06:10.053220+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736183+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:06:06.736183+0000","last_peered":"2024-05-09T05:06:06.736183+0000","last_clean":"2024-05-09T05:06:06.736183+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:06:06.736183+0000","last_undegraded":"2024-05-09T05:06:06.736183+0000","last_fullsized":"2024-05-09T05:06:06.736183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:50.236344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736055+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:06:06.736055+0000","last_peered":"2024-05-09T05:06:06.736055+0000","last_clean":"2024-05-09T05:06:06.736055+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:06:06.736055+0000","last_undegraded":"2024-05-09T05:06:06.736055+0000","last_fullsized":"2024-05-09T05:06:06.736055+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722580+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:06:06.722580+0000","last_peered":"2024-05-09T05:06:06.722580+0000","last_clean":"2024-05-09T05:06:06.722580+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:06:06.722580+0000","last_undegraded":"2024-05-09T05:06:06.722580+0000","last_fullsized":"2024-05-09T05:06:06.722580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058525+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:06:06.058525+0000","last_peered":"2024-05-09T05:06:06.058525+0000","last_clean":"2024-05-09T05:06:06.058525+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:06:06.058525+0000","last_undegraded":"2024-05-09T05:06:06.058525+0000","last_fullsized":"2024-05-09T05:06:06.058525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736386+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:06:06.736386+0000","last_peered":"2024-05-09T05:06:06.736386+0000","last_clean":"2024-05-09T05:06:06.736386+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:06:06.736386+0000","last_undegraded":"2024-05-09T05:06:06.736386+0000","last_fullsized":"2024-05-09T05:06:06.736386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":14,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058700+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:06:06.058700+0000","last_peered":"2024-05-09T05:06:06.058700+0000","last_clean":"2024-05-09T05:06:06.058700+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:06:06.058700+0000","last_undegraded":"2024-05-09T05:06:06.058700+0000","last_fullsized":"2024-05-09T05:06:06.058700+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736096+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:06:06.736096+0000","last_peered":"2024-05-09T05:06:06.736096+0000","last_clean":"2024-05-09T05:06:06.736096+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:06:06.736096+0000","last_undegraded":"2024-05-09T05:06:06.736096+0000","last_fullsized":"2024-05-09T05:06:06.736096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:58:35.651549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053430+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:06:10.053430+0000","last_peered":"2024-05-09T05:06:10.053430+0000","last_clean":"2024-05-09T05:06:10.053430+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:06:10.053430+0000","last_undegraded":"2024-05-09T05:06:10.053430+0000","last_fullsized":"2024-05-09T05:06:10.053430+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736285+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:06:06.736285+0000","last_peered":"2024-05-09T05:06:06.736285+0000","last_clean":"2024-05-09T05:06:06.736285+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:06:06.736285+0000","last_undegraded":"2024-05-09T05:06:06.736285+0000","last_fullsized":"2024-05-09T05:06:06.736285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062736+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062736+0000","last_peered":"2024-05-09T05:06:06.062736+0000","last_clean":"2024-05-09T05:06:06.062736+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:06:06.062736+0000","last_undegraded":"2024-05-09T05:06:06.062736+0000","last_fullsized":"2024-05-09T05:06:06.062736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053716+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:06:10.053716+0000","last_peered":"2024-05-09T05:06:10.053716+0000","last_clean":"2024-05-09T05:06:10.053716+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:06:10.053716+0000","last_undegraded":"2024-05-09T05:06:10.053716+0000","last_fullsized":"2024-05-09T05:06:10.053716+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062769+0000","last_change":"2024-05-09T05:06:05.063337+0000","last_active":"2024-05-09T05:06:06.062769+0000","last_peered":"2024-05-09T05:06:06.062769+0000","last_clean":"2024-05-09T05:06:06.062769+0000","last_became_active":"2024-05-09T05:06:05.063016+0000","last_became_peered":"2024-05-09T05:06:05.063016+0000","last_unstale":"2024-05-09T05:06:06.062769+0000","last_undegraded":"2024-05-09T05:06:06.062769+0000","last_fullsized":"2024-05-09T05:06:06.062769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.052083+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:06:10.052083+0000","last_peered":"2024-05-09T05:06:10.052083+0000","last_clean":"2024-05-09T05:06:10.052083+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:06:10.052083+0000","last_undegraded":"2024-05-09T05:06:10.052083+0000","last_fullsized":"2024-05-09T05:06:10.052083+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736394+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:06:06.736394+0000","last_peered":"2024-05-09T05:06:06.736394+0000","last_clean":"2024-05-09T05:06:06.736394+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:06:06.736394+0000","last_undegraded":"2024-05-09T05:06:06.736394+0000","last_fullsized":"2024-05-09T05:06:06.736394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:42.457145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736800+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:06:06.736800+0000","last_peered":"2024-05-09T05:06:06.736800+0000","last_clean":"2024-05-09T05:06:06.736800+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:06:06.736800+0000","last_undegraded":"2024-05-09T05:06:06.736800+0000","last_fullsized":"2024-05-09T05:06:06.736800+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736091+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:06:06.736091+0000","last_peered":"2024-05-09T05:06:06.736091+0000","last_clean":"2024-05-09T05:06:06.736091+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:06:06.736091+0000","last_undegraded":"2024-05-09T05:06:06.736091+0000","last_fullsized":"2024-05-09T05:06:06.736091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:32:29.331565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.054793+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:06:10.054793+0000","last_peered":"2024-05-09T05:06:10.054793+0000","last_clean":"2024-05-09T05:06:10.054793+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:06:10.054793+0000","last_undegraded":"2024-05-09T05:06:10.054793+0000","last_fullsized":"2024-05-09T05:06:10.054793+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":2410,"num_objects":21,"num_object_clones":0,"num_object_copies":42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":44,"num_write_kb":14,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":4820,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":38,"ondisk_log_size":38,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509446,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27804,"kb_used_data":788,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344036,"statfs":{"total":96636764160,"available":96608292864,"internally_reserved":0,"allocated":806912,"data_stored":664479,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509447,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27224,"kb_used_data":208,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344616,"statfs":{"total":96636764160,"available":96608886784,"internally_reserved":0,"allocated":212992,"data_stored":73619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542151,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":75141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542152,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27232,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344608,"statfs":{"total":96636764160,"available":96608878592,"internally_reserved":0,"allocated":221184,"data_stored":73655,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542152,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31392,"kb_used_data":216,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340448,"statfs":{"total":96636764160,"available":96604618752,"internally_reserved":0,"allocated":221184,"data_stored":75141,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542153,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31380,"kb_used_data":204,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340460,"statfs":{"total":96636764160,"available":96604631040,"internally_reserved":0,"allocated":208896,"data_stored":73529,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542152,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31388,"kb_used_data":212,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340452,"statfs":{"total":96636764160,"available":96604622848,"internally_reserved":0,"allocated":217088,"data_stored":73689,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542151,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36132,"kb_used_data":796,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335708,"statfs":{"total":96636764160,"available":96599764992,"internally_reserved":0,"allocated":815104,"data_stored":664535,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":638,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":126,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:06:16.312 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T05:06:16.312 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-09T05:06:16.312 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2024-05-09T05:06:16.313 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-09T05:06:16.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.461+0000 7f2688cb1640 1 Processor -- start 2024-05-09T05:06:16.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.461+0000 7f2688cb1640 1 -- start start 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2688cb1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2684156900 0x7f26840c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2688cb1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26840c7870 0x7f26840c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2688cb1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f268406de20 con 0x7f268406efa0 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2688cb1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2684156d10 con 0x7f26840c7870 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2688cb1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f268415c9a0 con 0x7f2684156900 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2682d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26840c7870 0x7f26840c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f268259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f268406efa0 0x7f26840cc560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35648/0 (socket says 172.21.15.1:35648) 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2682d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26840c7870 0x7f26840c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49598/0 (socket says 172.21.15.1:49598) 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f268259c640 1 -- 172.21.15.1:0/1087756478 learned_addr learned my addr 172.21.15.1:0/1087756478 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2681d9b640 1 --2- 172.21.15.1:0/1087756478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2684156900 0x7f26840c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2682d9d640 1 -- 172.21.15.1:0/1087756478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2684156900 msgr2=0x7f26840c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2682d9d640 1 --2- 172.21.15.1:0/1087756478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2684156900 0x7f26840c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2682d9d640 1 -- 172.21.15.1:0/1087756478 >> v1:172.21.15.1:6789/0 conn(0x7f268406efa0 legacy=0x7f26840cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f268159a640 1 -- 172.21.15.1:0/1087756478 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2426512419 0 0) 0x7f268406de20 con 0x7f268406efa0 2024-05-09T05:06:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.463+0000 7f2682d9d640 1 -- 172.21.15.1:0/1087756478 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26840b51a0 con 0x7f26840c7870 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.464+0000 7f2682d9d640 1 --2- 172.21.15.1:0/1087756478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26840c7870 0x7f26840c7c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f267c0097a0 tx=0x7f267c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b8441e321f4e154 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.464+0000 7f268159a640 1 -- 172.21.15.1:0/1087756478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c037070 con 0x7f26840c7870 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.464+0000 7f268159a640 1 -- 172.21.15.1:0/1087756478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c032440 con 0x7f26840c7870 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.464+0000 7f268159a640 1 -- 172.21.15.1:0/1087756478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c032810 con 0x7f26840c7870 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.465+0000 7f2688cb1640 1 -- 172.21.15.1:0/1087756478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26840c7870 msgr2=0x7f26840c7c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.465+0000 7f2688cb1640 1 --2- 172.21.15.1:0/1087756478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26840c7870 0x7f26840c7c80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f267c0097a0 tx=0x7f267c02a210 comp rx=0 tx=0).stop 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.465+0000 7f2688cb1640 1 -- 172.21.15.1:0/1087756478 shutdown_connections 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.465+0000 7f2688cb1640 1 --2- 172.21.15.1:0/1087756478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2684156900 0x7f26840c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.465+0000 7f2688cb1640 1 --2- 172.21.15.1:0/1087756478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26840c7870 0x7f26840c7c80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.465+0000 7f2688cb1640 1 -- 172.21.15.1:0/1087756478 >> 172.21.15.1:0/1087756478 conn(0x7f2684068d90 msgr2=0x7f26840d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.465+0000 7f2688cb1640 1 -- 172.21.15.1:0/1087756478 shutdown_connections 2024-05-09T05:06:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.465+0000 7f2688cb1640 1 -- 172.21.15.1:0/1087756478 wait complete. 2024-05-09T05:06:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.467+0000 7f2688cb1640 1 Processor -- start 2024-05-09T05:06:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.467+0000 7f2688cb1640 1 -- start start 2024-05-09T05:06:16.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2688cb1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f268406efa0 0x7f2684150390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2688cb1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26840c7870 0x7f26840d12d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2688cb1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2684156900 0x7f268414c200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2688cb1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f268415cc40 con 0x7f2684156900 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2688cb1640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f268406db40 con 0x7f268406efa0 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2688cb1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f268415cf20 con 0x7f26840c7870 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2682d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2684156900 0x7f268414c200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2682d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2684156900 0x7f268414c200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49606/0 (socket says 172.21.15.1:49606) 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2682d9d640 1 -- 172.21.15.1:0/349930991 learned_addr learned my addr 172.21.15.1:0/349930991 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f268259c640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f268406efa0 0x7f2684150390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2681d9b640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26840c7870 0x7f26840d12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2682d9d640 1 -- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26840c7870 msgr2=0x7f26840d12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2682d9d640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26840c7870 0x7f26840d12d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2682d9d640 1 -- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f268406efa0 msgr2=0x7f2684150390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2682d9d640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f268406efa0 0x7f2684150390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.468+0000 7f2682d9d640 1 -- 172.21.15.1:0/349930991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26841508d0 con 0x7f2684156900 2024-05-09T05:06:16.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.469+0000 7f2682d9d640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2684156900 0x7f268414c200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f267c0097a0 tx=0x7f267c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:16.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.469+0000 7f267affd640 1 -- 172.21.15.1:0/349930991 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c037040 con 0x7f2684156900 2024-05-09T05:06:16.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.469+0000 7f267affd640 1 -- 172.21.15.1:0/349930991 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f267c042070 con 0x7f2684156900 2024-05-09T05:06:16.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.469+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26841e79f0 con 0x7f2684156900 2024-05-09T05:06:16.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.469+0000 7f267affd640 1 -- 172.21.15.1:0/349930991 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f267c030860 con 0x7f2684156900 2024-05-09T05:06:16.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.469+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268414f8b0 con 0x7f2684156900 2024-05-09T05:06:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.470+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26840b7ec0 con 0x7f2684156900 2024-05-09T05:06:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.471+0000 7f267affd640 1 -- 172.21.15.1:0/349930991 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f267c030a00 con 0x7f2684156900 2024-05-09T05:06:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.471+0000 7f267affd640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26640762d0 0x7f26640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.471+0000 7f267affd640 1 -- 172.21.15.1:0/349930991 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f267c0b7c80 con 0x7f2684156900 2024-05-09T05:06:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.471+0000 7f268259c640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26640762d0 0x7f26640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.472+0000 7f268259c640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26640762d0 0x7f26640787e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f26740012e0 tx=0x7f2674001630 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:16.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.476+0000 7f267affd640 1 -- 172.21.15.1:0/349930991 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f26840b7ec0 con 0x7f2684156900 2024-05-09T05:06:16.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.706+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f2684050870 con 0x7f2684156900 2024-05-09T05:06:16.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.706+0000 7f267affd640 1 -- 172.21.15.1:0/349930991 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f267c04c020 con 0x7f2684156900 2024-05-09T05:06:16.707 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:16.707 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26640762d0 msgr2=0x7f26640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26640762d0 0x7f26640787e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f26740012e0 tx=0x7f2674001630 comp rx=0 tx=0).stop 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2684156900 msgr2=0x7f268414c200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2684156900 0x7f268414c200 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f267c0097a0 tx=0x7f267c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 shutdown_connections 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26640762d0 0x7f26640787e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2684156900 0x7f268414c200 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26840c7870 0x7f26840d12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 --2- 172.21.15.1:0/349930991 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f268406efa0 0x7f2684150390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 >> 172.21.15.1:0/349930991 conn(0x7f2684068d90 msgr2=0x7f268414e730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 shutdown_connections 2024-05-09T05:06:16.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:16.708+0000 7f2688cb1640 1 -- 172.21.15.1:0/349930991 wait complete. 2024-05-09T05:06:19.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-09T05:06:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.848+0000 7f117dbe0640 1 Processor -- start 2024-05-09T05:06:19.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.848+0000 7f117dbe0640 1 -- start start 2024-05-09T05:06:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.849+0000 7f117dbe0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f117815ca90 0x7f117815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.849+0000 7f117dbe0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f117806f6d0 con 0x7f117806f1b0 2024-05-09T05:06:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.849+0000 7f117dbe0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11780cc4d0 con 0x7f11780d16b0 2024-05-09T05:06:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.849+0000 7f117dbe0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f11780cee50 con 0x7f117815ca90 2024-05-09T05:06:19.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.850+0000 7f1177fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f117815ca90 0x7f117815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.850+0000 7f1176ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f117806f1b0 0x7f11780ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59366/0 (socket says 172.21.15.1:59366) 2024-05-09T05:06:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.850+0000 7f1176ffd640 1 -- 172.21.15.1:0/1152668526 learned_addr learned my addr 172.21.15.1:0/1152668526 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.850+0000 7f11777fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f11780d16b0 0x7f11780d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:40074/0 (socket says 172.21.15.1:40074) 2024-05-09T05:06:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.850+0000 7f11767fc640 1 -- 172.21.15.1:0/1152668526 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2315438172 0 0) 0x7f117806f6d0 con 0x7f117806f1b0 2024-05-09T05:06:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.850+0000 7f1177fff640 1 -- 172.21.15.1:0/1152668526 >> v1:172.21.15.196:6789/0 conn(0x7f11780d16b0 legacy=0x7f11780d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.850+0000 7f1177fff640 1 -- 172.21.15.1:0/1152668526 >> v1:172.21.15.1:6789/0 conn(0x7f117806f1b0 legacy=0x7f11780ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:19.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.850+0000 7f1177fff640 1 -- 172.21.15.1:0/1152668526 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11780cc7b0 con 0x7f117815ca90 2024-05-09T05:06:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.851+0000 7f1177fff640 1 --2- 172.21.15.1:0/1152668526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 0x7f117815cea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1170009d90 tx=0x7f117002a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e4b8318c5ebcc15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.851+0000 7f11767fc640 1 -- 172.21.15.1:0/1152668526 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170007540 con 0x7f117815ca90 2024-05-09T05:06:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.851+0000 7f11767fc640 1 -- 172.21.15.1:0/1152668526 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11700076e0 con 0x7f117815ca90 2024-05-09T05:06:19.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.851+0000 7f11767fc640 1 -- 172.21.15.1:0/1152668526 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170007ab0 con 0x7f117815ca90 2024-05-09T05:06:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.852+0000 7f117dbe0640 1 -- 172.21.15.1:0/1152668526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 msgr2=0x7f117815cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.852+0000 7f117dbe0640 1 --2- 172.21.15.1:0/1152668526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 0x7f117815cea0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f1170009d90 tx=0x7f117002a6a0 comp rx=0 tx=0).stop 2024-05-09T05:06:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.852+0000 7f117dbe0640 1 -- 172.21.15.1:0/1152668526 shutdown_connections 2024-05-09T05:06:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.852+0000 7f117dbe0640 1 --2- 172.21.15.1:0/1152668526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 0x7f117815cea0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.852+0000 7f117dbe0640 1 -- 172.21.15.1:0/1152668526 >> 172.21.15.1:0/1152668526 conn(0x7f11780b3660 msgr2=0x7f11780b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.852+0000 7f117dbe0640 1 -- 172.21.15.1:0/1152668526 shutdown_connections 2024-05-09T05:06:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.852+0000 7f117dbe0640 1 -- 172.21.15.1:0/1152668526 wait complete. 2024-05-09T05:06:19.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.853+0000 7f117dbe0640 1 Processor -- start 2024-05-09T05:06:19.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.854+0000 7f117dbe0640 1 -- start start 2024-05-09T05:06:19.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f117dbe0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f117806f1b0 0x7f11780c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f117dbe0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11780d16b0 0x7f11781f5cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f117dbe0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 0x7f11781e5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f117dbe0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11780ccd70 con 0x7f11780d16b0 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f117dbe0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f11780cd0e0 con 0x7f117815ca90 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f117dbe0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f117815d3e0 con 0x7f117806f1b0 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1176ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11780d16b0 0x7f11781f5cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1176ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11780d16b0 0x7f11781f5cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54992/0 (socket says 172.21.15.1:54992) 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1176ffd640 1 -- 172.21.15.1:0/465356955 learned_addr learned my addr 172.21.15.1:0/465356955 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f11777fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f117806f1b0 0x7f11780c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1177fff640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 0x7f11781e5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1176ffd640 1 -- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f117806f1b0 msgr2=0x7f11780c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1176ffd640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f117806f1b0 0x7f11780c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:19.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1176ffd640 1 -- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 msgr2=0x7f11781e5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1176ffd640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 0x7f11781e5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.855+0000 7f1176ffd640 1 -- 172.21.15.1:0/465356955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11780c9f60 con 0x7f11780d16b0 2024-05-09T05:06:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.856+0000 7f1176ffd640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11780d16b0 0x7f11781f5cb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f116c00c410 tx=0x7f116c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.856+0000 7f115ffff640 1 -- 172.21.15.1:0/465356955 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c0074e0 con 0x7f11780d16b0 2024-05-09T05:06:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.856+0000 7f115ffff640 1 -- 172.21.15.1:0/465356955 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f116c0047e0 con 0x7f11780d16b0 2024-05-09T05:06:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.856+0000 7f115ffff640 1 -- 172.21.15.1:0/465356955 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f116c004bd0 con 0x7f11780d16b0 2024-05-09T05:06:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.856+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11780c6380 con 0x7f11780d16b0 2024-05-09T05:06:19.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.856+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11780cbc30 con 0x7f11780d16b0 2024-05-09T05:06:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.857+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11780748b0 con 0x7f11780d16b0 2024-05-09T05:06:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.858+0000 7f115ffff640 1 -- 172.21.15.1:0/465356955 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f116c011070 con 0x7f11780d16b0 2024-05-09T05:06:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.858+0000 7f115ffff640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1164076300 0x7f1164078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.858+0000 7f115ffff640 1 -- 172.21.15.1:0/465356955 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f116c097160 con 0x7f11780d16b0 2024-05-09T05:06:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.858+0000 7f11777fe640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1164076300 0x7f1164078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:19.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.859+0000 7f11777fe640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1164076300 0x7f1164078810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1168007bd0 tx=0x7f1168007b20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:19.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:19.863+0000 7f115ffff640 1 -- 172.21.15.1:0/465356955 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f11780748b0 con 0x7f11780d16b0 2024-05-09T05:06:20.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.093+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f117806ecd0 con 0x7f11780d16b0 2024-05-09T05:06:20.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.094+0000 7f115ffff640 1 -- 172.21.15.1:0/465356955 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7f116c0603d0 con 0x7f11780d16b0 2024-05-09T05:06:20.094 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:20.094 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-09T05:06:20.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1164076300 msgr2=0x7f1164078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1164076300 0x7f1164078810 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1168007bd0 tx=0x7f1168007b20 comp rx=0 tx=0).stop 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11780d16b0 msgr2=0x7f11781f5cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11780d16b0 0x7f11781f5cb0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f116c00c410 tx=0x7f116c00c8d0 comp rx=0 tx=0).stop 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 shutdown_connections 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1164076300 0x7f1164078810 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f117815ca90 0x7f11781e5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11780d16b0 0x7f11781f5cb0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 --2- 172.21.15.1:0/465356955 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f117806f1b0 0x7f11780c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 >> 172.21.15.1:0/465356955 conn(0x7f11780b3660 msgr2=0x7f11780d1ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 shutdown_connections 2024-05-09T05:06:20.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:20.096+0000 7f117dbe0640 1 -- 172.21.15.1:0/465356955 wait complete. 2024-05-09T05:06:23.108 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-09T05:06:23.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.236+0000 7fb2e3c69640 1 Processor -- start 2024-05-09T05:06:23.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.236+0000 7fb2e3c69640 1 -- start start 2024-05-09T05:06:23.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e3c69640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2dc072120 0x7fb2dc156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:23.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e3c69640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc0caab0 0x7fb2dc0caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:23.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e3c69640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb2dc0c5e00 con 0x7fb2dc15d1d0 2024-05-09T05:06:23.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e3c69640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb2dc0ca340 con 0x7fb2dc072120 2024-05-09T05:06:23.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e3c69640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb2dc0ca620 con 0x7fb2dc0caab0 2024-05-09T05:06:23.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e1a05640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2dc072120 0x7fb2dc156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:23.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e1a05640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb2dc072120 0x7fb2dc156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54998/0 (socket says 172.21.15.1:54998) 2024-05-09T05:06:23.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e1a05640 1 -- 172.21.15.1:0/3403724922 learned_addr learned my addr 172.21.15.1:0/3403724922 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:23.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.237+0000 7fb2e2206640 1 --2- 172.21.15.1:0/3403724922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc0caab0 0x7fb2dc0caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:23.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.238+0000 7fb2e1a05640 1 -- 172.21.15.1:0/3403724922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc0caab0 msgr2=0x7fb2dc0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:23.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.238+0000 7fb2e1a05640 1 --2- 172.21.15.1:0/3403724922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc0caab0 0x7fb2dc0caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.238+0000 7fb2e1a05640 1 -- 172.21.15.1:0/3403724922 >> v1:172.21.15.196:6789/0 conn(0x7fb2dc15d1d0 legacy=0x7fb2dc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:23.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.238+0000 7fb2e1a05640 1 -- 172.21.15.1:0/3403724922 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2dc0c60e0 con 0x7fb2dc072120 2024-05-09T05:06:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.239+0000 7fb2e1a05640 1 --2- 172.21.15.1:0/3403724922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc072120 0x7fb2dc156a80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009870 tx=0x7fb2d402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25038ce804221c54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.239+0000 7fb2e0a03640 1 -- 172.21.15.1:0/3403724922 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4037070 con 0x7fb2dc072120 2024-05-09T05:06:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.239+0000 7fb2e0a03640 1 -- 172.21.15.1:0/3403724922 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2d4032440 con 0x7fb2dc072120 2024-05-09T05:06:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.239+0000 7fb2e0a03640 1 -- 172.21.15.1:0/3403724922 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4032810 con 0x7fb2dc072120 2024-05-09T05:06:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.240+0000 7fb2e3c69640 1 -- 172.21.15.1:0/3403724922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc072120 msgr2=0x7fb2dc156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.240+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/3403724922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc072120 0x7fb2dc156a80 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009870 tx=0x7fb2d402a270 comp rx=0 tx=0).stop 2024-05-09T05:06:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.240+0000 7fb2e3c69640 1 -- 172.21.15.1:0/3403724922 shutdown_connections 2024-05-09T05:06:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.240+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/3403724922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc0caab0 0x7fb2dc0caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.240+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/3403724922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc072120 0x7fb2dc156a80 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.240+0000 7fb2e3c69640 1 -- 172.21.15.1:0/3403724922 >> 172.21.15.1:0/3403724922 conn(0x7fb2dc0731d0 msgr2=0x7fb2dc0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.240+0000 7fb2e3c69640 1 -- 172.21.15.1:0/3403724922 shutdown_connections 2024-05-09T05:06:23.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.240+0000 7fb2e3c69640 1 -- 172.21.15.1:0/3403724922 wait complete. 2024-05-09T05:06:23.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.242+0000 7fb2e3c69640 1 Processor -- start 2024-05-09T05:06:23.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.242+0000 7fb2e3c69640 1 -- start start 2024-05-09T05:06:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e3c69640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc072120 0x7fb2dc15de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e3c69640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc0caab0 0x7fb2dc06ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e3c69640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2dc15d1d0 0x7fb2dc06f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e3c69640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc0cb400 con 0x7fb2dc0caab0 2024-05-09T05:06:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e3c69640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc0c66a0 con 0x7fb2dc15d1d0 2024-05-09T05:06:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e3c69640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb2dc15d5e0 con 0x7fb2dc072120 2024-05-09T05:06:23.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e1204640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc0caab0 0x7fb2dc06ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e1204640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc0caab0 0x7fb2dc06ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55012/0 (socket says 172.21.15.1:55012) 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e1a05640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc072120 0x7fb2dc15de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e1204640 1 -- 172.21.15.1:0/705777060 learned_addr learned my addr 172.21.15.1:0/705777060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e2206640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2dc15d1d0 0x7fb2dc06f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e1204640 1 -- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc072120 msgr2=0x7fb2dc15de90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.243+0000 7fb2e1204640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc072120 0x7fb2dc15de90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2e1204640 1 -- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2dc15d1d0 msgr2=0x7fb2dc06f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2e1204640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2dc15d1d0 0x7fb2dc06f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2e1204640 1 -- 172.21.15.1:0/705777060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2dc15e3d0 con 0x7fb2dc0caab0 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2e2206640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2dc15d1d0 0x7fb2dc06f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2e1204640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc0caab0 0x7fb2dc06ed80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000c3d0 tx=0x7fb2d000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2da7fc640 1 -- 172.21.15.1:0/705777060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0017070 con 0x7fb2dc0caab0 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2da7fc640 1 -- 172.21.15.1:0/705777060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb2d00052d0 con 0x7fb2dc0caab0 2024-05-09T05:06:23.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2dc1f5fb0 con 0x7fb2dc0caab0 2024-05-09T05:06:23.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2da7fc640 1 -- 172.21.15.1:0/705777060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0003830 con 0x7fb2dc0caab0 2024-05-09T05:06:23.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.244+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2dc1f6830 con 0x7fb2dc0caab0 2024-05-09T05:06:23.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.245+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2dc1f62c0 con 0x7fb2dc0caab0 2024-05-09T05:06:23.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.246+0000 7fb2da7fc640 1 -- 172.21.15.1:0/705777060 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb2d001a070 con 0x7fb2dc0caab0 2024-05-09T05:06:23.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.246+0000 7fb2da7fc640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2c8076300 0x7fb2c8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:23.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.246+0000 7fb2da7fc640 1 -- 172.21.15.1:0/705777060 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb2d0096870 con 0x7fb2dc0caab0 2024-05-09T05:06:23.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.246+0000 7fb2e1a05640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2c8076300 0x7fb2c8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:23.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.247+0000 7fb2e1a05640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2c8076300 0x7fb2c8078810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4034000 tx=0x7fb2d402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:23.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.251+0000 7fb2da7fc640 1 -- 172.21.15.1:0/705777060 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb2d005fb80 con 0x7fb2dc0caab0 2024-05-09T05:06:23.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.480+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fb2dc050870 con 0x7fb2dc0caab0 2024-05-09T05:06:23.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.481+0000 7fb2da7fc640 1 -- 172.21.15.1:0/705777060 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7fb2d0063e80 con 0x7fb2dc0caab0 2024-05-09T05:06:23.481 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:23.481 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-09T05:06:23.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.482+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2c8076300 msgr2=0x7fb2c8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:23.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.482+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2c8076300 0x7fb2c8078810 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4034000 tx=0x7fb2d402e000 comp rx=0 tx=0).stop 2024-05-09T05:06:23.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.482+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc0caab0 msgr2=0x7fb2dc06ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:23.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.482+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc0caab0 0x7fb2dc06ed80 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000c3d0 tx=0x7fb2d000c890 comp rx=0 tx=0).stop 2024-05-09T05:06:23.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.482+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 shutdown_connections 2024-05-09T05:06:23.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.482+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb2c8076300 0x7fb2c8078810 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.483+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb2dc15d1d0 0x7fb2dc06f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.483+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb2dc0caab0 0x7fb2dc06ed80 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.483+0000 7fb2e3c69640 1 --2- 172.21.15.1:0/705777060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb2dc072120 0x7fb2dc15de90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:23.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.483+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 >> 172.21.15.1:0/705777060 conn(0x7fb2dc0731d0 msgr2=0x7fb2dc0cc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:23.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.483+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 shutdown_connections 2024-05-09T05:06:23.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:23.483+0000 7fb2e3c69640 1 -- 172.21.15.1:0/705777060 wait complete. 2024-05-09T05:06:26.495 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-09T05:06:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.622+0000 7ff334b0d640 1 Processor -- start 2024-05-09T05:06:26.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.622+0000 7ff334b0d640 1 -- start start 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff334b0d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff3300c6a30 0x7ff33015d0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff334b0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3301568b0 0x7ff3300ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff334b0d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3300cd2e0 con 0x7ff3300b4850 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff334b0d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff3300cd090 con 0x7ff3300c6a30 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff334b0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff3300cd5c0 con 0x7ff3301568b0 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff3300c6a30 0x7ff33015d0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff3300c6a30 0x7ff33015d0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55014/0 (socket says 172.21.15.1:55014) 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32e59c640 1 -- 172.21.15.1:0/1168514521 learned_addr learned my addr 172.21.15.1:0/1168514521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32dd9b640 1 --1- 172.21.15.1:0/1168514521 >> v1:172.21.15.196:6789/0 conn(0x7ff3300b4850 0x7ff3300d1bc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:40090/0 (socket says 172.21.15.1:40090) 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32ed9d640 1 --2- 172.21.15.1:0/1168514521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3301568b0 0x7ff3300ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32e59c640 1 -- 172.21.15.1:0/1168514521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3301568b0 msgr2=0x7ff3300ce990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:26.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32e59c640 1 --2- 172.21.15.1:0/1168514521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3301568b0 0x7ff3300ce990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32d59a640 1 -- 172.21.15.1:0/1168514521 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 674239164 0 0) 0x7ff3300cd2e0 con 0x7ff3300b4850 2024-05-09T05:06:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32e59c640 1 -- 172.21.15.1:0/1168514521 >> v1:172.21.15.196:6789/0 conn(0x7ff3300b4850 legacy=0x7ff3300d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:26.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.624+0000 7ff32e59c640 1 -- 172.21.15.1:0/1168514521 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3300cdeb0 con 0x7ff3300c6a30 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.625+0000 7ff32e59c640 1 --2- 172.21.15.1:0/1168514521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff33015d0f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff328009850 tx=0x7ff32802a2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de865bde13dcde90 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.625+0000 7ff32d59a640 1 -- 172.21.15.1:0/1168514521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328004230 con 0x7ff3300c6a30 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.625+0000 7ff32d59a640 1 -- 172.21.15.1:0/1168514521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3280043d0 con 0x7ff3300c6a30 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.625+0000 7ff32d59a640 1 -- 172.21.15.1:0/1168514521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff328003810 con 0x7ff3300c6a30 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.626+0000 7ff334b0d640 1 -- 172.21.15.1:0/1168514521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 msgr2=0x7ff33015d0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.626+0000 7ff334b0d640 1 --2- 172.21.15.1:0/1168514521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff33015d0f0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7ff328009850 tx=0x7ff32802a2c0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.626+0000 7ff334b0d640 1 -- 172.21.15.1:0/1168514521 shutdown_connections 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.626+0000 7ff334b0d640 1 --2- 172.21.15.1:0/1168514521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3301568b0 0x7ff3300ce990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.626+0000 7ff334b0d640 1 --2- 172.21.15.1:0/1168514521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff33015d0f0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.626+0000 7ff334b0d640 1 -- 172.21.15.1:0/1168514521 >> 172.21.15.1:0/1168514521 conn(0x7ff33006dc00 msgr2=0x7ff33006e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:26.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.626+0000 7ff334b0d640 1 -- 172.21.15.1:0/1168514521 shutdown_connections 2024-05-09T05:06:26.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.626+0000 7ff334b0d640 1 -- 172.21.15.1:0/1168514521 wait complete. 2024-05-09T05:06:26.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.628+0000 7ff334b0d640 1 Processor -- start 2024-05-09T05:06:26.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.628+0000 7ff334b0d640 1 -- start start 2024-05-09T05:06:26.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff334b0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3300b4850 0x7ff33015aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:26.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff334b0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff3300c7980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:26.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff334b0d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3301568b0 0x7ff33015fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:26.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff334b0d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff330156cc0 con 0x7ff3300c6a30 2024-05-09T05:06:26.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff334b0d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff3300ce470 con 0x7ff3301568b0 2024-05-09T05:06:26.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff334b0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff3300d22d0 con 0x7ff3300b4850 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff32dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff3300c7980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff32ed9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3301568b0 0x7ff33015fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff32dd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff3300c7980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:55018/0 (socket says 172.21.15.1:55018) 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff32dd9b640 1 -- 172.21.15.1:0/659085790 learned_addr learned my addr 172.21.15.1:0/659085790 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.629+0000 7ff32e59c640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3300b4850 0x7ff33015aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff32dd9b640 1 -- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3300b4850 msgr2=0x7ff33015aa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff32dd9b640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3300b4850 0x7ff33015aa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff32dd9b640 1 -- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3301568b0 msgr2=0x7ff33015fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff32dd9b640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3301568b0 0x7ff33015fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff32dd9b640 1 -- 172.21.15.1:0/659085790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff33015afd0 con 0x7ff3300c6a30 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff32ed9d640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3301568b0 0x7ff33015fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff32dd9b640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff3300c7980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff31800c3f0 tx=0x7ff31800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff326ffd640 1 -- 172.21.15.1:0/659085790 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3180074e0 con 0x7ff3300c6a30 2024-05-09T05:06:26.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff326ffd640 1 -- 172.21.15.1:0/659085790 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff318007680 con 0x7ff3300c6a30 2024-05-09T05:06:26.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3300c8440 con 0x7ff3300c6a30 2024-05-09T05:06:26.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff326ffd640 1 -- 172.21.15.1:0/659085790 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff318004a70 con 0x7ff3300c6a30 2024-05-09T05:06:26.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.630+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff330152220 con 0x7ff3300c6a30 2024-05-09T05:06:26.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.631+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff330050870 con 0x7ff3300c6a30 2024-05-09T05:06:26.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.632+0000 7ff326ffd640 1 -- 172.21.15.1:0/659085790 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff318011070 con 0x7ff3300c6a30 2024-05-09T05:06:26.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.632+0000 7ff326ffd640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff3100762d0 0x7ff3100787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:26.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.633+0000 7ff326ffd640 1 -- 172.21.15.1:0/659085790 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff3180970c0 con 0x7ff3300c6a30 2024-05-09T05:06:26.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.633+0000 7ff32e59c640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff3100762d0 0x7ff3100787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:26.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.633+0000 7ff32e59c640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff3100762d0 0x7ff3100787e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff328033000 tx=0x7ff32802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:26.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.637+0000 7ff326ffd640 1 -- 172.21.15.1:0/659085790 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff318060330 con 0x7ff3300c6a30 2024-05-09T05:06:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.867+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7ff3300b4630 con 0x7ff3300c6a30 2024-05-09T05:06:26.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.867+0000 7ff326ffd640 1 -- 172.21.15.1:0/659085790 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+196 (secure 0 0 0) 0x7ff318064630 con 0x7ff3300c6a30 2024-05-09T05:06:26.869 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:26.869 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_WARN","checks":{"MDS_UP_LESS_THAN_MAX":{"severity":"HEALTH_WARN","summary":{"message":"1 filesystem is online with fewer MDS than max_mds","count":1},"muted":false}},"mutes":[]} 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff3100762d0 msgr2=0x7ff3100787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff3100762d0 0x7ff3100787e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7ff328033000 tx=0x7ff32802e000 comp rx=0 tx=0).stop 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 msgr2=0x7ff3300c7980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff3300c7980 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff31800c3f0 tx=0x7ff31800c8b0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 shutdown_connections 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff3100762d0 0x7ff3100787e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3301568b0 0x7ff33015fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3300c6a30 0x7ff3300c7980 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 --2- 172.21.15.1:0/659085790 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3300b4850 0x7ff33015aa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 >> 172.21.15.1:0/659085790 conn(0x7ff33006dc00 msgr2=0x7ff3301525f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 shutdown_connections 2024-05-09T05:06:26.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:26.869+0000 7ff334b0d640 1 -- 172.21.15.1:0/659085790 wait complete. 2024-05-09T05:06:29.883 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2024-05-09T05:06:30.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.008+0000 7fe790ce9640 1 Processor -- start 2024-05-09T05:06:30.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.009+0000 7fe790ce9640 1 -- start start 2024-05-09T05:06:30.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe790ce9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe78c15d230 0x7fe78c0cb910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe790ce9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe78c0b38a0 0x7fe78c0b3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe790ce9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe78c06fe40 con 0x7fe78c070930 2024-05-09T05:06:30.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe790ce9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe78c0c4f30 con 0x7fe78c0b38a0 2024-05-09T05:06:30.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe790ce9640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe78c1606e0 con 0x7fe78c15d230 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe78a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe78c070930 0x7fe78c0c4820 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59892/0 (socket says 172.21.15.1:59892) 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe78a59c640 1 -- 172.21.15.1:0/475264154 learned_addr learned my addr 172.21.15.1:0/475264154 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe78ad9d640 1 --2- 172.21.15.1:0/475264154 >> v2:172.21.15.1:3300/0 conn(0x7fe78c0b38a0 0x7fe78c0b3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe789d9b640 1 --2- 172.21.15.1:0/475264154 >> v2:172.21.15.196:3300/0 conn(0x7fe78c15d230 0x7fe78c0cb910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.010+0000 7fe78959a640 1 -- 172.21.15.1:0/475264154 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 420134555 0 0) 0x7fe78c06fe40 con 0x7fe78c070930 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.011+0000 7fe78959a640 1 -- 172.21.15.1:0/475264154 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe778003520 con 0x7fe78c070930 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.011+0000 7fe78ad9d640 1 -- 172.21.15.1:0/475264154 >> v2:172.21.15.196:3300/0 conn(0x7fe78c15d230 msgr2=0x7fe78c0cb910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.011+0000 7fe78ad9d640 1 --2- 172.21.15.1:0/475264154 >> v2:172.21.15.196:3300/0 conn(0x7fe78c15d230 0x7fe78c0cb910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.011+0000 7fe78ad9d640 1 -- 172.21.15.1:0/475264154 >> v1:172.21.15.1:6789/0 conn(0x7fe78c070930 legacy=0x7fe78c0c4820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.011+0000 7fe78ad9d640 1 -- 172.21.15.1:0/475264154 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78c070120 con 0x7fe78c0b38a0 2024-05-09T05:06:30.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.011+0000 7fe78ad9d640 1 --2- 172.21.15.1:0/475264154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0b3cb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0749a0 tx=0x7fe77c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=554d90e0d9906223 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe78959a640 1 -- 172.21.15.1:0/475264154 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c037070 con 0x7fe78c0b38a0 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe78959a640 1 -- 172.21.15.1:0/475264154 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe77c032440 con 0x7fe78c0b38a0 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe78959a640 1 -- 172.21.15.1:0/475264154 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77c032810 con 0x7fe78c0b38a0 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe790ce9640 1 -- 172.21.15.1:0/475264154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 msgr2=0x7fe78c0b3cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe790ce9640 1 --2- 172.21.15.1:0/475264154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0b3cb0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fe78c0749a0 tx=0x7fe77c02a250 comp rx=0 tx=0).stop 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe790ce9640 1 -- 172.21.15.1:0/475264154 shutdown_connections 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe790ce9640 1 --2- 172.21.15.1:0/475264154 >> v2:172.21.15.196:3300/0 conn(0x7fe78c15d230 0x7fe78c0cb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe790ce9640 1 --2- 172.21.15.1:0/475264154 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0b3cb0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe790ce9640 1 -- 172.21.15.1:0/475264154 >> 172.21.15.1:0/475264154 conn(0x7fe78c071b40 msgr2=0x7fe78c071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe790ce9640 1 -- 172.21.15.1:0/475264154 shutdown_connections 2024-05-09T05:06:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.012+0000 7fe790ce9640 1 -- 172.21.15.1:0/475264154 wait complete. 2024-05-09T05:06:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.014+0000 7fe790ce9640 1 Processor -- start 2024-05-09T05:06:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.014+0000 7fe790ce9640 1 -- start start 2024-05-09T05:06:30.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe790ce9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe78c070930 0x7fe78c0c6340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe790ce9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0cea80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe790ce9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe78c15d230 0x7fe78c151f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe790ce9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c160370 con 0x7fe78c0b38a0 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe790ce9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe78c160090 con 0x7fe78c15d230 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe790ce9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe78c15d640 con 0x7fe78c070930 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0cea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0cea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51340/0 (socket says 172.21.15.1:51340) 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 -- 172.21.15.1:0/1951784401 learned_addr learned my addr 172.21.15.1:0/1951784401 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe78a59c640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe78c070930 0x7fe78c0c6340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe78ad9d640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe78c15d230 0x7fe78c151f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 -- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe78c070930 msgr2=0x7fe78c0c6340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe78c070930 0x7fe78c0c6340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 -- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe78c15d230 msgr2=0x7fe78c151f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe78c15d230 0x7fe78c151f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 -- 172.21.15.1:0/1951784401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe78c0c6880 con 0x7fe78c0b38a0 2024-05-09T05:06:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe789d9b640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0cea80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe77400e8a0 tx=0x7fe77400ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe782ffd640 1 -- 172.21.15.1:0/1951784401 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe774017070 con 0x7fe78c0b38a0 2024-05-09T05:06:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe782ffd640 1 -- 172.21.15.1:0/1951784401 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe77400c940 con 0x7fe78c0b38a0 2024-05-09T05:06:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe78c1f30f0 con 0x7fe78c0b38a0 2024-05-09T05:06:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.016+0000 7fe782ffd640 1 -- 172.21.15.1:0/1951784401 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe77400cd30 con 0x7fe78c0b38a0 2024-05-09T05:06:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.017+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe78c15b5f0 con 0x7fe78c0b38a0 2024-05-09T05:06:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.017+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe78c15b830 con 0x7fe78c0b38a0 2024-05-09T05:06:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.018+0000 7fe782ffd640 1 -- 172.21.15.1:0/1951784401 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe77401a070 con 0x7fe78c0b38a0 2024-05-09T05:06:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.019+0000 7fe782ffd640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe76c0762d0 0x7fe76c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.019+0000 7fe782ffd640 1 -- 172.21.15.1:0/1951784401 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fe774097470 con 0x7fe78c0b38a0 2024-05-09T05:06:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.019+0000 7fe78a59c640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe76c0762d0 0x7fe76c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.019+0000 7fe78a59c640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe76c0762d0 0x7fe76c0787e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe784005c00 tx=0x7fe784009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.024+0000 7fe782ffd640 1 -- 172.21.15.1:0/1951784401 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe774060780 con 0x7fe78c0b38a0 2024-05-09T05:06:30.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.253+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7fe78c0cc710 con 0x7fe78c0b38a0 2024-05-09T05:06:30.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.254+0000 7fe782ffd640 1 -- 172.21.15.1:0/1951784401 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7fe774064a80 con 0x7fe78c0b38a0 2024-05-09T05:06:30.255 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:30.255 INFO:teuthology.orchestra.run.smithi001.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-09T05:06:30.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.255+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe76c0762d0 msgr2=0x7fe76c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.255+0000 7fe790ce9640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe76c0762d0 0x7fe76c0787e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe784005c00 tx=0x7fe784009040 comp rx=0 tx=0).stop 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.255+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 msgr2=0x7fe78c0cea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.255+0000 7fe790ce9640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0cea80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fe77400e8a0 tx=0x7fe77400ed60 comp rx=0 tx=0).stop 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.256+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 shutdown_connections 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.256+0000 7fe790ce9640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe76c0762d0 0x7fe76c0787e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.256+0000 7fe790ce9640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe78c15d230 0x7fe78c151f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.256+0000 7fe790ce9640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe78c0b38a0 0x7fe78c0cea80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.256+0000 7fe790ce9640 1 --2- 172.21.15.1:0/1951784401 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe78c070930 0x7fe78c0c6340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.256+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 >> 172.21.15.1:0/1951784401 conn(0x7fe78c071b40 msgr2=0x7fe78c0c9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.256+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 shutdown_connections 2024-05-09T05:06:30.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.256+0000 7fe790ce9640 1 -- 172.21.15.1:0/1951784401 wait complete. 2024-05-09T05:06:30.268 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2024-05-09T05:06:30.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:06:30.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.381+0000 7fd79f661640 1 Processor -- start 2024-05-09T05:06:30.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.381+0000 7fd79f661640 1 -- start start 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.382+0000 7fd79f661640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd798156b70 0x7fd79806c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.382+0000 7fd79f661640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7980c95c0 con 0x7fd7980c3b40 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.382+0000 7fd79f661640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd798071ed0 con 0x7fd79806cc30 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.382+0000 7fd79f661640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd79815dbe0 con 0x7fd798156b70 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd79d3fd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd7980c3b40 0x7fd79815ce20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59898/0 (socket says 172.21.15.1:59898) 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd79d3fd640 1 -- 172.21.15.1:0/3732025218 learned_addr learned my addr 172.21.15.1:0/3732025218 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd79cbfc640 1 --2- 172.21.15.1:0/3732025218 >> v2:172.21.15.196:3300/0 conn(0x7fd798156b70 0x7fd79806c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd78ffff640 1 -- 172.21.15.1:0/3732025218 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 392123879 0 0) 0x7fd7980c95c0 con 0x7fd7980c3b40 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd78ffff640 1 -- 172.21.15.1:0/3732025218 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd784003520 con 0x7fd7980c3b40 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd78ffff640 1 -- 172.21.15.1:0/3732025218 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3975937707 0 0) 0x7fd798071ed0 con 0x7fd79806cc30 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd78ffff640 1 -- 172.21.15.1:0/3732025218 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7980c95c0 con 0x7fd79806cc30 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd79cbfc640 1 -- 172.21.15.1:0/3732025218 >> v1:172.21.15.196:6789/0 conn(0x7fd79806cc30 legacy=0x7fd798167070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd79cbfc640 1 -- 172.21.15.1:0/3732025218 >> v1:172.21.15.1:6789/0 conn(0x7fd7980c3b40 legacy=0x7fd79815ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.383+0000 7fd79cbfc640 1 -- 172.21.15.1:0/3732025218 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd798071bf0 con 0x7fd798156b70 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.384+0000 7fd79cbfc640 1 --2- 172.21.15.1:0/3732025218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 0x7fd79806c6f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd7880097a0 tx=0x7fd78802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e0559136a211a95 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.384+0000 7fd78ffff640 1 -- 172.21.15.1:0/3732025218 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788037070 con 0x7fd798156b70 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.384+0000 7fd78ffff640 1 -- 172.21.15.1:0/3732025218 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd788032440 con 0x7fd798156b70 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.384+0000 7fd78ffff640 1 -- 172.21.15.1:0/3732025218 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd788032810 con 0x7fd798156b70 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.385+0000 7fd79f661640 1 -- 172.21.15.1:0/3732025218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 msgr2=0x7fd79806c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.385+0000 7fd79f661640 1 --2- 172.21.15.1:0/3732025218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 0x7fd79806c6f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd7880097a0 tx=0x7fd78802a230 comp rx=0 tx=0).stop 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.385+0000 7fd79f661640 1 -- 172.21.15.1:0/3732025218 shutdown_connections 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.385+0000 7fd79f661640 1 --2- 172.21.15.1:0/3732025218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 0x7fd79806c6f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.385+0000 7fd79f661640 1 -- 172.21.15.1:0/3732025218 >> 172.21.15.1:0/3732025218 conn(0x7fd79806de30 msgr2=0x7fd798073eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.385+0000 7fd79f661640 1 -- 172.21.15.1:0/3732025218 shutdown_connections 2024-05-09T05:06:30.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.385+0000 7fd79f661640 1 -- 172.21.15.1:0/3732025218 wait complete. 2024-05-09T05:06:30.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.387+0000 7fd79f661640 1 Processor -- start 2024-05-09T05:06:30.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.387+0000 7fd79f661640 1 -- start start 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79f661640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd79806cc30 0x7fd798154870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79f661640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7980c3b40 0x7fd798152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79f661640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 0x7fd798152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79f661640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd7980718b0 con 0x7fd79806cc30 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79f661640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd79815c900 con 0x7fd798156b70 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79f661640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd79815d530 con 0x7fd7980c3b40 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd79806cc30 0x7fd798154870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd79806cc30 0x7fd798154870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51354/0 (socket says 172.21.15.1:51354) 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 -- 172.21.15.1:0/2909625664 learned_addr learned my addr 172.21.15.1:0/2909625664 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79cbfc640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7980c3b40 0x7fd798152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79dbfe640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 0x7fd798152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 -- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7980c3b40 msgr2=0x7fd798152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7980c3b40 0x7fd798152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 -- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 msgr2=0x7fd798152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 0x7fd798152b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 -- 172.21.15.1:0/2909625664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79815b4f0 con 0x7fd79806cc30 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79cbfc640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7980c3b40 0x7fd798152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.388+0000 7fd79d3fd640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd79806cc30 0x7fd798154870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c410 tx=0x7fd79400c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.389+0000 7fd78dffb640 1 -- 172.21.15.1:0/2909625664 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd794017070 con 0x7fd79806cc30 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.389+0000 7fd78dffb640 1 -- 172.21.15.1:0/2909625664 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd794005da0 con 0x7fd79806cc30 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.389+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd798154db0 con 0x7fd79806cc30 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.389+0000 7fd78dffb640 1 -- 172.21.15.1:0/2909625664 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7940072f0 con 0x7fd79806cc30 2024-05-09T05:06:30.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.389+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79815b7f0 con 0x7fd79806cc30 2024-05-09T05:06:30.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.390+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79806a3c0 con 0x7fd79806cc30 2024-05-09T05:06:30.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.391+0000 7fd78dffb640 1 -- 172.21.15.1:0/2909625664 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd79401a070 con 0x7fd79806cc30 2024-05-09T05:06:30.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.391+0000 7fd78dffb640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd77c0762d0 0x7fd77c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.391+0000 7fd78dffb640 1 -- 172.21.15.1:0/2909625664 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd7940969e0 con 0x7fd79806cc30 2024-05-09T05:06:30.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.391+0000 7fd79cbfc640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd77c0762d0 0x7fd77c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.392+0000 7fd79cbfc640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd77c0762d0 0x7fd77c0787e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd788034000 tx=0x7fd78802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.396+0000 7fd78dffb640 1 -- 172.21.15.1:0/2909625664 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd79806a3c0 con 0x7fd79806cc30 2024-05-09T05:06:30.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.624+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd798050870 con 0x7fd79806cc30 2024-05-09T05:06:30.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.624+0000 7fd78dffb640 1 -- 172.21.15.1:0/2909625664 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6911 (secure 0 0 0) 0x7fd79405fc60 con 0x7fd79806cc30 2024-05-09T05:06:30.626 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:30.626 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:28.149589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34160,"mds_1":34157,"mds_2":4311,"mds_3":4287,"mds_4":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4311":{"gid":4311,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[34160,34157,4311,4287]},"id":1}]} 2024-05-09T05:06:30.626 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 15 2024-05-09T05:06:30.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.626+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd77c0762d0 msgr2=0x7fd77c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.626+0000 7fd79f661640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd77c0762d0 0x7fd77c0787e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fd788034000 tx=0x7fd78802e000 comp rx=0 tx=0).stop 2024-05-09T05:06:30.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.626+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd79806cc30 msgr2=0x7fd798154870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.626+0000 7fd79f661640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd79806cc30 0x7fd798154870 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fd79400c410 tx=0x7fd79400c8d0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.627+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 shutdown_connections 2024-05-09T05:06:30.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.627+0000 7fd79f661640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd77c0762d0 0x7fd77c0787e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.627+0000 7fd79f661640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd798156b70 0x7fd798152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.627+0000 7fd79f661640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7980c3b40 0x7fd798152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.627+0000 7fd79f661640 1 --2- 172.21.15.1:0/2909625664 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd79806cc30 0x7fd798154870 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.627+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 >> 172.21.15.1:0/2909625664 conn(0x7fd79806de30 msgr2=0x7fd7980c55b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:30.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.627+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 shutdown_connections 2024-05-09T05:06:30.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.627+0000 7fd79f661640 1 -- 172.21.15.1:0/2909625664 wait complete. 2024-05-09T05:06:30.639 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:06:30.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.752+0000 7f744338d640 1 Processor -- start 2024-05-09T05:06:30.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.752+0000 7f744338d640 1 -- start start 2024-05-09T05:06:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f744338d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f743c072120 0x7f743c0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f744338d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f743c0c62e0 0x7f743c070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f744338d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f743c15b110 con 0x7f743c0ca340 2024-05-09T05:06:30.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f744338d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f743c0c53f0 con 0x7f743c0c62e0 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f744338d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f743c0c55b0 con 0x7f743c072120 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f744192a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f743c0c62e0 0x7f743c070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f7441129640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f743c0ca340 0x7f743c1547b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59900/0 (socket says 172.21.15.1:59900) 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f744192a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f743c0c62e0 0x7f743c070e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51366/0 (socket says 172.21.15.1:51366) 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f7441129640 1 -- 172.21.15.1:0/984569572 learned_addr learned my addr 172.21.15.1:0/984569572 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.753+0000 7f7440928640 1 --2- 172.21.15.1:0/984569572 >> v2:172.21.15.196:3300/0 conn(0x7f743c072120 0x7f743c0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.754+0000 7f744192a640 1 -- 172.21.15.1:0/984569572 >> v2:172.21.15.196:3300/0 conn(0x7f743c072120 msgr2=0x7f743c0c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.754+0000 7f744192a640 1 --2- 172.21.15.1:0/984569572 >> v2:172.21.15.196:3300/0 conn(0x7f743c072120 0x7f743c0c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.754+0000 7f744192a640 1 -- 172.21.15.1:0/984569572 >> v1:172.21.15.1:6789/0 conn(0x7f743c0ca340 legacy=0x7f743c1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.754+0000 7f744192a640 1 -- 172.21.15.1:0/984569572 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743c15b3b0 con 0x7f743c0c62e0 2024-05-09T05:06:30.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.754+0000 7f7433fff640 1 -- 172.21.15.1:0/984569572 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4170629160 0 0) 0x7f743c15b110 con 0x7f743c0ca340 2024-05-09T05:06:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.754+0000 7f744192a640 1 --2- 172.21.15.1:0/984569572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c62e0 0x7f743c070e20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f743c0b37a0 tx=0x7f743402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9463f76b8c83eb54 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f7433fff640 1 -- 172.21.15.1:0/984569572 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434007740 con 0x7f743c0c62e0 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f7433fff640 1 -- 172.21.15.1:0/984569572 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74340078e0 con 0x7f743c0c62e0 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f7433fff640 1 -- 172.21.15.1:0/984569572 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7434007cb0 con 0x7f743c0c62e0 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f744338d640 1 -- 172.21.15.1:0/984569572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c62e0 msgr2=0x7f743c070e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f744338d640 1 --2- 172.21.15.1:0/984569572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c62e0 0x7f743c070e20 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f743c0b37a0 tx=0x7f743402a5f0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f744338d640 1 -- 172.21.15.1:0/984569572 shutdown_connections 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f744338d640 1 --2- 172.21.15.1:0/984569572 >> v2:172.21.15.196:3300/0 conn(0x7f743c072120 0x7f743c0c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f744338d640 1 --2- 172.21.15.1:0/984569572 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c0c62e0 0x7f743c070e20 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f744338d640 1 -- 172.21.15.1:0/984569572 >> 172.21.15.1:0/984569572 conn(0x7f743c0731d0 msgr2=0x7f743c0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f744338d640 1 -- 172.21.15.1:0/984569572 shutdown_connections 2024-05-09T05:06:30.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.755+0000 7f744338d640 1 -- 172.21.15.1:0/984569572 wait complete. 2024-05-09T05:06:30.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.757+0000 7f744338d640 1 Processor -- start 2024-05-09T05:06:30.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.757+0000 7f744338d640 1 -- start start 2024-05-09T05:06:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.758+0000 7f744338d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c072120 0x7f743c1e5b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.758+0000 7f744338d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f743c0c62e0 0x7f743c0ce970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.758+0000 7f744338d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f743c0ca340 0x7f743c1f4140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.758+0000 7f744338d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f743c0c5dc0 con 0x7f743c072120 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.758+0000 7f744338d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f743c0c5ae0 con 0x7f743c0ca340 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.758+0000 7f744338d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f743c0c66f0 con 0x7f743c0c62e0 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c072120 0x7f743c1e5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c072120 0x7f743c1e5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51380/0 (socket says 172.21.15.1:51380) 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 -- 172.21.15.1:0/2878500263 learned_addr learned my addr 172.21.15.1:0/2878500263 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f744192a640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f743c0ca340 0x7f743c1f4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7440928640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f743c0c62e0 0x7f743c0ce970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 -- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f743c0c62e0 msgr2=0x7f743c0ce970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f743c0c62e0 0x7f743c0ce970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 -- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f743c0ca340 msgr2=0x7f743c1f4140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:30.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f743c0ca340 0x7f743c1f4140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 -- 172.21.15.1:0/2878500263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f743c1e43b0 con 0x7f743c072120 2024-05-09T05:06:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7441129640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c072120 0x7f743c1e5b40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f743800c410 tx=0x7f743800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7431ffb640 1 -- 172.21.15.1:0/2878500263 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74380074e0 con 0x7f743c072120 2024-05-09T05:06:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7431ffb640 1 -- 172.21.15.1:0/2878500263 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f74380047e0 con 0x7f743c072120 2024-05-09T05:06:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f743c1e6080 con 0x7f743c072120 2024-05-09T05:06:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f7431ffb640 1 -- 172.21.15.1:0/2878500263 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7438004bd0 con 0x7f743c072120 2024-05-09T05:06:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.759+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f743c1e3df0 con 0x7f743c072120 2024-05-09T05:06:30.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.760+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f743c050870 con 0x7f743c072120 2024-05-09T05:06:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.761+0000 7f7431ffb640 1 -- 172.21.15.1:0/2878500263 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7438011070 con 0x7f743c072120 2024-05-09T05:06:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.762+0000 7f7431ffb640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7420076300 0x7f7420078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.762+0000 7f7431ffb640 1 -- 172.21.15.1:0/2878500263 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f74380971e0 con 0x7f743c072120 2024-05-09T05:06:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.762+0000 7f7440928640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7420076300 0x7f7420078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:30.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.762+0000 7f7440928640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7420076300 0x7f7420078810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f743c1f17f0 tx=0x7f742c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:30.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.767+0000 7f7431ffb640 1 -- 172.21.15.1:0/2878500263 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f74380603c0 con 0x7f743c072120 2024-05-09T05:06:30.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.995+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f743c0cba10 con 0x7f743c072120 2024-05-09T05:06:30.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.996+0000 7f7431ffb640 1 -- 172.21.15.1:0/2878500263 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6911 (secure 0 0 0) 0x7f74380646c0 con 0x7f743c072120 2024-05-09T05:06:30.997 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:30.997 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:28.149589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34160,"mds_1":34157,"mds_2":4311,"mds_3":4287,"mds_4":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4311":{"gid":4311,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[34160,34157,4311,4287]},"id":1}]} 2024-05-09T05:06:30.998 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 15 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7420076300 msgr2=0x7f7420078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7420076300 0x7f7420078810 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f743c1f17f0 tx=0x7f742c009190 comp rx=0 tx=0).stop 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c072120 msgr2=0x7f743c1e5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c072120 0x7f743c1e5b40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f743800c410 tx=0x7f743800c8d0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 shutdown_connections 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7420076300 0x7f7420078810 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f743c0ca340 0x7f743c1f4140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f743c0c62e0 0x7f743c0ce970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 --2- 172.21.15.1:0/2878500263 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f743c072120 0x7f743c1e5b40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 >> 172.21.15.1:0/2878500263 conn(0x7f743c0731d0 msgr2=0x7f743c158890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 shutdown_connections 2024-05-09T05:06:31.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:30.998+0000 7f744338d640 1 -- 172.21.15.1:0/2878500263 wait complete. 2024-05-09T05:06:31.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:06:31.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.123+0000 7f82043b2640 1 Processor -- start 2024-05-09T05:06:31.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.123+0000 7f82043b2640 1 -- start start 2024-05-09T05:06:31.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f82043b2640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f81fc15d0f0 0x7f81fc0b4850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f82043b2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81fc1568b0 0x7f81fc0ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f82043b2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81fc0cd2e0 con 0x7f81fc0c6a30 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f82043b2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f81fc0cd090 con 0x7f81fc1568b0 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f82043b2640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f81fc0cd5c0 con 0x7f81fc15d0f0 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820294f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81fc1568b0 0x7f81fc0ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820294f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f81fc1568b0 0x7f81fc0ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51388/0 (socket says 172.21.15.1:51388) 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820294f640 1 -- 172.21.15.1:0/3009194778 learned_addr learned my addr 172.21.15.1:0/3009194778 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820194d640 1 --2- 172.21.15.1:0/3009194778 >> v2:172.21.15.196:3300/0 conn(0x7f81fc15d0f0 0x7f81fc0b4850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820114c640 1 -- 172.21.15.1:0/3009194778 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2331294340 0 0) 0x7f81fc0cd2e0 con 0x7f81fc0c6a30 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820294f640 1 -- 172.21.15.1:0/3009194778 >> v2:172.21.15.196:3300/0 conn(0x7f81fc15d0f0 msgr2=0x7f81fc0b4850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820294f640 1 --2- 172.21.15.1:0/3009194778 >> v2:172.21.15.196:3300/0 conn(0x7f81fc15d0f0 0x7f81fc0b4850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820294f640 1 -- 172.21.15.1:0/3009194778 >> v1:172.21.15.196:6789/0 conn(0x7f81fc0c6a30 legacy=0x7f81fc0d1bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.125+0000 7f820294f640 1 -- 172.21.15.1:0/3009194778 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc0cdeb0 con 0x7f81fc1568b0 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.126+0000 7f820294f640 1 --2- 172.21.15.1:0/3009194778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc1568b0 0x7f81fc0ce990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0d2c50 tx=0x7f81ec02a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8377e68d5ba51e27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.126+0000 7f820114c640 1 -- 172.21.15.1:0/3009194778 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ec036070 con 0x7f81fc1568b0 2024-05-09T05:06:31.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.126+0000 7f820114c640 1 -- 172.21.15.1:0/3009194778 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81ec031600 con 0x7f81fc1568b0 2024-05-09T05:06:31.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.126+0000 7f820114c640 1 -- 172.21.15.1:0/3009194778 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81ec0319f0 con 0x7f81fc1568b0 2024-05-09T05:06:31.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.127+0000 7f82043b2640 1 -- 172.21.15.1:0/3009194778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc1568b0 msgr2=0x7f81fc0ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.127+0000 7f82043b2640 1 --2- 172.21.15.1:0/3009194778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc1568b0 0x7f81fc0ce990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f81fc0d2c50 tx=0x7f81ec02a7c0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.127+0000 7f82043b2640 1 -- 172.21.15.1:0/3009194778 shutdown_connections 2024-05-09T05:06:31.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.127+0000 7f82043b2640 1 --2- 172.21.15.1:0/3009194778 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc1568b0 0x7f81fc0ce990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.127+0000 7f82043b2640 1 --2- 172.21.15.1:0/3009194778 >> v2:172.21.15.196:3300/0 conn(0x7f81fc15d0f0 0x7f81fc0b4850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.127+0000 7f82043b2640 1 -- 172.21.15.1:0/3009194778 >> 172.21.15.1:0/3009194778 conn(0x7f81fc06dc00 msgr2=0x7f81fc06e010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:31.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.127+0000 7f82043b2640 1 -- 172.21.15.1:0/3009194778 shutdown_connections 2024-05-09T05:06:31.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.127+0000 7f82043b2640 1 -- 172.21.15.1:0/3009194778 wait complete. 2024-05-09T05:06:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.129+0000 7f82043b2640 1 Processor -- start 2024-05-09T05:06:31.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.129+0000 7f82043b2640 1 -- start start 2024-05-09T05:06:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f82043b2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc0c6a30 0x7f81fc15b400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f82043b2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81fc1568b0 0x7f81fc0d03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f82043b2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81fc15d0f0 0x7f81fc0d0930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f82043b2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc156cc0 con 0x7f81fc0c6a30 2024-05-09T05:06:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f82043b2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f81fc0ce470 con 0x7f81fc1568b0 2024-05-09T05:06:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f82043b2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f81fc0d22d0 con 0x7f81fc15d0f0 2024-05-09T05:06:31.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f820214e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc0c6a30 0x7f81fc15b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f820214e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc0c6a30 0x7f81fc15b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51392/0 (socket says 172.21.15.1:51392) 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f820214e640 1 -- 172.21.15.1:0/1478792318 learned_addr learned my addr 172.21.15.1:0/1478792318 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f820194d640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81fc1568b0 0x7f81fc0d03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.130+0000 7f820294f640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81fc15d0f0 0x7f81fc0d0930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f820214e640 1 -- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81fc15d0f0 msgr2=0x7f81fc0d0930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f820214e640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81fc15d0f0 0x7f81fc0d0930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f820214e640 1 -- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81fc1568b0 msgr2=0x7f81fc0d03f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f820214e640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81fc1568b0 0x7f81fc0d03f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f820214e640 1 -- 172.21.15.1:0/1478792318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81fc15be10 con 0x7f81fc0c6a30 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f820214e640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc0c6a30 0x7f81fc15b400 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c470 tx=0x7f81f400c930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f81faffd640 1 -- 172.21.15.1:0/1478792318 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f4017070 con 0x7f81fc0c6a30 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f81faffd640 1 -- 172.21.15.1:0/1478792318 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81f4003640 con 0x7f81fc0c6a30 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81fc15b940 con 0x7f81fc0c6a30 2024-05-09T05:06:31.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f81faffd640 1 -- 172.21.15.1:0/1478792318 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81f40054e0 con 0x7f81fc0c6a30 2024-05-09T05:06:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.131+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81fc15a150 con 0x7f81fc0c6a30 2024-05-09T05:06:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.132+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f81fc0763f0 con 0x7f81fc0c6a30 2024-05-09T05:06:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.133+0000 7f81faffd640 1 -- 172.21.15.1:0/1478792318 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f81f401a070 con 0x7f81fc0c6a30 2024-05-09T05:06:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.133+0000 7f81faffd640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81e80762d0 0x7f81e80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.134+0000 7f81faffd640 1 -- 172.21.15.1:0/1478792318 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f81f40967c0 con 0x7f81fc0c6a30 2024-05-09T05:06:31.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.134+0000 7f820194d640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81e80762d0 0x7f81e80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.134+0000 7f820194d640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81e80762d0 0x7f81e80787e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f81fc154080 tx=0x7f81f0009110 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:31.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.138+0000 7f81faffd640 1 -- 172.21.15.1:0/1478792318 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f81fc0763f0 con 0x7f81fc0c6a30 2024-05-09T05:06:31.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.329+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f81fc050870 con 0x7f81fc0c6a30 2024-05-09T05:06:31.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.330+0000 7f81faffd640 1 -- 172.21.15.1:0/1478792318 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7f81f405fb60 con 0x7f81fc0c6a30 2024-05-09T05:06:31.332 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:31.332 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:06:06.049536+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:06:31.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81e80762d0 msgr2=0x7f81e80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81e80762d0 0x7f81e80787e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f81fc154080 tx=0x7f81f0009110 comp rx=0 tx=0).stop 2024-05-09T05:06:31.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc0c6a30 msgr2=0x7f81fc15b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc0c6a30 0x7f81fc15b400 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f81f400c470 tx=0x7f81f400c930 comp rx=0 tx=0).stop 2024-05-09T05:06:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 shutdown_connections 2024-05-09T05:06:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81e80762d0 0x7f81e80787e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81fc1568b0 0x7f81fc0d03f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81fc15d0f0 0x7f81fc0d0930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 --2- 172.21.15.1:0/1478792318 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f81fc0c6a30 0x7f81fc15b400 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 >> 172.21.15.1:0/1478792318 conn(0x7f81fc06dc00 msgr2=0x7f81fc0b6080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 shutdown_connections 2024-05-09T05:06:31.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.332+0000 7f82043b2640 1 -- 172.21.15.1:0/1478792318 wait complete. 2024-05-09T05:06:31.345 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:06:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.458+0000 7f6a1155f640 1 Processor -- start 2024-05-09T05:06:31.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.458+0000 7f6a1155f640 1 -- start start 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a1155f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a0c0ca9b0 0x7f6a0c15e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a1155f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c154850 0x7f6a0c15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a1155f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f6a0c0c7eb0 0x7f6a0c0c82c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a1155f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6a0c15ec50 con 0x7f6a0c0ca9b0 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a1155f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f6a0c0cb060 con 0x7f6a0c0c7eb0 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a1155f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6a0c15a820 con 0x7f6a0c154850 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a0c0ca9b0 0x7f6a0c15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6a0c0ca9b0 0x7f6a0c15e710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51396/0 (socket says 172.21.15.1:51396) 2024-05-09T05:06:31.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0affd640 1 -- 172.21.15.1:0/3592097995 learned_addr learned my addr 172.21.15.1:0/3592097995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0a7fc640 1 --2- 172.21.15.1:0/3592097995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c154850 0x7f6a0c15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0b7fe640 1 --2- 172.21.15.1:0/3592097995 >> v2:172.21.15.196:3300/0 conn(0x7f6a0c0c7eb0 0x7f6a0c0c82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0affd640 1 -- 172.21.15.1:0/3592097995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c154850 msgr2=0x7f6a0c15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0affd640 1 --2- 172.21.15.1:0/3592097995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c154850 0x7f6a0c15afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0affd640 1 -- 172.21.15.1:0/3592097995 >> v2:172.21.15.196:3300/0 conn(0x7f6a0c0c7eb0 msgr2=0x7f6a0c0c82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.459+0000 7f6a0affd640 1 --2- 172.21.15.1:0/3592097995 >> v2:172.21.15.196:3300/0 conn(0x7f6a0c0c7eb0 0x7f6a0c0c82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.460+0000 7f6a0affd640 1 -- 172.21.15.1:0/3592097995 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0c0cadc0 con 0x7f6a0c0ca9b0 2024-05-09T05:06:31.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.460+0000 7f6a0affd640 1 --2- 172.21.15.1:0/3592097995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c0ca9b0 0x7f6a0c15e710 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0c5a10 tx=0x7f69fc02a140 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d261ed64ecd800 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:31.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.460+0000 7f6a09ffb640 1 -- 172.21.15.1:0/3592097995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc037070 con 0x7f6a0c0ca9b0 2024-05-09T05:06:31.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.460+0000 7f6a09ffb640 1 -- 172.21.15.1:0/3592097995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69fc032440 con 0x7f6a0c0ca9b0 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.460+0000 7f6a09ffb640 1 -- 172.21.15.1:0/3592097995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f69fc032810 con 0x7f6a0c0ca9b0 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 -- 172.21.15.1:0/3592097995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c0ca9b0 msgr2=0x7f6a0c15e710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3592097995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c0ca9b0 0x7f6a0c15e710 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6a0c0c5a10 tx=0x7f69fc02a140 comp rx=0 tx=0).stop 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 -- 172.21.15.1:0/3592097995 shutdown_connections 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3592097995 >> v2:172.21.15.196:3300/0 conn(0x7f6a0c0c7eb0 0x7f6a0c0c82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3592097995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c154850 0x7f6a0c15afe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3592097995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c0ca9b0 0x7f6a0c15e710 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 -- 172.21.15.1:0/3592097995 >> 172.21.15.1:0/3592097995 conn(0x7f6a0c0745a0 msgr2=0x7f6a0c071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 -- 172.21.15.1:0/3592097995 shutdown_connections 2024-05-09T05:06:31.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.461+0000 7f6a1155f640 1 -- 172.21.15.1:0/3592097995 wait complete. 2024-05-09T05:06:31.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.463+0000 7f6a1155f640 1 Processor -- start 2024-05-09T05:06:31.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.463+0000 7f6a1155f640 1 -- start start 2024-05-09T05:06:31.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a1155f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a0c0c7eb0 0x7f6a0c14d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a1155f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c0ca9b0 0x7f6a0c145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a1155f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c154850 0x7f6a0c14f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a1155f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c15aac0 con 0x7f6a0c154850 2024-05-09T05:06:31.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a1155f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6a0c0c9810 con 0x7f6a0c0c7eb0 2024-05-09T05:06:31.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a1155f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6a0c0c9af0 con 0x7f6a0c0ca9b0 2024-05-09T05:06:31.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a0b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c154850 0x7f6a0c14f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a0b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c154850 0x7f6a0c14f2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51410/0 (socket says 172.21.15.1:51410) 2024-05-09T05:06:31.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.464+0000 7f6a0b7fe640 1 -- 172.21.15.1:0/3915432369 learned_addr learned my addr 172.21.15.1:0/3915432369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:31.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0affd640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a0c0c7eb0 0x7f6a0c14d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0a7fc640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c0ca9b0 0x7f6a0c145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0b7fe640 1 -- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c0ca9b0 msgr2=0x7f6a0c145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0b7fe640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c0ca9b0 0x7f6a0c145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0b7fe640 1 -- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a0c0c7eb0 msgr2=0x7f6a0c14d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0b7fe640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a0c0c7eb0 0x7f6a0c14d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0b7fe640 1 -- 172.21.15.1:0/3915432369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a0c0c6120 con 0x7f6a0c154850 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0affd640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a0c0c7eb0 0x7f6a0c14d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a0b7fe640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c154850 0x7f6a0c14f2b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a0400eed0 tx=0x7f6a0400b480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a037fe640 1 -- 172.21.15.1:0/3915432369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a04018020 con 0x7f6a0c154850 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a037fe640 1 -- 172.21.15.1:0/3915432369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6a0400bcd0 con 0x7f6a0c154850 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a037fe640 1 -- 172.21.15.1:0/3915432369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a04014640 con 0x7f6a0c154850 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a0c1f2f50 con 0x7f6a0c154850 2024-05-09T05:06:31.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.465+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a0c1f3510 con 0x7f6a0c154850 2024-05-09T05:06:31.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.466+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a0c050870 con 0x7f6a0c154850 2024-05-09T05:06:31.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.467+0000 7f6a037fe640 1 -- 172.21.15.1:0/3915432369 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6a04005d10 con 0x7f6a0c154850 2024-05-09T05:06:31.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.467+0000 7f6a037fe640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69ec076300 0x7f69ec078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.468+0000 7f6a037fe640 1 -- 172.21.15.1:0/3915432369 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f6a04097c10 con 0x7f6a0c154850 2024-05-09T05:06:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.468+0000 7f6a0affd640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69ec076300 0x7f69ec078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:31.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.468+0000 7f6a0affd640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69ec076300 0x7f69ec078810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f69fc034000 tx=0x7f69fc02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:31.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.472+0000 7f6a037fe640 1 -- 172.21.15.1:0/3915432369 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6a04022030 con 0x7f6a0c154850 2024-05-09T05:06:31.544 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:06:31.547 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:06:31.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.704+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6a0c158130 con 0x7f6a0c154850 2024-05-09T05:06:31.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.705+0000 7f6a037fe640 1 -- 172.21.15.1:0/3915432369 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) ==== 94+0+6911 (secure 0 0 0) 0x7f6a04060fb0 con 0x7f6a0c154850 2024-05-09T05:06:31.707 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:31.707 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":15,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":15,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:28.149589+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34160,"mds_1":34157,"mds_2":4311,"mds_3":4287,"mds_4":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":4,"incarnation":15,"state":"up:creating","state_seq":1,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4311":{"gid":4311,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[34160,34157,4311,4287]},"id":1}]} 2024-05-09T05:06:31.707 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 15 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69ec076300 msgr2=0x7f69ec078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69ec076300 0x7f69ec078810 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f69fc034000 tx=0x7f69fc02e000 comp rx=0 tx=0).stop 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c154850 msgr2=0x7f6a0c14f2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c154850 0x7f6a0c14f2b0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6a0400eed0 tx=0x7f6a0400b480 comp rx=0 tx=0).stop 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 shutdown_connections 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69ec076300 0x7f69ec078810 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6a0c154850 0x7f6a0c14f2b0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6a0c0ca9b0 0x7f6a0c145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 --2- 172.21.15.1:0/3915432369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6a0c0c7eb0 0x7f6a0c14d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:31.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 >> 172.21.15.1:0/3915432369 conn(0x7f6a0c0745a0 msgr2=0x7f6a0c158f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:31.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 shutdown_connections 2024-05-09T05:06:31.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:31.707+0000 7f6a1155f640 1 -- 172.21.15.1:0/3915432369 wait complete. 2024-05-09T05:06:32.325 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 15, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T05:06:04.442853+0000', 'modified': '2024-05-09T05:06:28.149589+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 34160, 'mds_1': 34157, 'mds_2': 4311, 'mds_3': 4287, 'mds_4': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'e', 'rank': 4, 'incarnation': 15, 'state': 'up:creating', 'state_seq': 1, 'addr': '172.21.15.1:6837/3883243235', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6836', 'nonce': 3883243235}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 3883243235}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4287': {'gid': 4287, 'name': 'c', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.1:6834/1980556788', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1980556788}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1980556788}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4311': {'gid': 4311, 'name': 'd', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.196:6839/2336267187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6838', 'nonce': 2336267187}, {'type': 'v1', 'addr': '172.21.15.196:6839', 'nonce': 2336267187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34157': {'gid': 34157, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.196:6835/1932237812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6834', 'nonce': 1932237812}, {'type': 'v1', 'addr': '172.21.15.196:6835', 'nonce': 1932237812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34160': {'gid': 34160, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.196:6837/3384416671', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6836', 'nonce': 3384416671}, {'type': 'v1', 'addr': '172.21.15.196:6837', 'nonce': 3384416671}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34160, 'qdb_cluster': [34160, 34157, 4311, 4287]} 2024-05-09T05:06:32.325 WARNING:tasks.cephfs.filesystem:Unhealthy mds state gid_4273:up:creating 2024-05-09T05:06:33.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:06:33.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.455+0000 7f8da61e3640 1 Processor -- start 2024-05-09T05:06:33.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.455+0000 7f8da61e3640 1 -- start start 2024-05-09T05:06:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8da61e3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8da00d16b0 0x7f8da00d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8da61e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 0x7f8da015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8da61e3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8da006f6d0 con 0x7f8da006f1b0 2024-05-09T05:06:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8da61e3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8da00cee50 con 0x7f8da00d16b0 2024-05-09T05:06:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8da61e3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8da00cd0e0 con 0x7f8da015ca90 2024-05-09T05:06:33.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9f7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8da00d16b0 0x7f8da00d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8da006f1b0 0x7f8da00ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59904/0 (socket says 172.21.15.1:59904) 2024-05-09T05:06:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9effd640 1 -- 172.21.15.1:0/507335095 learned_addr learned my addr 172.21.15.1:0/507335095 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9ffff640 1 --2- 172.21.15.1:0/507335095 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 0x7f8da015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9f7fe640 1 -- 172.21.15.1:0/507335095 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 msgr2=0x7f8da015cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9f7fe640 1 --2- 172.21.15.1:0/507335095 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 0x7f8da015cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9f7fe640 1 -- 172.21.15.1:0/507335095 >> v1:172.21.15.1:6789/0 conn(0x7f8da006f1b0 legacy=0x7f8da00ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:33.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9f7fe640 1 -- 172.21.15.1:0/507335095 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da00cc4d0 con 0x7f8da00d16b0 2024-05-09T05:06:33.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.456+0000 7f8d9f7fe640 1 --2- 172.21.15.1:0/507335095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00d03d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8d900097a0 tx=0x7f8d9002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33ca44eef9bef282 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:33.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8d9e7fc640 1 -- 172.21.15.1:0/507335095 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90037070 con 0x7f8da00d16b0 2024-05-09T05:06:33.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8d9e7fc640 1 -- 172.21.15.1:0/507335095 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d90032440 con 0x7f8da00d16b0 2024-05-09T05:06:33.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8d9e7fc640 1 -- 172.21.15.1:0/507335095 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d90032810 con 0x7f8da00d16b0 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8da61e3640 1 -- 172.21.15.1:0/507335095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 msgr2=0x7f8da00d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8da61e3640 1 --2- 172.21.15.1:0/507335095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00d03d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f8d900097a0 tx=0x7f8d9002a210 comp rx=0 tx=0).stop 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8da61e3640 1 -- 172.21.15.1:0/507335095 shutdown_connections 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8da61e3640 1 --2- 172.21.15.1:0/507335095 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 0x7f8da015cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8da61e3640 1 --2- 172.21.15.1:0/507335095 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00d03d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8da61e3640 1 -- 172.21.15.1:0/507335095 >> 172.21.15.1:0/507335095 conn(0x7f8da00b3660 msgr2=0x7f8da00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8da61e3640 1 -- 172.21.15.1:0/507335095 shutdown_connections 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.457+0000 7f8da61e3640 1 -- 172.21.15.1:0/507335095 wait complete. 2024-05-09T05:06:33.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.459+0000 7f8da61e3640 1 Processor -- start 2024-05-09T05:06:33.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.459+0000 7f8da61e3640 1 -- start start 2024-05-09T05:06:33.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8da61e3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8da006f1b0 0x7f8da01f36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:33.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8da61e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00c76b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:33.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8da61e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 0x7f8da015a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8da61e3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8da00ccd70 con 0x7f8da00d16b0 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8da61e3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8da00cca90 con 0x7f8da006f1b0 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8da61e3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8da00d0910 con 0x7f8da015ca90 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00c76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00c76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51428/0 (socket says 172.21.15.1:51428) 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9f7fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8da006f1b0 0x7f8da01f36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 -- 172.21.15.1:0/874216369 learned_addr learned my addr 172.21.15.1:0/874216369 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9ffff640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 0x7f8da015a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 -- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 msgr2=0x7f8da015a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 0x7f8da015a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 -- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8da006f1b0 msgr2=0x7f8da01f36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8da006f1b0 0x7f8da01f36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 -- 172.21.15.1:0/874216369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8da01e99f0 con 0x7f8da00d16b0 2024-05-09T05:06:33.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9f7fe640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8da006f1b0 0x7f8da01f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:33.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d9effd640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00c76b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8d94001eb0 tx=0x7f8d9400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:33.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d87fff640 1 -- 172.21.15.1:0/874216369 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94017070 con 0x7f8da00d16b0 2024-05-09T05:06:33.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8d87fff640 1 -- 172.21.15.1:0/874216369 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d94005d00 con 0x7f8da00d16b0 2024-05-09T05:06:33.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.461+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8da00b73c0 con 0x7f8da00d16b0 2024-05-09T05:06:33.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.462+0000 7f8d87fff640 1 -- 172.21.15.1:0/874216369 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94007310 con 0x7f8da00d16b0 2024-05-09T05:06:33.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.462+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8da00b4940 con 0x7f8da00d16b0 2024-05-09T05:06:33.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.462+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8da00d13b0 con 0x7f8da00d16b0 2024-05-09T05:06:33.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.463+0000 7f8d87fff640 1 -- 172.21.15.1:0/874216369 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8d9401a070 con 0x7f8da00d16b0 2024-05-09T05:06:33.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.464+0000 7f8d87fff640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d8c076300 0x7f8d8c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:33.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.464+0000 7f8d87fff640 1 -- 172.21.15.1:0/874216369 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8d94096920 con 0x7f8da00d16b0 2024-05-09T05:06:33.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.464+0000 7f8d9f7fe640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d8c076300 0x7f8d8c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:33.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.465+0000 7f8d9f7fe640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d8c076300 0x7f8d8c078810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8d90034000 tx=0x7f8d9002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.470+0000 7f8d87fff640 1 -- 172.21.15.1:0/874216369 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8da00d13b0 con 0x7f8da00d16b0 2024-05-09T05:06:33.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.699+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8da0050870 con 0x7f8da00d16b0 2024-05-09T05:06:33.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.700+0000 7f8d87fff640 1 -- 172.21.15.1:0/874216369 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6915 (secure 0 0 0) 0x7f8d9405fb90 con 0x7f8da00d16b0 2024-05-09T05:06:33.702 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:33.702 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:33.214091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34160,"mds_1":34157,"mds_2":4311,"mds_3":4287,"mds_4":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4311":{"gid":4311,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[4273,34160,4287,4311,34157]},"id":1}]} 2024-05-09T05:06:33.702 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d8c076300 msgr2=0x7f8d8c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d8c076300 0x7f8d8c078810 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f8d90034000 tx=0x7f8d9002e000 comp rx=0 tx=0).stop 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 msgr2=0x7f8da00c76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00c76b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8d94001eb0 tx=0x7f8d9400c890 comp rx=0 tx=0).stop 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 shutdown_connections 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d8c076300 0x7f8d8c078810 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8da015ca90 0x7f8da015a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8da00d16b0 0x7f8da00c76b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 --2- 172.21.15.1:0/874216369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8da006f1b0 0x7f8da01f36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:33.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 >> 172.21.15.1:0/874216369 conn(0x7f8da00b3660 msgr2=0x7f8da0153440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:33.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 shutdown_connections 2024-05-09T05:06:33.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:33.702+0000 7f8da61e3640 1 -- 172.21.15.1:0/874216369 wait complete. 2024-05-09T05:06:34.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T05:06:04.442853+0000', 'modified': '2024-05-09T05:06:33.214091+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 34160, 'mds_1': 34157, 'mds_2': 4311, 'mds_3': 4287, 'mds_4': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'e', 'rank': 4, 'incarnation': 15, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.1:6837/3883243235', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6836', 'nonce': 3883243235}, {'type': 'v1', 'addr': '172.21.15.1:6837', 'nonce': 3883243235}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4287': {'gid': 4287, 'name': 'c', 'rank': 3, 'incarnation': 13, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.1:6834/1980556788', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.1:6832', 'nonce': 1980556788}, {'type': 'v1', 'addr': '172.21.15.1:6834', 'nonce': 1980556788}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_4311': {'gid': 4311, 'name': 'd', 'rank': 2, 'incarnation': 11, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.196:6839/2336267187', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6838', 'nonce': 2336267187}, {'type': 'v1', 'addr': '172.21.15.196:6839', 'nonce': 2336267187}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34157': {'gid': 34157, 'name': 'b', 'rank': 1, 'incarnation': 9, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.196:6835/1932237812', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6834', 'nonce': 1932237812}, {'type': 'v1', 'addr': '172.21.15.196:6835', 'nonce': 1932237812}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_34160': {'gid': 34160, 'name': 'f', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.196:6837/3384416671', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.196:6836', 'nonce': 3384416671}, {'type': 'v1', 'addr': '172.21.15.196:6837', 'nonce': 3384416671}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 34160, 'qdb_cluster': [4273, 34160, 4287, 4311, 34157]} 2024-05-09T05:06:34.320 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-05-09T05:06:34.320 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-05-09T05:06:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.444+0000 7fb536f72640 1 Processor -- start 2024-05-09T05:06:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.444+0000 7fb536f72640 1 -- start start 2024-05-09T05:06:34.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.445+0000 7fb536f72640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb5300ca970 0x7fb530153730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.445+0000 7fb536f72640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5300c8a50 con 0x7fb5300c67b0 2024-05-09T05:06:34.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.445+0000 7fb536f72640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb5300c8d10 con 0x7fb5300cae10 2024-05-09T05:06:34.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.445+0000 7fb536f72640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb5300c9d00 con 0x7fb5300ca970 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.445+0000 7fb53550f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb5300c67b0 0x7fb530163d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59908/0 (socket says 172.21.15.1:59908) 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.445+0000 7fb53550f640 1 -- 172.21.15.1:0/3756169454 learned_addr learned my addr 172.21.15.1:0/3756169454 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.445+0000 7fb534d0e640 1 --2- 172.21.15.1:0/3756169454 >> v2:172.21.15.196:3300/0 conn(0x7fb5300ca970 0x7fb530153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.445+0000 7fb52f7fe640 1 -- 172.21.15.1:0/3756169454 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4007372865 0 0) 0x7fb5300c8a50 con 0x7fb5300c67b0 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.446+0000 7fb52f7fe640 1 -- 172.21.15.1:0/3756169454 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb51c003520 con 0x7fb5300c67b0 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.446+0000 7fb52f7fe640 1 -- 172.21.15.1:0/3756169454 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2193028144 0 0) 0x7fb5300c8d10 con 0x7fb5300cae10 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.446+0000 7fb52f7fe640 1 -- 172.21.15.1:0/3756169454 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb5300c8a50 con 0x7fb5300cae10 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.446+0000 7fb534d0e640 1 -- 172.21.15.1:0/3756169454 >> v1:172.21.15.196:6789/0 conn(0x7fb5300cae10 legacy=0x7fb53015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.446+0000 7fb534d0e640 1 -- 172.21.15.1:0/3756169454 >> v1:172.21.15.1:6789/0 conn(0x7fb5300c67b0 legacy=0x7fb530163d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:34.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.446+0000 7fb534d0e640 1 -- 172.21.15.1:0/3756169454 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5300c5e10 con 0x7fb5300ca970 2024-05-09T05:06:34.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.446+0000 7fb534d0e640 1 --2- 172.21.15.1:0/3756169454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300ca970 0x7fb530153730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb528009d90 tx=0x7fb52802a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a908194781499fbe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:34.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb52f7fe640 1 -- 172.21.15.1:0/3756169454 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528007740 con 0x7fb5300ca970 2024-05-09T05:06:34.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb52f7fe640 1 -- 172.21.15.1:0/3756169454 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5280078e0 con 0x7fb5300ca970 2024-05-09T05:06:34.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb52f7fe640 1 -- 172.21.15.1:0/3756169454 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb528007cb0 con 0x7fb5300ca970 2024-05-09T05:06:34.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb536f72640 1 -- 172.21.15.1:0/3756169454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300ca970 msgr2=0x7fb530153730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:34.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb536f72640 1 --2- 172.21.15.1:0/3756169454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300ca970 0x7fb530153730 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fb528009d90 tx=0x7fb52802a6f0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb536f72640 1 -- 172.21.15.1:0/3756169454 shutdown_connections 2024-05-09T05:06:34.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb536f72640 1 --2- 172.21.15.1:0/3756169454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300ca970 0x7fb530153730 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb536f72640 1 -- 172.21.15.1:0/3756169454 >> 172.21.15.1:0/3756169454 conn(0x7fb5300ce910 msgr2=0x7fb5300ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:34.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.447+0000 7fb536f72640 1 -- 172.21.15.1:0/3756169454 shutdown_connections 2024-05-09T05:06:34.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.448+0000 7fb536f72640 1 -- 172.21.15.1:0/3756169454 wait complete. 2024-05-09T05:06:34.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.449+0000 7fb536f72640 1 Processor -- start 2024-05-09T05:06:34.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.449+0000 7fb536f72640 1 -- start start 2024-05-09T05:06:34.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb536f72640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb5300c67b0 0x7fb5301f26f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb536f72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5300ca970 0x7fb530140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb536f72640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300cae10 0x7fb53013a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb536f72640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb5300c63d0 con 0x7fb5300ca970 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb536f72640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb5300c9870 con 0x7fb5300cae10 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb536f72640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb5300c9500 con 0x7fb5300c67b0 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5300ca970 0x7fb530140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb534d0e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb5300c67b0 0x7fb5301f26f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5300ca970 0x7fb530140540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51444/0 (socket says 172.21.15.1:51444) 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 -- 172.21.15.1:0/996366835 learned_addr learned my addr 172.21.15.1:0/996366835 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb53550f640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300cae10 0x7fb53013a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 -- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb5300c67b0 msgr2=0x7fb5301f26f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb5300c67b0 0x7fb5301f26f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 -- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300cae10 msgr2=0x7fb53013a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300cae10 0x7fb53013a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 -- 172.21.15.1:0/996366835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5301f2c30 con 0x7fb5300ca970 2024-05-09T05:06:34.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52ffff640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5300ca970 0x7fb530140540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb51800c460 tx=0x7fb51800c920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:34.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.451+0000 7fb52d7fa640 1 -- 172.21.15.1:0/996366835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5180052d0 con 0x7fb5300ca970 2024-05-09T05:06:34.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.452+0000 7fb52d7fa640 1 -- 172.21.15.1:0/996366835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb518005490 con 0x7fb5300ca970 2024-05-09T05:06:34.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.452+0000 7fb536f72640 1 -- 172.21.15.1:0/996366835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb53013ddd0 con 0x7fb5300ca970 2024-05-09T05:06:34.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.452+0000 7fb52d7fa640 1 -- 172.21.15.1:0/996366835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518012870 con 0x7fb5300ca970 2024-05-09T05:06:34.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.452+0000 7fb536f72640 1 -- 172.21.15.1:0/996366835 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb53013e350 con 0x7fb5300ca970 2024-05-09T05:06:34.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.453+0000 7fb52d7fa640 1 -- 172.21.15.1:0/996366835 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb518012a10 con 0x7fb5300ca970 2024-05-09T05:06:34.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.454+0000 7fb52d7fa640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb510076300 0x7fb510078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.454+0000 7fb52d7fa640 1 -- 172.21.15.1:0/996366835 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb5180970d0 con 0x7fb5300ca970 2024-05-09T05:06:34.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.454+0000 7fb534d0e640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb510076300 0x7fb510078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.455+0000 7fb53de26740 1 Processor -- start 2024-05-09T05:06:34.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.455+0000 7fb534d0e640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb510076300 0x7fb510078810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb528034000 tx=0x7fb52802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:34.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.455+0000 7fb53de26740 1 -- start start 2024-05-09T05:06:34.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb53de26740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 0x55ae29e0fc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb53de26740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a003b0 0x55ae29df32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb53de26740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 0x55ae29df37e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb53de26740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ae29d314e0 con 0x55ae29a003b0 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb53de26740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ae29cccff0 con 0x55ae29a08590 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb53de26740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55ae29d2d4a0 con 0x55ae29a29270 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a003b0 0x55ae29df32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a003b0 0x55ae29df32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51446/0 (socket says 172.21.15.1:51446) 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb534d0e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 0x55ae29e0fc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb52ffff640 1 -- 172.21.15.1:0/935022963 learned_addr learned my addr 172.21.15.1:0/935022963 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb53550f640 1 --2- 172.21.15.1:0/935022963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 0x55ae29df37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb52ffff640 1 -- 172.21.15.1:0/935022963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 msgr2=0x55ae29e0fc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb52ffff640 1 --2- 172.21.15.1:0/935022963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 0x55ae29e0fc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb52ffff640 1 -- 172.21.15.1:0/935022963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 msgr2=0x55ae29df37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb52ffff640 1 --2- 172.21.15.1:0/935022963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 0x55ae29df37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.456+0000 7fb52ffff640 1 -- 172.21.15.1:0/935022963 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae29cb3720 con 0x55ae29a003b0 2024-05-09T05:06:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.457+0000 7fb52ffff640 1 --2- 172.21.15.1:0/935022963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a003b0 0x55ae29df32a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb51805c940 tx=0x7fb51805fac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e117d06ec08e8477 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.457+0000 7fb5267fc640 1 -- 172.21.15.1:0/935022963 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518064650 con 0x55ae29a003b0 2024-05-09T05:06:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.457+0000 7fb5267fc640 1 -- 172.21.15.1:0/935022963 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb518069100 con 0x55ae29a003b0 2024-05-09T05:06:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.457+0000 7fb5267fc640 1 -- 172.21.15.1:0/935022963 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518064380 con 0x55ae29a003b0 2024-05-09T05:06:34.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.457+0000 7fb53de26740 1 -- 172.21.15.1:0/935022963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a003b0 msgr2=0x55ae29df32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.457+0000 7fb53de26740 1 --2- 172.21.15.1:0/935022963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a003b0 0x55ae29df32a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb51805c940 tx=0x7fb51805fac0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.458+0000 7fb53de26740 1 -- 172.21.15.1:0/935022963 shutdown_connections 2024-05-09T05:06:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.458+0000 7fb53de26740 1 --2- 172.21.15.1:0/935022963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 0x55ae29df37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.458+0000 7fb53de26740 1 --2- 172.21.15.1:0/935022963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a003b0 0x55ae29df32a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.458+0000 7fb53de26740 1 --2- 172.21.15.1:0/935022963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 0x55ae29e0fc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.458+0000 7fb53de26740 1 -- 172.21.15.1:0/935022963 >> 172.21.15.1:0/935022963 conn(0x55ae29a1d3c0 msgr2=0x55ae29dcd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.458+0000 7fb53de26740 1 -- 172.21.15.1:0/935022963 shutdown_connections 2024-05-09T05:06:34.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.458+0000 7fb53de26740 1 -- 172.21.15.1:0/935022963 wait complete. 2024-05-09T05:06:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.458+0000 7fb53de26740 1 -- start start 2024-05-09T05:06:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.459+0000 7fb53de26740 1 Processor -- start 2024-05-09T05:06:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb53de26740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a003b0 0x55ae29e16c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb53de26740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29e09c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb53de26740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 0x55ae29defd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb53de26740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ae29dcd700 con 0x55ae29a08590 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb53de26740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ae29cb8760 con 0x55ae29a003b0 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb53de26740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55ae29e0a620 con 0x55ae29a29270 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb53de26740 10 client.0 _ll_register_callbacks cb 0x55ae29cdb4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29e09c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29e09c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51452/0 (socket says 172.21.15.1:51452) 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb52ffff640 1 -- 172.21.15.1:0/895332616 learned_addr learned my addr 172.21.15.1:0/895332616 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb534d0e640 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a003b0 0x55ae29e16c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb53550f640 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 0x55ae29defd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb52ffff640 1 -- 172.21.15.1:0/895332616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 msgr2=0x55ae29defd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb52ffff640 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 0x55ae29defd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb52ffff640 1 -- 172.21.15.1:0/895332616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a003b0 msgr2=0x55ae29e16c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb52ffff640 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a003b0 0x55ae29e16c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:34.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.460+0000 7fb52ffff640 1 -- 172.21.15.1:0/895332616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae29cf9bd0 con 0x55ae29a08590 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb52ffff640 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29e09c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb51805c940 tx=0x7fb518000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=261ee612d4aaa33f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb50f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb50f7fe640 1 -- 172.21.15.1:0/895332616 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518092030 con 0x55ae29a08590 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 1 -- 172.21.15.1:0/895332616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb518069100 con 0x55ae29a08590 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb50f7fe640 1 -- 172.21.15.1:0/895332616 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb518064390 con 0x55ae29a08590 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb50f7fe640 1 -- 172.21.15.1:0/895332616 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fb518095320 con 0x55ae29a08590 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 10 client.4668 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 1 -- 172.21.15.1:0/895332616 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55ae29cdf280 con 0x55ae29a08590 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb50f7fe640 1 -- 172.21.15.1:0/895332616 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb518069100 con 0x55ae29a08590 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb50f7fe640 1 -- 172.21.15.1:0/895332616 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7fb51805f5b0 con 0x55ae29a08590 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 10 client.4668 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 10 client.4668 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 20 client.4668 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:34.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] conn(0x55ae299e7e40 0x55ae29e22dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 4 client.4668 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:06:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb53de26740 1 -- 172.21.15.1:0/895332616 --> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae29d3f0f0 con 0x55ae299e7e40 2024-05-09T05:06:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.461+0000 7fb534d0e640 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] conn(0x55ae299e7e40 0x55ae29e22dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.462+0000 7fb50f7fe640 10 client.4668 ms_handle_connect on v2:172.21.15.1:6836/3883243235 2024-05-09T05:06:34.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:34.462+0000 7fb534d0e640 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] conn(0x55ae299e7e40 0x55ae29e22dd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=14fc54a0b03f30b8 server_cookie=c4cf7d8411382e86 in_seq=0 out_seq=0 2024-05-09T05:06:38.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.217+0000 7fb50f7fe640 1 -- 172.21.15.1:0/895332616 <== mds.4 v2:172.21.15.1:6836/3883243235 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55ae29d3f0f0 con 0x55ae299e7e40 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.217+0000 7fb50f7fe640 10 client.4668 handle_command_reply: tid=0 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.217+0000 7fb53de26740 1 client.4668 shutdown 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.217+0000 7fb53de26740 1 -- 172.21.15.1:0/895332616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 msgr2=0x55ae29e09c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.217+0000 7fb53de26740 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29e09c10 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fb51805c940 tx=0x7fb518000fe0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.217+0000 7fb53de26740 1 -- 172.21.15.1:0/895332616 shutdown_connections 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.217+0000 7fb53de26740 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] conn(0x55ae299e7e40 0x55ae29e22dd0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.217+0000 7fb53de26740 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a29270 0x55ae29defd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.218+0000 7fb53de26740 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29e09c10 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.218+0000 7fb53de26740 1 --2- 172.21.15.1:0/895332616 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a003b0 0x55ae29e16c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.218+0000 7fb50f7fe640 0 client.4668 ms_handle_reset on v2:172.21.15.1:6836/3883243235 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.218+0000 7fb53de26740 1 -- 172.21.15.1:0/895332616 >> 172.21.15.1:0/895332616 conn(0x55ae29a1d3c0 msgr2=0x55ae29dc7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.218+0000 7fb53de26740 1 -- 172.21.15.1:0/895332616 shutdown_connections 2024-05-09T05:06:38.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.218+0000 7fb53de26740 1 -- 172.21.15.1:0/895332616 wait complete. 2024-05-09T05:06:38.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.218+0000 7fb53de26740 20 client.4668 trim_cache size 0 max 16384 2024-05-09T05:06:38.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.239+0000 7fb53de26740 1 Processor -- start 2024-05-09T05:06:38.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.239+0000 7fb53de26740 1 -- start start 2024-05-09T05:06:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.240+0000 7fb53de26740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29dcfe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.240+0000 7fb53de26740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29dd03a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.240+0000 7fb53de26740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a29270 0x55ae29e0a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.240+0000 7fb53de26740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb53013b490 con 0x55ae29a08590 2024-05-09T05:06:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.240+0000 7fb53de26740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ae29cb8760 con 0x55ae29a29270 2024-05-09T05:06:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.240+0000 7fb53de26740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55ae29dcd700 con 0x55ae29a003b0 2024-05-09T05:06:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29dd03a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29dd03a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51454/0 (socket says 172.21.15.1:51454) 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 -- 172.21.15.1:0/2589709495 learned_addr learned my addr 172.21.15.1:0/2589709495 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb53550f640 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a29270 0x55ae29e0a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb534d0e640 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29dcfe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 -- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 msgr2=0x55ae29dcfe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29dcfe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 -- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a29270 msgr2=0x55ae29e0a7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a29270 0x55ae29e0a7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 -- 172.21.15.1:0/2589709495 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae29a94850 con 0x55ae29a08590 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb534d0e640 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29dcfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb52ffff640 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29dd03a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb5180647e0 tx=0x7fb51801d070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f116faee731371b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb525ffb640 1 -- 172.21.15.1:0/2589709495 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb518095320 con 0x55ae29a08590 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb525ffb640 1 -- 172.21.15.1:0/2589709495 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb518069100 con 0x55ae29a08590 2024-05-09T05:06:38.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.241+0000 7fb525ffb640 1 -- 172.21.15.1:0/2589709495 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb51805c5e0 con 0x55ae29a08590 2024-05-09T05:06:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.242+0000 7fb53de26740 1 -- 172.21.15.1:0/2589709495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 msgr2=0x55ae29dd03a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.242+0000 7fb53de26740 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29dd03a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fb5180647e0 tx=0x7fb51801d070 comp rx=0 tx=0).stop 2024-05-09T05:06:38.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.242+0000 7fb53de26740 1 -- 172.21.15.1:0/2589709495 shutdown_connections 2024-05-09T05:06:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.242+0000 7fb53de26740 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a29270 0x55ae29e0a7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.242+0000 7fb53de26740 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a08590 0x55ae29dd03a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.242+0000 7fb53de26740 1 --2- 172.21.15.1:0/2589709495 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29dcfe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.242+0000 7fb53de26740 1 -- 172.21.15.1:0/2589709495 >> 172.21.15.1:0/2589709495 conn(0x55ae29a1d3c0 msgr2=0x55ae29dccf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.243+0000 7fb53de26740 1 -- 172.21.15.1:0/2589709495 shutdown_connections 2024-05-09T05:06:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.243+0000 7fb53de26740 1 -- 172.21.15.1:0/2589709495 wait complete. 2024-05-09T05:06:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.243+0000 7fb53de26740 1 -- start start 2024-05-09T05:06:38.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.244+0000 7fb53de26740 1 Processor -- start 2024-05-09T05:06:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53de26740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29e0ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53de26740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 0x55ae29dc8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53de26740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a29270 0x55ae29de9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53de26740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55ae29e22c50 con 0x55ae29a29270 2024-05-09T05:06:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53de26740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55ae29e22600 con 0x55ae29a08590 2024-05-09T05:06:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53de26740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55ae29e153f0 con 0x55ae29a003b0 2024-05-09T05:06:38.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53de26740 10 client.0 _ll_register_callbacks cb 0x7fb51c003880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53550f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a29270 0x55ae29de9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53550f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a29270 0x55ae29de9960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51464/0 (socket says 172.21.15.1:51464) 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb53550f640 1 -- 172.21.15.1:0/57835939 learned_addr learned my addr 172.21.15.1:0/57835939 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb52ffff640 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 0x55ae29dc8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.245+0000 7fb534d0e640 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29e0ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb53550f640 1 -- 172.21.15.1:0/57835939 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 msgr2=0x55ae29e0ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb53550f640 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29e0ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb53550f640 1 -- 172.21.15.1:0/57835939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 msgr2=0x55ae29dc8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb53550f640 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 0x55ae29dc8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb53550f640 1 -- 172.21.15.1:0/57835939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae29b0b6b0 con 0x55ae29a29270 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb53550f640 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a29270 0x55ae29de9960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb530072e10 tx=0x7fb520001800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c2fae12abdd8010 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb50f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb50f7fe640 1 -- 172.21.15.1:0/57835939 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb520016630 con 0x55ae29a29270 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb50f7fe640 1 -- 172.21.15.1:0/57835939 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb52001c8a0 con 0x55ae29a29270 2024-05-09T05:06:38.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.246+0000 7fb53de26740 1 -- 172.21.15.1:0/57835939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb518069100 con 0x55ae29a29270 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb50f7fe640 1 -- 172.21.15.1:0/57835939 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fb520026050 con 0x55ae29a29270 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb53de26740 10 client.4674 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb53de26740 1 -- 172.21.15.1:0/57835939 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55ae29ced630 con 0x55ae29a29270 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb50f7fe640 1 -- 172.21.15.1:0/57835939 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb518069100 con 0x55ae29a29270 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb50f7fe640 1 -- 172.21.15.1:0/57835939 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7fb520029020 con 0x55ae29a29270 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb53de26740 10 client.4674 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb53de26740 10 client.4674 resolve_mds: resolved name 'e' to daemon mds.e 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb53de26740 20 client.4674 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb53de26740 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] conn(0x55ae29e8af10 0x55ae29e0e3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb53de26740 4 client.4674 mds_command: new command op to 4273 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb53de26740 1 -- 172.21.15.1:0/57835939 --> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] -- command(tid 0: {"prefix": "status"}) -- 0x55ae29d38f60 con 0x55ae29e8af10 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.247+0000 7fb534d0e640 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] conn(0x55ae29e8af10 0x55ae29e0e3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb50f7fe640 10 client.4674 ms_handle_connect on v2:172.21.15.1:6836/3883243235 2024-05-09T05:06:38.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb534d0e640 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] conn(0x55ae29e8af10 0x55ae29e0e3a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=ef94f75a7ba6978c server_cookie=adde7dad9ca2dc30 in_seq=0 out_seq=0 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb50f7fe640 1 -- 172.21.15.1:0/57835939 <== mds.4 v2:172.21.15.1:6836/3883243235 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x55ae29d38f60 con 0x55ae29e8af10 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb50f7fe640 10 client.4674 handle_command_reply: tid=0 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 client.4674 shutdown 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 -- 172.21.15.1:0/57835939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a29270 msgr2=0x55ae29de9960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a29270 0x55ae29de9960 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb530072e10 tx=0x7fb520001800 comp rx=0 tx=0).stop 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 -- 172.21.15.1:0/57835939 shutdown_connections 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.1:6836/3883243235,v1:172.21.15.1:6837/3883243235] conn(0x55ae29e8af10 0x55ae29e0e3a0 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55ae29a08590 0x55ae29dc8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55ae29a29270 0x55ae29de9960 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 --2- 172.21.15.1:0/57835939 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55ae29a003b0 0x55ae29e0ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb50f7fe640 0 client.4674 ms_handle_reset on v2:172.21.15.1:6836/3883243235 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.248+0000 7fb53de26740 1 -- 172.21.15.1:0/57835939 >> 172.21.15.1:0/57835939 conn(0x55ae29a1d3c0 msgr2=0x55ae29e17ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb53de26740 1 -- 172.21.15.1:0/57835939 shutdown_connections 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb53de26740 1 -- 172.21.15.1:0/57835939 wait complete. 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb53de26740 20 client.4674 trim_cache size 0 max 16384 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 4, 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4273, 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 10.097422698000001, 2024-05-09T05:06:38.253 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-05-09T05:06:38.254 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 21, 2024-05-09T05:06:38.254 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 21, 2024-05-09T05:06:38.254 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 39.231156968999997 2024-05-09T05:06:38.254 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:38.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 -- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb510076300 msgr2=0x7fb510078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb510076300 0x7fb510078810 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fb528034000 tx=0x7fb52802f000 comp rx=0 tx=0).stop 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 -- 172.21.15.1:0/996366835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5300ca970 msgr2=0x7fb530140540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5300ca970 0x7fb530140540 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb51800c460 tx=0x7fb51800c920 comp rx=0 tx=0).stop 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 -- 172.21.15.1:0/996366835 shutdown_connections 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb510076300 0x7fb510078810 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb5300cae10 0x7fb53013a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb5300ca970 0x7fb530140540 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 --2- 172.21.15.1:0/996366835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb5300c67b0 0x7fb5301f26f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.249+0000 7fb536f72640 1 -- 172.21.15.1:0/996366835 >> 172.21.15.1:0/996366835 conn(0x7fb5300ce910 msgr2=0x7fb53006c8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.250+0000 7fb536f72640 1 -- 172.21.15.1:0/996366835 shutdown_connections 2024-05-09T05:06:38.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.250+0000 7fb536f72640 1 -- 172.21.15.1:0/996366835 wait complete. 2024-05-09T05:06:38.264 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-05-09T05:06:38.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.385+0000 7f10c1b00640 1 Processor -- start 2024-05-09T05:06:38.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.385+0000 7f10c1b00640 1 -- start start 2024-05-09T05:06:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10c1b00640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f10bc0cf360 0x7f10bc0b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10c1b00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10c1b00640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10bc0ce5c0 con 0x7f10bc15c9a0 2024-05-09T05:06:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10c1b00640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f10bc15d4c0 con 0x7f10bc0cf360 2024-05-09T05:06:38.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10c1b00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f10bc15d7a0 con 0x7f10bc0b4570 2024-05-09T05:06:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10baffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10bb7fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f10bc0cf360 0x7f10bc0b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10bbfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f10bc15c9a0 0x7f10bc0cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:45708/0 (socket says 172.21.15.1:45708) 2024-05-09T05:06:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10bbfff640 1 -- 172.21.15.1:0/2727726354 learned_addr learned my addr 172.21.15.1:0/2727726354 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10baffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:43884/0 (socket says 172.21.15.1:43884) 2024-05-09T05:06:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10bb7fe640 1 -- 172.21.15.1:0/2727726354 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 msgr2=0x7f10bc156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/2727726354 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10bb7fe640 1 -- 172.21.15.1:0/2727726354 >> v1:172.21.15.196:6789/0 conn(0x7f10bc15c9a0 legacy=0x7f10bc0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10ba7fc640 1 -- 172.21.15.1:0/2727726354 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1229744474 0 0) 0x7f10bc0ce5c0 con 0x7f10bc15c9a0 2024-05-09T05:06:38.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.387+0000 7f10bb7fe640 1 -- 172.21.15.1:0/2727726354 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10bc0ce880 con 0x7f10bc0cf360 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.388+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/2727726354 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc0cf360 0x7f10bc0b4b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0097a0 tx=0x7f10ac02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5b63da8b03b3b185 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.388+0000 7f10ba7fc640 1 -- 172.21.15.1:0/2727726354 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac037070 con 0x7f10bc0cf360 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.388+0000 7f10ba7fc640 1 -- 172.21.15.1:0/2727726354 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ac032440 con 0x7f10bc0cf360 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.388+0000 7f10ba7fc640 1 -- 172.21.15.1:0/2727726354 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac032810 con 0x7f10bc0cf360 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.389+0000 7f10c1b00640 1 -- 172.21.15.1:0/2727726354 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc0cf360 msgr2=0x7f10bc0b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.389+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2727726354 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc0cf360 0x7f10bc0b4b50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f10ac0097a0 tx=0x7f10ac02a210 comp rx=0 tx=0).stop 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.389+0000 7f10c1b00640 1 -- 172.21.15.1:0/2727726354 shutdown_connections 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.389+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2727726354 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.389+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2727726354 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc0cf360 0x7f10bc0b4b50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.389+0000 7f10c1b00640 1 -- 172.21.15.1:0/2727726354 >> 172.21.15.1:0/2727726354 conn(0x7f10bc06de10 msgr2=0x7f10bc06e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.389+0000 7f10c1b00640 1 -- 172.21.15.1:0/2727726354 shutdown_connections 2024-05-09T05:06:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.389+0000 7f10c1b00640 1 -- 172.21.15.1:0/2727726354 wait complete. 2024-05-09T05:06:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.391+0000 7f10c1b00640 1 Processor -- start 2024-05-09T05:06:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.391+0000 7f10c1b00640 1 -- start start 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10c1b00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc1f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10c1b00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc15c9a0 0x7f10bc159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10c1b00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10bc157310 0x7f10bc1e6050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10c1b00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc0b5090 con 0x7f10bc157310 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10c1b00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc0cee40 con 0x7f10bc15c9a0 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10c1b00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f10bc157090 con 0x7f10bc0b4570 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10bc157310 0x7f10bc1e6050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10bc157310 0x7f10bc1e6050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51474/0 (socket says 172.21.15.1:51474) 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10bbfff640 1 -- 172.21.15.1:0/2807270729 learned_addr learned my addr 172.21.15.1:0/2807270729 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10baffd640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc15c9a0 0x7f10bc159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.392+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc1f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10bbfff640 1 -- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 msgr2=0x7f10bc1f2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10bbfff640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc1f2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10bbfff640 1 -- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc15c9a0 msgr2=0x7f10bc159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10bbfff640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc15c9a0 0x7f10bc159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10bbfff640 1 -- 172.21.15.1:0/2807270729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10bc0cafd0 con 0x7f10bc157310 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10bbfff640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10bc157310 0x7f10bc1e6050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f10b400c6b0 tx=0x7f10b400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10a3fff640 1 -- 172.21.15.1:0/2807270729 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b4018070 con 0x7f10bc157310 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10a3fff640 1 -- 172.21.15.1:0/2807270729 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b4004da0 con 0x7f10bc157310 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10a3fff640 1 -- 172.21.15.1:0/2807270729 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b400a880 con 0x7f10bc157310 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10c1b00640 1 -- 172.21.15.1:0/2807270729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10bc0cb550 con 0x7f10bc157310 2024-05-09T05:06:38.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.393+0000 7f10c1b00640 1 -- 172.21.15.1:0/2807270729 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10bc1f6300 con 0x7f10bc157310 2024-05-09T05:06:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.395+0000 7f10a3fff640 1 -- 172.21.15.1:0/2807270729 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f10b400aa20 con 0x7f10bc157310 2024-05-09T05:06:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.396+0000 7f10a3fff640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f10a80762d0 0x7f10a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.396+0000 7f10a3fff640 1 -- 172.21.15.1:0/2807270729 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f10b4096ba0 con 0x7f10bc157310 2024-05-09T05:06:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.396+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f10a80762d0 0x7f10a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.397+0000 7f10c89b8740 1 Processor -- start 2024-05-09T05:06:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.397+0000 7f10c89b8740 1 -- start start 2024-05-09T05:06:38.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.397+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f10a80762d0 0x7f10a80787e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f10ac034000 tx=0x7f10ac02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6eb270 0x557a6aad1c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aab52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6ca590 0x557a6aab57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10c89b8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x557a6a9f34e0 con 0x557a6a6c23b0 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10c89b8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557a6a98eff0 con 0x557a6a6ca590 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10c89b8740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x557a6a9ef4a0 con 0x557a6a6eb270 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10baffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aab52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10baffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aab52a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51478/0 (socket says 172.21.15.1:51478) 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10baffd640 1 -- 172.21.15.1:0/118595822 learned_addr learned my addr 172.21.15.1:0/118595822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10bb7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6eb270 0x557a6aad1c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10bbfff640 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6ca590 0x557a6aab57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10bbfff640 1 -- 172.21.15.1:0/118595822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6eb270 msgr2=0x557a6aad1c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10bbfff640 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6eb270 0x557a6aad1c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10bbfff640 1 -- 172.21.15.1:0/118595822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 msgr2=0x557a6aab52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10bbfff640 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aab52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10bbfff640 1 -- 172.21.15.1:0/118595822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a6a975720 con 0x557a6a6ca590 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.398+0000 7f10baffd640 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aab52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:38.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.399+0000 7f10bbfff640 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6ca590 0x557a6aab57e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f10b40950a0 tx=0x7f10b405f510 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76132311ec7c1a90 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.399+0000 7f10a17fa640 1 -- 172.21.15.1:0/118595822 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b40640a0 con 0x557a6a6ca590 2024-05-09T05:06:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.399+0000 7f10a17fa640 1 -- 172.21.15.1:0/118595822 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b4068b50 con 0x557a6a6ca590 2024-05-09T05:06:38.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.399+0000 7f10a17fa640 1 -- 172.21.15.1:0/118595822 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b405f000 con 0x557a6a6ca590 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 -- 172.21.15.1:0/118595822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6ca590 msgr2=0x557a6aab57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6ca590 0x557a6aab57e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f10b40950a0 tx=0x7f10b405f510 comp rx=0 tx=0).stop 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 -- 172.21.15.1:0/118595822 shutdown_connections 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6ca590 0x557a6aab57e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aab52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 --2- 172.21.15.1:0/118595822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6eb270 0x557a6aad1c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 -- 172.21.15.1:0/118595822 >> 172.21.15.1:0/118595822 conn(0x557a6a6df3c0 msgr2=0x557a6aa8f8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 -- 172.21.15.1:0/118595822 shutdown_connections 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.400+0000 7f10c89b8740 1 -- 172.21.15.1:0/118595822 wait complete. 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.401+0000 7f10c89b8740 1 -- start start 2024-05-09T05:06:38.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.401+0000 7f10c89b8740 1 Processor -- start 2024-05-09T05:06:38.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6c23b0 0x557a6aad91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 0x557a6aae2d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6eb270 0x557a6aad4fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10c89b8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x557a6aa8f700 con 0x557a6a6eb270 2024-05-09T05:06:38.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10c89b8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557a6a97a760 con 0x557a6a6c23b0 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10c89b8740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x557a6aae37a0 con 0x557a6a6ca590 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10c89b8740 10 client.0 _ll_register_callbacks cb 0x557a6a99d4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6eb270 0x557a6aad4fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6eb270 0x557a6aad4fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51480/0 (socket says 172.21.15.1:51480) 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 -- 172.21.15.1:0/1535064905 learned_addr learned my addr 172.21.15.1:0/1535064905 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6c23b0 0x557a6aad91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10baffd640 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 0x557a6aae2d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 -- 172.21.15.1:0/1535064905 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 msgr2=0x557a6aae2d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 0x557a6aae2d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 -- 172.21.15.1:0/1535064905 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6c23b0 msgr2=0x557a6aad91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6c23b0 0x557a6aad91e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 -- 172.21.15.1:0/1535064905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a6a9bbbd0 con 0x557a6a6eb270 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f10bbfff640 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6eb270 0x557a6aad4fe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f10b405fd30 tx=0x7f10b4004f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35285f66067d7f4f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.402+0000 7f109a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f109a7fc640 1 -- 172.21.15.1:0/1535064905 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b40640a0 con 0x557a6a6eb270 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f109a7fc640 1 -- 172.21.15.1:0/1535064905 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f10b405f200 con 0x557a6a6eb270 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 1 -- 172.21.15.1:0/1535064905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10b4068b50 con 0x557a6a6eb270 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f109a7fc640 1 -- 172.21.15.1:0/1535064905 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f10b405b710 con 0x557a6a6eb270 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 10 client.4683 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 1 -- 172.21.15.1:0/1535064905 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x557a6a9a1280 con 0x557a6a6eb270 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f109a7fc640 1 -- 172.21.15.1:0/1535064905 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10b4068b50 con 0x557a6a6eb270 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f109a7fc640 1 -- 172.21.15.1:0/1535064905 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7f10b405b8b0 con 0x557a6a6eb270 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 10 client.4683 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 10 client.4683 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 20 client.4683 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] conn(0x557a6a6a9e40 0x557a6aaef7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 4 client.4683 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:06:38.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10c89b8740 1 -- 172.21.15.1:0/1535064905 --> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557a6aa010f0 con 0x557a6a6a9e40 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.403+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] conn(0x557a6a6a9e40 0x557a6aaef7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.404+0000 7f109a7fc640 10 client.4683 ms_handle_connect on v2:172.21.15.1:6832/1980556788 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.404+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] conn(0x557a6a6a9e40 0x557a6aaef7a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=44c2678adb2f1aa9 server_cookie=fe5d9a086ce4b702 in_seq=0 out_seq=0 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.404+0000 7f109a7fc640 1 -- 172.21.15.1:0/1535064905 <== mds.3 v2:172.21.15.1:6832/1980556788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x557a6aa010f0 con 0x557a6a6a9e40 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.404+0000 7f109a7fc640 10 client.4683 handle_command_reply: tid=0 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.404+0000 7f10c89b8740 1 client.4683 shutdown 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 -- 172.21.15.1:0/1535064905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6eb270 msgr2=0x557a6aad4fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6eb270 0x557a6aad4fe0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f10b405fd30 tx=0x7f10b4004f90 comp rx=0 tx=0).stop 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 -- 172.21.15.1:0/1535064905 shutdown_connections 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] conn(0x557a6a6a9e40 0x557a6aaef7a0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6eb270 0x557a6aad4fe0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 0x557a6aae2d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1535064905 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6c23b0 0x557a6aad91e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f109a7fc640 0 client.4683 ms_handle_reset on v2:172.21.15.1:6832/1980556788 2024-05-09T05:06:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 -- 172.21.15.1:0/1535064905 >> 172.21.15.1:0/1535064905 conn(0x557a6a6df3c0 msgr2=0x557a6aa89c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 -- 172.21.15.1:0/1535064905 shutdown_connections 2024-05-09T05:06:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 1 -- 172.21.15.1:0/1535064905 wait complete. 2024-05-09T05:06:38.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.405+0000 7f10c89b8740 20 client.4683 trim_cache size 0 max 16384 2024-05-09T05:06:38.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.424+0000 7f10c89b8740 1 Processor -- start 2024-05-09T05:06:38.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.424+0000 7f10c89b8740 1 -- start start 2024-05-09T05:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aa8f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 0x557a6aa8b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 0x557a6aace330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10c89b8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f10bc0c9750 con 0x557a6a6c23b0 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10c89b8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557a6aa8f700 con 0x557a6a6eb270 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10c89b8740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x557a6aacb930 con 0x557a6a6ca590 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10bb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aa8f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10bb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aa8f1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51488/0 (socket says 172.21.15.1:51488) 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10bb7fe640 1 -- 172.21.15.1:0/1334881050 learned_addr learned my addr 172.21.15.1:0/1334881050 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10baffd640 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 0x557a6aa8b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.425+0000 7f10bbfff640 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 0x557a6aace330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10bb7fe640 1 -- 172.21.15.1:0/1334881050 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 msgr2=0x557a6aa8b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 0x557a6aa8b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10bb7fe640 1 -- 172.21.15.1:0/1334881050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 msgr2=0x557a6aace330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 0x557a6aace330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10bb7fe640 1 -- 172.21.15.1:0/1334881050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557a6a756850 con 0x557a6a6c23b0 2024-05-09T05:06:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aa8f1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f10ac02a800 tx=0x7f10ac02c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de67c0076144cd4a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10a0ff9640 1 -- 172.21.15.1:0/1334881050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac037040 con 0x557a6a6c23b0 2024-05-09T05:06:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10a0ff9640 1 -- 172.21.15.1:0/1334881050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ac050490 con 0x557a6a6c23b0 2024-05-09T05:06:38.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.426+0000 7f10a0ff9640 1 -- 172.21.15.1:0/1334881050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10ac050840 con 0x557a6a6c23b0 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 -- 172.21.15.1:0/1334881050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 msgr2=0x557a6aa8f1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aa8f1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f10ac02a800 tx=0x7f10ac02c000 comp rx=0 tx=0).stop 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 -- 172.21.15.1:0/1334881050 shutdown_connections 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 0x557a6aace330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6ca590 0x557a6aa8b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 --2- 172.21.15.1:0/1334881050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6c23b0 0x557a6aa8f1a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 -- 172.21.15.1:0/1334881050 >> 172.21.15.1:0/1334881050 conn(0x557a6a6df3c0 msgr2=0x557a6aad9840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 -- 172.21.15.1:0/1334881050 shutdown_connections 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.427+0000 7f10c89b8740 1 -- 172.21.15.1:0/1334881050 wait complete. 2024-05-09T05:06:38.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.428+0000 7f10c89b8740 1 -- start start 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.428+0000 7f10c89b8740 1 Processor -- start 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6c23b0 0x557a6aace7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6ca590 0x557a6aad9bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10c89b8740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 0x557a6aa90b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10c89b8740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x557a6aad05d0 con 0x557a6a6ca590 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10c89b8740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x557a6aae3600 con 0x557a6a6eb270 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10c89b8740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x557a6aad0c80 con 0x557a6a6c23b0 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10c89b8740 10 client.0 _ll_register_callbacks cb 0x7f10b0009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10baffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6ca590 0x557a6aad9bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10baffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6ca590 0x557a6aad9bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51498/0 (socket says 172.21.15.1:51498) 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10baffd640 1 -- 172.21.15.1:0/2408808655 learned_addr learned my addr 172.21.15.1:0/2408808655 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10bbfff640 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 0x557a6aa90b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6c23b0 0x557a6aace7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10baffd640 1 -- 172.21.15.1:0/2408808655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6c23b0 msgr2=0x557a6aace7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10baffd640 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6c23b0 0x557a6aace7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10baffd640 1 -- 172.21.15.1:0/2408808655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 msgr2=0x557a6aa90b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10baffd640 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 0x557a6aa90b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.429+0000 7f10baffd640 1 -- 172.21.15.1:0/2408808655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557a6a7cd6b0 con 0x557a6a6ca590 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10baffd640 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6ca590 0x557a6aad9bb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x557a6aa92640 tx=0x7f10b0015830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33e23b76c3d3cdeb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f109a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f109a7fc640 1 -- 172.21.15.1:0/2408808655 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10b0024060 con 0x557a6a6ca590 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f109a7fc640 1 -- 172.21.15.1:0/2408808655 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f10b0004240 con 0x557a6a6ca590 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 1 -- 172.21.15.1:0/2408808655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10ac050490 con 0x557a6a6ca590 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f109a7fc640 1 -- 172.21.15.1:0/2408808655 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f10b002a050 con 0x557a6a6ca590 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 10 client.4689 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 1 -- 172.21.15.1:0/2408808655 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x557a6a9af630 con 0x557a6a6ca590 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f109a7fc640 1 -- 172.21.15.1:0/2408808655 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f10ac050490 con 0x557a6a6ca590 2024-05-09T05:06:38.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f109a7fc640 1 -- 172.21.15.1:0/2408808655 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7f10b002d020 con 0x557a6a6ca590 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 10 client.4689 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 10 client.4689 resolve_mds: resolved name 'c' to daemon mds.c 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 20 client.4689 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] conn(0x557a6ab4b210 0x557a6aac8ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 4 client.4689 mds_command: new command op to 4287 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10c89b8740 1 -- 172.21.15.1:0/2408808655 --> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] -- command(tid 0: {"prefix": "status"}) -- 0x557a6a9faf60 con 0x557a6ab4b210 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.430+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] conn(0x557a6ab4b210 0x557a6aac8ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f109a7fc640 10 client.4689 ms_handle_connect on v2:172.21.15.1:6832/1980556788 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10bb7fe640 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] conn(0x557a6ab4b210 0x557a6aac8ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a99649c2ed32d683 server_cookie=d0dac36ed141b1cf in_seq=0 out_seq=0 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f109a7fc640 1 -- 172.21.15.1:0/2408808655 <== mds.3 v2:172.21.15.1:6832/1980556788 1 ==== command_reply(tid 0: 0 ) ==== 8+0+325 (crc 0 0 0) 0x557a6a9faf60 con 0x557a6ab4b210 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f109a7fc640 10 client.4689 handle_command_reply: tid=0 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 client.4689 shutdown 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 -- 172.21.15.1:0/2408808655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6ca590 msgr2=0x557a6aad9bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6ca590 0x557a6aad9bb0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x557a6aa92640 tx=0x7f10b0015830 comp rx=0 tx=0).stop 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 -- 172.21.15.1:0/2408808655 shutdown_connections 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.1:6832/1980556788,v1:172.21.15.1:6834/1980556788] conn(0x557a6ab4b210 0x557a6aac8ab0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x557a6a6eb270 0x557a6aa90b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x557a6a6ca590 0x557a6aad9bb0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f109a7fc640 0 client.4689 ms_handle_reset on v2:172.21.15.1:6832/1980556788 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 --2- 172.21.15.1:0/2408808655 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x557a6a6c23b0 0x557a6aace7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 -- 172.21.15.1:0/2408808655 >> 172.21.15.1:0/2408808655 conn(0x557a6a6df3c0 msgr2=0x557a6aae3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 -- 172.21.15.1:0/2408808655 shutdown_connections 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.431+0000 7f10c89b8740 1 -- 172.21.15.1:0/2408808655 wait complete. 2024-05-09T05:06:38.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c89b8740 20 client.4689 trim_cache size 0 max 16384 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 3, 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4287, 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 16.304424911000002, 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 21, 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 21, 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 39.422048644 2024-05-09T05:06:38.438 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:38.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 -- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f10a80762d0 msgr2=0x7f10a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f10a80762d0 0x7f10a80787e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f10ac034000 tx=0x7f10ac02e000 comp rx=0 tx=0).stop 2024-05-09T05:06:38.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 -- 172.21.15.1:0/2807270729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10bc157310 msgr2=0x7f10bc1e6050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10bc157310 0x7f10bc1e6050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f10b400c6b0 tx=0x7f10b400cb70 comp rx=0 tx=0).stop 2024-05-09T05:06:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 -- 172.21.15.1:0/2807270729 shutdown_connections 2024-05-09T05:06:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f10a80762d0 0x7f10a80787e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10bc15c9a0 0x7f10bc159b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10bc157310 0x7f10bc1e6050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 --2- 172.21.15.1:0/2807270729 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10bc0b4570 0x7f10bc1f2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 -- 172.21.15.1:0/2807270729 >> 172.21.15.1:0/2807270729 conn(0x7f10bc06de10 msgr2=0x7f10bc15b300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 -- 172.21.15.1:0/2807270729 shutdown_connections 2024-05-09T05:06:38.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.432+0000 7f10c1b00640 1 -- 172.21.15.1:0/2807270729 wait complete. 2024-05-09T05:06:38.444 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.d status 2024-05-09T05:06:38.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.559+0000 7ff7da261640 1 Processor -- start 2024-05-09T05:06:38.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.559+0000 7ff7da261640 1 -- start start 2024-05-09T05:06:38.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.560+0000 7ff7da261640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40cbde0 0x7ff7d4154500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.560+0000 7ff7da261640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7d40c49b0 0x7ff7d40cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.560+0000 7ff7da261640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff7d4154a40 con 0x7ff7d40d04f0 2024-05-09T05:06:38.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.560+0000 7ff7da261640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff7d40cfb20 con 0x7ff7d40c49b0 2024-05-09T05:06:38.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.560+0000 7ff7da261640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d415b170 con 0x7ff7d40cbde0 2024-05-09T05:06:38.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7cbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7d40c49b0 0x7ff7d40cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7cbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff7d40c49b0 0x7ff7d40cffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51506/0 (socket says 172.21.15.1:51506) 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7cbfff640 1 -- 172.21.15.1:0/1776511873 learned_addr learned my addr 172.21.15.1:0/1776511873 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/1776511873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40cbde0 0x7ff7d4154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7d2ffd640 1 -- 172.21.15.1:0/1776511873 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3529184940 0 0) 0x7ff7d4154a40 con 0x7ff7d40d04f0 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7cbfff640 1 -- 172.21.15.1:0/1776511873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40cbde0 msgr2=0x7ff7d4154500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/1776511873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40cbde0 0x7ff7d4154500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7cbfff640 1 -- 172.21.15.1:0/1776511873 >> v1:172.21.15.196:6789/0 conn(0x7ff7d40d04f0 legacy=0x7ff7d40cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7cbfff640 1 -- 172.21.15.1:0/1776511873 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d40cb120 con 0x7ff7d40c49b0 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.561+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/1776511873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40cbde0 0x7ff7d4154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.562+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/1776511873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40c49b0 0x7ff7d40cffb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009870 tx=0x7ff7bc02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f50297a0265a81b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.562+0000 7ff7d2ffd640 1 -- 172.21.15.1:0/1776511873 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc037070 con 0x7ff7d40c49b0 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.562+0000 7ff7d2ffd640 1 -- 172.21.15.1:0/1776511873 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7bc032440 con 0x7ff7d40c49b0 2024-05-09T05:06:38.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.562+0000 7ff7d2ffd640 1 -- 172.21.15.1:0/1776511873 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc032810 con 0x7ff7d40c49b0 2024-05-09T05:06:38.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.563+0000 7ff7da261640 1 -- 172.21.15.1:0/1776511873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40c49b0 msgr2=0x7ff7d40cffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.563+0000 7ff7da261640 1 --2- 172.21.15.1:0/1776511873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40c49b0 0x7ff7d40cffb0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc009870 tx=0x7ff7bc02a250 comp rx=0 tx=0).stop 2024-05-09T05:06:38.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.563+0000 7ff7da261640 1 -- 172.21.15.1:0/1776511873 shutdown_connections 2024-05-09T05:06:38.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.563+0000 7ff7da261640 1 --2- 172.21.15.1:0/1776511873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40c49b0 0x7ff7d40cffb0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.563+0000 7ff7da261640 1 --2- 172.21.15.1:0/1776511873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40cbde0 0x7ff7d4154500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.563+0000 7ff7da261640 1 -- 172.21.15.1:0/1776511873 >> 172.21.15.1:0/1776511873 conn(0x7ff7d4068830 msgr2=0x7ff7d4068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.563+0000 7ff7da261640 1 -- 172.21.15.1:0/1776511873 shutdown_connections 2024-05-09T05:06:38.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.563+0000 7ff7da261640 1 -- 172.21.15.1:0/1776511873 wait complete. 2024-05-09T05:06:38.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.564+0000 7ff7da261640 1 Processor -- start 2024-05-09T05:06:38.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.564+0000 7ff7da261640 1 -- start start 2024-05-09T05:06:38.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7da261640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7d40c49b0 0x7ff7d415d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7da261640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40cbde0 0x7ff7d40c8c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7da261640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40d04f0 0x7ff7d4151280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7da261640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d40cb8b0 con 0x7ff7d40cbde0 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7da261640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4154c80 con 0x7ff7d40c49b0 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7da261640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff7d415b430 con 0x7ff7d40d04f0 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40cbde0 0x7ff7d40c8c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40cbde0 0x7ff7d40c8c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51512/0 (socket says 172.21.15.1:51512) 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 -- 172.21.15.1:0/3967233634 learned_addr learned my addr 172.21.15.1:0/3967233634 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7d40c49b0 0x7ff7d415d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7d3fff640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40d04f0 0x7ff7d4151280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 -- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40d04f0 msgr2=0x7ff7d4151280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40d04f0 0x7ff7d4151280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 -- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7d40c49b0 msgr2=0x7ff7d415d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7d40c49b0 0x7ff7d415d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 -- 172.21.15.1:0/3967233634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7d415dec0 con 0x7ff7d40cbde0 2024-05-09T05:06:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40cbde0 0x7ff7d40c8c80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc000c00 tx=0x7ff7bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7d0ff9640 1 -- 172.21.15.1:0/3967233634 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc037040 con 0x7ff7d40cbde0 2024-05-09T05:06:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7d0ff9640 1 -- 172.21.15.1:0/3967233634 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7bc042070 con 0x7ff7d40cbde0 2024-05-09T05:06:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.566+0000 7ff7da261640 1 -- 172.21.15.1:0/3967233634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7d406d800 con 0x7ff7d40cbde0 2024-05-09T05:06:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.567+0000 7ff7d0ff9640 1 -- 172.21.15.1:0/3967233634 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc030880 con 0x7ff7d40cbde0 2024-05-09T05:06:38.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.567+0000 7ff7da261640 1 -- 172.21.15.1:0/3967233634 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7d41f2ef0 con 0x7ff7d40cbde0 2024-05-09T05:06:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.568+0000 7ff7d0ff9640 1 -- 172.21.15.1:0/3967233634 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff7bc030a20 con 0x7ff7d40cbde0 2024-05-09T05:06:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.569+0000 7ff7d0ff9640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7b40762d0 0x7ff7b40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.569+0000 7ff7d0ff9640 1 -- 172.21.15.1:0/3967233634 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff7bc0b7d30 con 0x7ff7d40cbde0 2024-05-09T05:06:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.569+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7b40762d0 0x7ff7b40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.569+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7b40762d0 0x7ff7b40787e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009560 tx=0x7ff7c4009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.570+0000 7ff7e11fa740 1 Processor -- start 2024-05-09T05:06:38.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.570+0000 7ff7e11fa740 1 -- start start 2024-05-09T05:06:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37f5c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33e63b0 0x55b5b37d92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33ee590 0x55b5b37d97e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b5b37174e0 con 0x55b5b340f270 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b5b36b2ff0 con 0x55b5b33e63b0 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55b5b37134a0 con 0x55b5b33ee590 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37f5c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37f5c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51522/0 (socket says 172.21.15.1:51522) 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d37fe640 1 -- 172.21.15.1:0/3990815211 learned_addr learned my addr 172.21.15.1:0/3990815211 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33e63b0 0x55b5b37d92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d3fff640 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33ee590 0x55b5b37d97e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d37fe640 1 -- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33ee590 msgr2=0x55b5b37d97e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33ee590 0x55b5b37d97e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d37fe640 1 -- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33e63b0 msgr2=0x55b5b37d92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33e63b0 0x55b5b37d92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7d37fe640 1 -- 172.21.15.1:0/3990815211 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5b3699720 con 0x55b5b340f270 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.571+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33e63b0 0x55b5b37d92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.572+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37f5c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4004490 tx=0x7ff7c4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fce5d05ab424e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.572+0000 7ff7c97fa640 1 -- 172.21.15.1:0/3990815211 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c401d050 con 0x55b5b340f270 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.572+0000 7ff7c97fa640 1 -- 172.21.15.1:0/3990815211 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c4027070 con 0x55b5b340f270 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.572+0000 7ff7c97fa640 1 -- 172.21.15.1:0/3990815211 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7c4003a20 con 0x55b5b340f270 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 -- 172.21.15.1:0/3990815211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 msgr2=0x55b5b37f5c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37f5c80 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4004490 tx=0x7ff7c4014000 comp rx=0 tx=0).stop 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 -- 172.21.15.1:0/3990815211 shutdown_connections 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33e63b0 0x55b5b37d92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33ee590 0x55b5b37d97e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/3990815211 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37f5c80 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 -- 172.21.15.1:0/3990815211 >> 172.21.15.1:0/3990815211 conn(0x55b5b34033c0 msgr2=0x55b5b37b38a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 -- 172.21.15.1:0/3990815211 shutdown_connections 2024-05-09T05:06:38.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.573+0000 7ff7e11fa740 1 -- 172.21.15.1:0/3990815211 wait complete. 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.574+0000 7ff7e11fa740 1 -- start start 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.574+0000 7ff7e11fa740 1 Processor -- start 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37f0310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37d4140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b3808770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b5b37b3700 con 0x55b5b33ee590 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b5b369e760 con 0x55b5b340f270 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55b5b37ed960 con 0x55b5b33e63b0 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7e11fa740 10 client.0 _ll_register_callbacks cb 0x55b5b36c14b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37d4140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37d4140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51528/0 (socket says 172.21.15.1:51528) 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7cbfff640 1 -- 172.21.15.1:0/2183638627 learned_addr learned my addr 172.21.15.1:0/2183638627 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7d3fff640 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b3808770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37f0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7cbfff640 1 -- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 msgr2=0x55b5b37f0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37f0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7cbfff640 1 -- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 msgr2=0x55b5b3808770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b3808770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7cbfff640 1 -- 172.21.15.1:0/2183638627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5b36dfbd0 con 0x55b5b33ee590 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.575+0000 7ff7d3fff640 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b3808770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37d4140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc07e2b0 tx=0x7ff7bc030f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b679f11ea5a516ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7b27fc640 1 -- 172.21.15.1:0/2183638627 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc089d60 con 0x55b5b33ee590 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7b27fc640 1 -- 172.21.15.1:0/2183638627 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff7bc0802a0 con 0x55b5b33ee590 2024-05-09T05:06:38.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 1 -- 172.21.15.1:0/2183638627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7c4027070 con 0x55b5b33ee590 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7b27fc640 1 -- 172.21.15.1:0/2183638627 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7ff7bc0804a0 con 0x55b5b33ee590 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 10 client.4701 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 1 -- 172.21.15.1:0/2183638627 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55b5b36c5280 con 0x55b5b33ee590 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7b27fc640 1 -- 172.21.15.1:0/2183638627 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7c4027070 con 0x55b5b33ee590 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7b27fc640 1 -- 172.21.15.1:0/2183638627 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7ff7bc03a070 con 0x55b5b33ee590 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 10 client.4701 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 10 client.4701 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 20 client.4701 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] conn(0x55b5b33cde40 0x55b5b3813e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 4 client.4701 mds_command: new command op to 4311 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.576+0000 7ff7e11fa740 1 -- 172.21.15.1:0/2183638627 --> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5b37250f0 con 0x55b5b33cde40 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.577+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] conn(0x55b5b33cde40 0x55b5b3813e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.577+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] conn(0x55b5b33cde40 0x55b5b3813e00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a0973860cd925f4f server_cookie=631c3d37197e84ce in_seq=0 out_seq=0 2024-05-09T05:06:38.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.577+0000 7ff7b27fc640 10 client.4701 ms_handle_connect on v2:172.21.15.196:6838/2336267187 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7b27fc640 1 -- 172.21.15.1:0/2183638627 <== mds.2 v2:172.21.15.196:6838/2336267187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55b5b37250f0 con 0x55b5b33cde40 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7b27fc640 10 client.4701 handle_command_reply: tid=0 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 client.4701 shutdown 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 -- 172.21.15.1:0/2183638627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 msgr2=0x55b5b37d4140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37d4140 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc07e2b0 tx=0x7ff7bc030f90 comp rx=0 tx=0).stop 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 -- 172.21.15.1:0/2183638627 shutdown_connections 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] conn(0x55b5b33cde40 0x55b5b3813e00 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b3808770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37d4140 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7b27fc640 0 client.4701 ms_handle_reset on v2:172.21.15.196:6838/2336267187 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/2183638627 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37f0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 -- 172.21.15.1:0/2183638627 >> 172.21.15.1:0/2183638627 conn(0x55b5b34033c0 msgr2=0x55b5b37ae590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 -- 172.21.15.1:0/2183638627 shutdown_connections 2024-05-09T05:06:38.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 1 -- 172.21.15.1:0/2183638627 wait complete. 2024-05-09T05:06:38.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.578+0000 7ff7e11fa740 20 client.4701 trim_cache size 0 max 16384 2024-05-09T05:06:38.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.597+0000 7ff7e11fa740 1 Processor -- start 2024-05-09T05:06:38.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.597+0000 7ff7e11fa740 1 -- start start 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.598+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37ecc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.598+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33ee590 0x55b5b37ed140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.598+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37b3610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.598+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff7d4159bf0 con 0x55b5b340f270 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.598+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b5b37d4680 con 0x55b5b33ee590 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.598+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55b5b37ed6a0 con 0x55b5b33e63b0 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37b3610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51530/0 (socket says 172.21.15.1:51530) 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 -- 172.21.15.1:0/1764766993 learned_addr learned my addr 172.21.15.1:0/1764766993 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37ecc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33ee590 0x55b5b37ed140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 -- 172.21.15.1:0/1764766993 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 msgr2=0x55b5b37ecc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37ecc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 -- 172.21.15.1:0/1764766993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33ee590 msgr2=0x55b5b37ed140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33ee590 0x55b5b37ed140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 -- 172.21.15.1:0/1764766993 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5b37d44b0 con 0x55b5b340f270 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7d3fff640 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37b3610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc003f90 tx=0x7ff7cc0015b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14c916426884cdb2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7c8ff9640 1 -- 172.21.15.1:0/1764766993 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0055b0 con 0x55b5b340f270 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7c8ff9640 1 -- 172.21.15.1:0/1764766993 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7cc01b020 con 0x55b5b340f270 2024-05-09T05:06:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.599+0000 7ff7c8ff9640 1 -- 172.21.15.1:0/1764766993 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7cc0057b0 con 0x55b5b340f270 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 -- 172.21.15.1:0/1764766993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 msgr2=0x55b5b37b3610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37b3610 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff7cc003f90 tx=0x7ff7cc0015b0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 -- 172.21.15.1:0/1764766993 shutdown_connections 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b33ee590 0x55b5b37ed140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b340f270 0x55b5b37b3610 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/1764766993 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37ecc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 -- 172.21.15.1:0/1764766993 >> 172.21.15.1:0/1764766993 conn(0x55b5b34033c0 msgr2=0x55b5b37af3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 -- 172.21.15.1:0/1764766993 shutdown_connections 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.600+0000 7ff7e11fa740 1 -- 172.21.15.1:0/1764766993 wait complete. 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.601+0000 7ff7e11fa740 1 -- start start 2024-05-09T05:06:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.601+0000 7ff7e11fa740 1 Processor -- start 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37dd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37ee790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7e11fa740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b386d380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55b5b3809300 con 0x55b5b33ee590 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55b5b3808cb0 con 0x55b5b340f270 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7e11fa740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55b5b3805430 con 0x55b5b33e63b0 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7e11fa740 10 client.0 _ll_register_callbacks cb 0x7ff7bc080e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37ee790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7cbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37ee790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51540/0 (socket says 172.21.15.1:51540) 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.602+0000 7ff7cbfff640 1 -- 172.21.15.1:0/167737762 learned_addr learned my addr 172.21.15.1:0/167737762 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7d3fff640 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b386d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37dd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7cbfff640 1 -- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 msgr2=0x55b5b37dd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37dd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7cbfff640 1 -- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 msgr2=0x55b5b386d380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b386d380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7cbfff640 1 -- 172.21.15.1:0/167737762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5b347a850 con 0x55b5b33ee590 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7cbfff640 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37ee790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc07d630 tx=0x7ff7bc0b6130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41e161215fdb40b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7b27fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7b27fc640 1 -- 172.21.15.1:0/167737762 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7bc09e070 con 0x55b5b33ee590 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7b27fc640 1 -- 172.21.15.1:0/167737762 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7ff7bc07fd10 con 0x55b5b33ee590 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 1 -- 172.21.15.1:0/167737762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff7cc01b020 con 0x55b5b33ee590 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7b27fc640 1 -- 172.21.15.1:0/167737762 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7ff7bc08e050 con 0x55b5b33ee590 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 10 client.4707 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 1 -- 172.21.15.1:0/167737762 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55b5b34f16b0 con 0x55b5b33ee590 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7b27fc640 1 -- 172.21.15.1:0/167737762 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff7cc01b020 con 0x55b5b33ee590 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7b27fc640 1 -- 172.21.15.1:0/167737762 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7ff7bc0a1020 con 0x55b5b33ee590 2024-05-09T05:06:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 10 client.4707 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 10 client.4707 resolve_mds: resolved name 'd' to daemon mds.d 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 20 client.4707 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] conn(0x55b5b37ae1f0 0x55b5b37ae600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 4 client.4707 mds_command: new command op to 4311 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.603+0000 7ff7e11fa740 1 -- 172.21.15.1:0/167737762 --> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] -- command(tid 0: {"prefix": "status"}) -- 0x55b5b371ef60 con 0x55b5b37ae1f0 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] conn(0x55b5b37ae1f0 0x55b5b37ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7d37fe640 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] conn(0x55b5b37ae1f0 0x55b5b37ae600 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=88f0f12ae1110f71 server_cookie=1b106dddbaf8b996 in_seq=0 out_seq=0 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7b27fc640 10 client.4707 ms_handle_connect on v2:172.21.15.196:6838/2336267187 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7b27fc640 1 -- 172.21.15.1:0/167737762 <== mds.2 v2:172.21.15.196:6838/2336267187 1 ==== command_reply(tid 0: 0 ) ==== 8+0+331 (crc 0 0 0) 0x7ff7c4018160 con 0x55b5b37ae1f0 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7b27fc640 10 client.4707 handle_command_reply: tid=0 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 client.4707 shutdown 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 -- 172.21.15.1:0/167737762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 msgr2=0x55b5b37ee790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37ee790 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc07d630 tx=0x7ff7bc0b6130 comp rx=0 tx=0).stop 2024-05-09T05:06:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 -- 172.21.15.1:0/167737762 shutdown_connections 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:6838/2336267187,v1:172.21.15.196:6839/2336267187] conn(0x55b5b37ae1f0 0x55b5b37ae600 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55b5b340f270 0x55b5b386d380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55b5b33ee590 0x55b5b37ee790 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 --2- 172.21.15.1:0/167737762 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55b5b33e63b0 0x55b5b37dd820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7b27fc640 0 client.4707 ms_handle_reset on v2:172.21.15.196:6838/2336267187 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.604+0000 7ff7e11fa740 1 -- 172.21.15.1:0/167737762 >> 172.21.15.1:0/167737762 conn(0x55b5b34033c0 msgr2=0x55b5b37b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7e11fa740 1 -- 172.21.15.1:0/167737762 shutdown_connections 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7e11fa740 1 -- 172.21.15.1:0/167737762 wait complete. 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7e11fa740 20 client.4707 trim_cache size 0 max 16384 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:38.610 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 2, 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4311, 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 22.497545949999999, 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 21, 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 21, 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 39.577454879000001 2024-05-09T05:06:38.611 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 -- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7b40762d0 msgr2=0x7ff7b40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7b40762d0 0x7ff7b40787e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7ff7c4009560 tx=0x7ff7c4009190 comp rx=0 tx=0).stop 2024-05-09T05:06:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 -- 172.21.15.1:0/3967233634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40cbde0 msgr2=0x7ff7d40c8c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40cbde0 0x7ff7d40c8c80 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff7bc000c00 tx=0x7ff7bc0046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 -- 172.21.15.1:0/3967233634 shutdown_connections 2024-05-09T05:06:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7b40762d0 0x7ff7b40787e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff7d40d04f0 0x7ff7d4151280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff7d40cbde0 0x7ff7d40c8c80 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 --2- 172.21.15.1:0/3967233634 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff7d40c49b0 0x7ff7d415d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 -- 172.21.15.1:0/3967233634 >> 172.21.15.1:0/3967233634 conn(0x7ff7d4068830 msgr2=0x7ff7d40ca4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 -- 172.21.15.1:0/3967233634 shutdown_connections 2024-05-09T05:06:38.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.605+0000 7ff7da261640 1 -- 172.21.15.1:0/3967233634 wait complete. 2024-05-09T05:06:38.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T05:06:38.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.732+0000 7fbb7615e640 1 Processor -- start 2024-05-09T05:06:38.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.732+0000 7fbb7615e640 1 -- start start 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.733+0000 7fbb7615e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70163ef0 0x7fbb70166310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.733+0000 7fbb7615e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb700cbf90 con 0x7fbb70154530 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.733+0000 7fbb7615e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb7015a780 con 0x7fbb70072400 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.733+0000 7fbb7615e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbb700ca3c0 con 0x7fbb70163ef0 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.733+0000 7fbb6effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbb70154530 0x7fbb700cae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59922/0 (socket says 172.21.15.1:59922) 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.733+0000 7fbb6effd640 1 -- 172.21.15.1:0/2076240362 learned_addr learned my addr 172.21.15.1:0/2076240362 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.733+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/2076240362 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70163ef0 0x7fbb70166310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.734+0000 7fbb6e7fc640 1 -- 172.21.15.1:0/2076240362 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 400288238 0 0) 0x7fbb700cbf90 con 0x7fbb70154530 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.734+0000 7fbb6e7fc640 1 -- 172.21.15.1:0/2076240362 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb58003520 con 0x7fbb70154530 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.734+0000 7fbb6e7fc640 1 -- 172.21.15.1:0/2076240362 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2070852039 0 0) 0x7fbb7015a780 con 0x7fbb70072400 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.734+0000 7fbb6e7fc640 1 -- 172.21.15.1:0/2076240362 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb700cbf90 con 0x7fbb70072400 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.734+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2076240362 >> v1:172.21.15.196:6789/0 conn(0x7fbb70072400 legacy=0x7fbb70070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.734+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2076240362 >> v1:172.21.15.1:6789/0 conn(0x7fbb70154530 legacy=0x7fbb700cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.734+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2076240362 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb700ca610 con 0x7fbb70163ef0 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.735+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/2076240362 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70163ef0 0x7fbb70166310 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009dc0 tx=0x7fbb6802a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5d064eef1e6e5e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.735+0000 7fbb6e7fc640 1 -- 172.21.15.1:0/2076240362 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68007740 con 0x7fbb70163ef0 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.735+0000 7fbb6e7fc640 1 -- 172.21.15.1:0/2076240362 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb680078e0 con 0x7fbb70163ef0 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.735+0000 7fbb6e7fc640 1 -- 172.21.15.1:0/2076240362 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68007c90 con 0x7fbb70163ef0 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.735+0000 7fbb7615e640 1 -- 172.21.15.1:0/2076240362 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70163ef0 msgr2=0x7fbb70166310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.735+0000 7fbb7615e640 1 --2- 172.21.15.1:0/2076240362 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70163ef0 0x7fbb70166310 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbb68009dc0 tx=0x7fbb6802a6d0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.735+0000 7fbb7615e640 1 -- 172.21.15.1:0/2076240362 shutdown_connections 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.735+0000 7fbb7615e640 1 --2- 172.21.15.1:0/2076240362 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70163ef0 0x7fbb70166310 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.736+0000 7fbb7615e640 1 -- 172.21.15.1:0/2076240362 >> 172.21.15.1:0/2076240362 conn(0x7fbb700d0470 msgr2=0x7fbb700cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.736+0000 7fbb7615e640 1 -- 172.21.15.1:0/2076240362 shutdown_connections 2024-05-09T05:06:38.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.736+0000 7fbb7615e640 1 -- 172.21.15.1:0/2076240362 wait complete. 2024-05-09T05:06:38.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.737+0000 7fbb7615e640 1 Processor -- start 2024-05-09T05:06:38.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.737+0000 7fbb7615e640 1 -- start start 2024-05-09T05:06:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb7615e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb70072400 0x7fbb7015cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb7615e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70154530 0x7fbb7014fe90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb7615e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70163ef0 0x7fbb70163080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb7615e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb700b2630 con 0x7fbb70163ef0 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb7615e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbb70154940 con 0x7fbb70072400 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb7615e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbb700b27b0 con 0x7fbb70154530 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70163ef0 0x7fbb70163080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6f7fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb70072400 0x7fbb7015cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70163ef0 0x7fbb70163080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51556/0 (socket says 172.21.15.1:51556) 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6effd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70154530 0x7fbb7014fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 -- 172.21.15.1:0/273351489 learned_addr learned my addr 172.21.15.1:0/273351489 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6f7fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb70072400 0x7fbb7015cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:60486/0 (socket says 172.21.15.1:60486) 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 -- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70154530 msgr2=0x7fbb7014fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70154530 0x7fbb7014fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 -- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb70072400 msgr2=0x7fbb7015cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb70072400 0x7fbb7015cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 -- 172.21.15.1:0/273351489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb7015d500 con 0x7fbb70163ef0 2024-05-09T05:06:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb70072400 0x7fbb7015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70163ef0 0x7fbb70163080 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbb680316b0 tx=0x7fbb68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb57fff640 1 -- 172.21.15.1:0/273351489 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68009450 con 0x7fbb70163ef0 2024-05-09T05:06:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb57fff640 1 -- 172.21.15.1:0/273351489 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb68031ca0 con 0x7fbb70163ef0 2024-05-09T05:06:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.739+0000 7fbb7615e640 1 -- 172.21.15.1:0/273351489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb7015e660 con 0x7fbb70163ef0 2024-05-09T05:06:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.740+0000 7fbb57fff640 1 -- 172.21.15.1:0/273351489 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6803b630 con 0x7fbb70163ef0 2024-05-09T05:06:38.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.740+0000 7fbb7615e640 1 -- 172.21.15.1:0/273351489 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb700c63c0 con 0x7fbb70163ef0 2024-05-09T05:06:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.741+0000 7fbb57fff640 1 -- 172.21.15.1:0/273351489 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbb6803b7d0 con 0x7fbb70163ef0 2024-05-09T05:06:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.742+0000 7fbb57fff640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb5c076300 0x7fbb5c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.742+0000 7fbb57fff640 1 -- 172.21.15.1:0/273351489 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb680b89d0 con 0x7fbb70163ef0 2024-05-09T05:06:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.742+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb5c076300 0x7fbb5c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.742+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb5c076300 0x7fbb5c078810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbb60001120 tx=0x7fbb60009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.743+0000 7fbb7d00e740 1 Processor -- start 2024-05-09T05:06:38.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.743+0000 7fbb7d00e740 1 -- start start 2024-05-09T05:06:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bc0c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 0x556f96ba42a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96ba47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x556f96ae24e0 con 0x556f967b9590 2024-05-09T05:06:38.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556f96a7dff0 con 0x556f967da270 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x556f96ade4a0 con 0x556f967b13b0 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96ba47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96ba47e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51560/0 (socket says 172.21.15.1:51560) 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2606661898 learned_addr learned my addr 172.21.15.1:0/2606661898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bc0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb6effd640 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 0x556f96ba42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 msgr2=0x556f96ba42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 0x556f96ba42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.744+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 msgr2=0x556f96bc0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.745+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bc0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.745+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2606661898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f96a64720 con 0x556f967b9590 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.745+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bc0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.745+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96ba47e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb680b6f20 tx=0x7fbb680816f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54ed74408f21f9db server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.745+0000 7fbb557fa640 1 -- 172.21.15.1:0/2606661898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68088070 con 0x556f967b9590 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.745+0000 7fbb557fa640 1 -- 172.21.15.1:0/2606661898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb68081ce0 con 0x556f967b9590 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.745+0000 7fbb557fa640 1 -- 172.21.15.1:0/2606661898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68085fe0 con 0x556f967b9590 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2606661898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 msgr2=0x556f96ba47e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96ba47e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fbb680b6f20 tx=0x7fbb680816f0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2606661898 shutdown_connections 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96ba47e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 0x556f96ba42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/2606661898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bc0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2606661898 >> 172.21.15.1:0/2606661898 conn(0x556f967ce3c0 msgr2=0x556f96b7e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2606661898 shutdown_connections 2024-05-09T05:06:38.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.746+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2606661898 wait complete. 2024-05-09T05:06:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.747+0000 7fbb7d00e740 1 -- start start 2024-05-09T05:06:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.747+0000 7fbb7d00e740 1 Processor -- start 2024-05-09T05:06:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 0x556f96bb8af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bbb1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bbb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x556f96b7e700 con 0x556f967b9590 2024-05-09T05:06:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556f96a69760 con 0x556f967da270 2024-05-09T05:06:38.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x556f96bb9240 con 0x556f967b13b0 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb7d00e740 10 client.0 _ll_register_callbacks cb 0x556f96a8c4b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bbb1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bbb1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51566/0 (socket says 172.21.15.1:51566) 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6effd640 1 -- 172.21.15.1:0/3727502509 learned_addr learned my addr 172.21.15.1:0/3727502509 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 0x556f96bb8af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bbb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6effd640 1 -- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 msgr2=0x556f96bb8af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6effd640 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 0x556f96bb8af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6effd640 1 -- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 msgr2=0x556f96bbb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6effd640 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bbb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.748+0000 7fbb6effd640 1 -- 172.21.15.1:0/3727502509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f96aaabd0 con 0x556f967b9590 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb6effd640 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bbb1b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbb64007620 tx=0x7fbb64001720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d904d2943339f02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb4e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3727502509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb64007710 con 0x556f967b9590 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3727502509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb64013540 con 0x556f967b9590 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3727502509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb68081ce0 con 0x556f967b9590 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3727502509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fbb64021050 con 0x556f967b9590 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 10 client.4719 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3727502509 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x556f96a90280 con 0x556f967b9590 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3727502509 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb68081ce0 con 0x556f967b9590 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3727502509 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7fbb64024020 con 0x556f967b9590 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 10 client.4719 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 10 client.4719 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 20 client.4719 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] conn(0x556f96798e40 0x556f96bcfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 4 client.4719 mds_command: new command op to 34157 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.749+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3727502509 --> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f96af00f0 con 0x556f96798e40 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.750+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] conn(0x556f96798e40 0x556f96bcfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.750+0000 7fbb4e7fc640 10 client.4719 ms_handle_connect on v2:172.21.15.196:6834/1932237812 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.750+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] conn(0x556f96798e40 0x556f96bcfe00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a794d4bb99e429c9 server_cookie=12504d00629dd815 in_seq=0 out_seq=0 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3727502509 <== mds.1 v2:172.21.15.196:6834/1932237812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x556f96af00f0 con 0x556f96798e40 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb4e7fc640 10 client.4719 handle_command_reply: tid=0 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 client.4719 shutdown 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3727502509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 msgr2=0x556f96bbb1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bbb1b0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbb64007620 tx=0x7fbb64001720 comp rx=0 tx=0).stop 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3727502509 shutdown_connections 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] conn(0x556f96798e40 0x556f96bcfe00 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967da270 0x556f96bbb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bbb1b0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb4e7fc640 0 client.4719 ms_handle_reset on v2:172.21.15.196:6834/1932237812 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3727502509 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b13b0 0x556f96bb8af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3727502509 >> 172.21.15.1:0/3727502509 conn(0x556f967ce3c0 msgr2=0x556f96b79590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3727502509 shutdown_connections 2024-05-09T05:06:38.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3727502509 wait complete. 2024-05-09T05:06:38.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.751+0000 7fbb7d00e740 20 client.4719 trim_cache size 0 max 16384 2024-05-09T05:06:38.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.770+0000 7fbb7d00e740 1 Processor -- start 2024-05-09T05:06:38.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.770+0000 7fbb7d00e740 1 -- start start 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.771+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f96798e40 0x556f96b7a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.771+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96b7aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.771+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bc83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.771+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb701f6730 con 0x556f967b9590 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.771+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556f96bb98c0 con 0x556f967b13b0 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.771+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x556f96ba1e10 con 0x556f96798e40 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bc83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bc83a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51580/0 (socket says 172.21.15.1:51580) 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2209596995 learned_addr learned my addr 172.21.15.1:0/2209596995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6effd640 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96b7aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f96798e40 0x556f96b7a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f96798e40 msgr2=0x556f96b7a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f96798e40 0x556f96b7a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 msgr2=0x556f96b7aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96b7aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 -- 172.21.15.1:0/2209596995 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f96845850 con 0x556f967b9590 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6effd640 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96b7aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f96798e40 0x556f96b7a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bc83a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb6807dd60 tx=0x7fbb68088000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=888b327f3315ed50 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb54ff9640 1 -- 172.21.15.1:0/2209596995 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb6808aa00 con 0x556f967b9590 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb54ff9640 1 -- 172.21.15.1:0/2209596995 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb680b3040 con 0x556f967b9590 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.772+0000 7fbb54ff9640 1 -- 172.21.15.1:0/2209596995 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb68085fe0 con 0x556f967b9590 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2209596995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 msgr2=0x556f96bc83a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bc83a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fbb6807dd60 tx=0x7fbb68088000 comp rx=0 tx=0).stop 2024-05-09T05:06:38.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2209596995 shutdown_connections 2024-05-09T05:06:38.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96b7aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f967b9590 0x556f96bc83a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/2209596995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f96798e40 0x556f96b7a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2209596995 >> 172.21.15.1:0/2209596995 conn(0x556f967ce3c0 msgr2=0x556f96ba8ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2209596995 shutdown_connections 2024-05-09T05:06:38.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.773+0000 7fbb7d00e740 1 -- 172.21.15.1:0/2209596995 wait complete. 2024-05-09T05:06:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.774+0000 7fbb7d00e740 1 -- start start 2024-05-09T05:06:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.774+0000 7fbb7d00e740 1 Processor -- start 2024-05-09T05:06:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f96798e40 0x556f96bd4690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96bd58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb7d00e740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b9590 0x556f96bd94c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x556f96bcfc80 con 0x556f96798e40 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x556f96bd3930 con 0x556f967b13b0 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb7d00e740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x556f96bd0350 con 0x556f967b9590 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb7d00e740 10 client.0 _ll_register_callbacks cb 0x7fbb64007280 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f96798e40 0x556f96bd4690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f96798e40 0x556f96bd4690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51582/0 (socket says 172.21.15.1:51582) 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb6f7fe640 1 -- 172.21.15.1:0/3757404278 learned_addr learned my addr 172.21.15.1:0/3757404278 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb6effd640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96bd58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.775+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b9590 0x556f96bd94c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb6f7fe640 1 -- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b9590 msgr2=0x556f96bd94c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b9590 0x556f96bd94c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb6f7fe640 1 -- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 msgr2=0x556f96bd58a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96bd58a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb6f7fe640 1 -- 172.21.15.1:0/3757404278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f968bc6b0 con 0x556f96798e40 2024-05-09T05:06:38.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb6effd640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96bd58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb6ffff640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b9590 0x556f96bd94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb6f7fe640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f96798e40 0x556f96bd4690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb600246c0 tx=0x7fbb60000fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc9fd087cdebb494 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb4e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3757404278 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb600294a0 con 0x556f96798e40 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3757404278 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fbb60029640 con 0x556f96798e40 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3757404278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbb680b3040 con 0x556f96798e40 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3757404278 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7fbb60037050 con 0x556f96798e40 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb7d00e740 10 client.4725 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3757404278 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x556f96a9e630 con 0x556f96798e40 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3757404278 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb680b3040 con 0x556f96798e40 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3757404278 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7fbb6003a020 con 0x556f96798e40 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb7d00e740 10 client.4725 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb7d00e740 10 client.4725 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T05:06:38.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.776+0000 7fbb7d00e740 20 client.4725 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] conn(0x556f9676e280 0x556f96c41960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 4 client.4725 mds_command: new command op to 34157 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3757404278 --> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] -- command(tid 0: {"prefix": "status"}) -- 0x556f96ae9f60 con 0x556f9676e280 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb6effd640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] conn(0x556f9676e280 0x556f96c41960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb6effd640 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] conn(0x556f9676e280 0x556f96c41960 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f782d457fe7e3975 server_cookie=b35b731da45a2431 in_seq=0 out_seq=0 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb4e7fc640 10 client.4725 ms_handle_connect on v2:172.21.15.196:6834/1932237812 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb4e7fc640 1 -- 172.21.15.1:0/3757404278 <== mds.1 v2:172.21.15.196:6834/1932237812 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x556f96ae9f60 con 0x556f9676e280 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb4e7fc640 10 client.4725 handle_command_reply: tid=0 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 client.4725 shutdown 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3757404278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f96798e40 msgr2=0x556f96bd4690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f96798e40 0x556f96bd4690 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fbb600246c0 tx=0x7fbb60000fe0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3757404278 shutdown_connections 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:6834/1932237812,v1:172.21.15.196:6835/1932237812] conn(0x556f9676e280 0x556f96c41960 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x556f967b13b0 0x556f96bd58a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x556f967b9590 0x556f96bd94c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb4e7fc640 0 client.4725 ms_handle_reset on v2:172.21.15.196:6834/1932237812 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 --2- 172.21.15.1:0/3757404278 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x556f96798e40 0x556f96bd4690 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.777+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3757404278 >> 172.21.15.1:0/3757404278 conn(0x556f967ce3c0 msgr2=0x556f96b7a590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3757404278 shutdown_connections 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7d00e740 1 -- 172.21.15.1:0/3757404278 wait complete. 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7d00e740 20 client.4725 trim_cache size 0 max 16384 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 1, 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 34157, 2024-05-09T05:06:38.784 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-05-09T05:06:38.785 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-05-09T05:06:38.785 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-05-09T05:06:38.785 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 28.694031839000001, 2024-05-09T05:06:38.785 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-05-09T05:06:38.785 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 21, 2024-05-09T05:06:38.785 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 21, 2024-05-09T05:06:38.785 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 39.752359749999997 2024-05-09T05:06:38.785 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 -- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb5c076300 msgr2=0x7fbb5c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb5c076300 0x7fbb5c078810 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbb60001120 tx=0x7fbb60009040 comp rx=0 tx=0).stop 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 -- 172.21.15.1:0/273351489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70163ef0 msgr2=0x7fbb70163080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70163ef0 0x7fbb70163080 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fbb680316b0 tx=0x7fbb68004a40 comp rx=0 tx=0).stop 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 -- 172.21.15.1:0/273351489 shutdown_connections 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb5c076300 0x7fbb5c078810 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb70163ef0 0x7fbb70163080 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb70154530 0x7fbb7014fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 --2- 172.21.15.1:0/273351489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb70072400 0x7fbb7015cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 -- 172.21.15.1:0/273351489 >> 172.21.15.1:0/273351489 conn(0x7fbb700d0470 msgr2=0x7fbb700b49a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 -- 172.21.15.1:0/273351489 shutdown_connections 2024-05-09T05:06:38.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.778+0000 7fbb7615e640 1 -- 172.21.15.1:0/273351489 wait complete. 2024-05-09T05:06:38.790 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-05-09T05:06:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.905+0000 7f3e10f8f640 1 Processor -- start 2024-05-09T05:06:38.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.905+0000 7f3e10f8f640 1 -- start start 2024-05-09T05:06:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.906+0000 7f3e10f8f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0b4570 0x7f3e0c156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.906+0000 7f3e10f8f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3e0c15c9a0 0x7f3e0c15cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.906+0000 7f3e10f8f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e0c0ce5c0 con 0x7f3e0c0cf360 2024-05-09T05:06:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.906+0000 7f3e10f8f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f3e0c15d4c0 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.906+0000 7f3e10f8f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0c15d7a0 con 0x7f3e0c0b4570 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e09d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0b4570 0x7f3e0c156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e0ad9d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3e0c15c9a0 0x7f3e0c15cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e0a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f3e0c0cf360 0x7f3e0c0b4b50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:45734/0 (socket says 172.21.15.1:45734) 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e0ad9d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3e0c15c9a0 0x7f3e0c15cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:60526/0 (socket says 172.21.15.1:60526) 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e0a59c640 1 -- 172.21.15.1:0/867820586 learned_addr learned my addr 172.21.15.1:0/867820586 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e09d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0b4570 0x7f3e0c156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:44046/0 (socket says 172.21.15.1:44046) 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e09d9b640 1 -- 172.21.15.1:0/867820586 >> v2:172.21.15.196:3300/0 conn(0x7f3e0c15c9a0 msgr2=0x7f3e0c15cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e0959a640 1 -- 172.21.15.1:0/867820586 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 792229805 0 0) 0x7f3e0c0ce5c0 con 0x7f3e0c0cf360 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/867820586 >> v2:172.21.15.196:3300/0 conn(0x7f3e0c15c9a0 0x7f3e0c15cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e09d9b640 1 -- 172.21.15.1:0/867820586 >> v1:172.21.15.196:6789/0 conn(0x7f3e0c0cf360 legacy=0x7f3e0c0b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e09d9b640 1 -- 172.21.15.1:0/867820586 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e0c0ce880 con 0x7f3e0c0b4570 2024-05-09T05:06:38.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.907+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/867820586 >> v2:172.21.15.196:3300/0 conn(0x7f3e0c15c9a0 0x7f3e0c15cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.908+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/867820586 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0b4570 0x7f3e0c156b50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e000097a0 tx=0x7f3e0002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=409727b50504622c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.908+0000 7f3e0959a640 1 -- 172.21.15.1:0/867820586 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00037070 con 0x7f3e0c0b4570 2024-05-09T05:06:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.908+0000 7f3e0959a640 1 -- 172.21.15.1:0/867820586 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e00032440 con 0x7f3e0c0b4570 2024-05-09T05:06:38.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.908+0000 7f3e0959a640 1 -- 172.21.15.1:0/867820586 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00032810 con 0x7f3e0c0b4570 2024-05-09T05:06:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.909+0000 7f3e10f8f640 1 -- 172.21.15.1:0/867820586 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0b4570 msgr2=0x7f3e0c156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.909+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/867820586 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0b4570 0x7f3e0c156b50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3e000097a0 tx=0x7f3e0002a210 comp rx=0 tx=0).stop 2024-05-09T05:06:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.909+0000 7f3e10f8f640 1 -- 172.21.15.1:0/867820586 shutdown_connections 2024-05-09T05:06:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.909+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/867820586 >> v2:172.21.15.196:3300/0 conn(0x7f3e0c15c9a0 0x7f3e0c15cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.909+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/867820586 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0b4570 0x7f3e0c156b50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.909+0000 7f3e10f8f640 1 -- 172.21.15.1:0/867820586 >> 172.21.15.1:0/867820586 conn(0x7f3e0c06de10 msgr2=0x7f3e0c06e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.909+0000 7f3e10f8f640 1 -- 172.21.15.1:0/867820586 shutdown_connections 2024-05-09T05:06:38.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.909+0000 7f3e10f8f640 1 -- 172.21.15.1:0/867820586 wait complete. 2024-05-09T05:06:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.911+0000 7f3e10f8f640 1 Processor -- start 2024-05-09T05:06:38.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.911+0000 7f3e10f8f640 1 -- start start 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e10f8f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e0c0b4570 0x7f3e0c159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e10f8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0c15c9a0 0x7f3e0c1f2570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e10f8f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0d06a0 0x7f3e0c153100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e10f8f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0c157090 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e10f8f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0c0cee40 con 0x7f3e0c0b4570 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e10f8f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3e0c0cd3f0 con 0x7f3e0c0d06a0 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e09d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0c15c9a0 0x7f3e0c1f2570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e09d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0c15c9a0 0x7f3e0c1f2570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51588/0 (socket says 172.21.15.1:51588) 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e09d9b640 1 -- 172.21.15.1:0/831849471 learned_addr learned my addr 172.21.15.1:0/831849471 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e0c0b4570 0x7f3e0c159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.912+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0d06a0 0x7f3e0c153100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3e09d9b640 1 -- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0d06a0 msgr2=0x7f3e0c153100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0d06a0 0x7f3e0c153100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3e09d9b640 1 -- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e0c0b4570 msgr2=0x7f3e0c159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e0c0b4570 0x7f3e0c159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3e09d9b640 1 -- 172.21.15.1:0/831849471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e0c1f2ab0 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0c15c9a0 0x7f3e0c1f2570 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3e000097a0 tx=0x7f3e000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3dfaffd640 1 -- 172.21.15.1:0/831849471 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00037050 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3dfaffd640 1 -- 172.21.15.1:0/831849471 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e00038040 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3e10f8f640 1 -- 172.21.15.1:0/831849471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e0c1f5c00 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3dfaffd640 1 -- 172.21.15.1:0/831849471 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00004490 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.913+0000 7f3e10f8f640 1 -- 172.21.15.1:0/831849471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e0c1f6180 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.915+0000 7f3dfaffd640 1 -- 172.21.15.1:0/831849471 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3e00003610 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.915+0000 7f3dfaffd640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3dec076300 0x7f3dec078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.916+0000 7f3dfaffd640 1 -- 172.21.15.1:0/831849471 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3e000b7d20 con 0x7f3e0c15c9a0 2024-05-09T05:06:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.916+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3dec076300 0x7f3dec078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.916+0000 7f3e17e45740 1 Processor -- start 2024-05-09T05:06:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.916+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3dec076300 0x7f3dec078810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc007fd0 tx=0x7f3dfc007de0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.916+0000 7f3e17e45740 1 -- start start 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.917+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cf9c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.917+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58ea3b0 0x55afd5cdd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.917+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cdd7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.917+0000 7f3e17e45740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55afd5c1b4e0 con 0x55afd58ea3b0 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.917+0000 7f3e17e45740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55afd5bb6ff0 con 0x55afd58f2590 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.917+0000 7f3e17e45740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55afd5c174a0 con 0x55afd5913270 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.917+0000 7f3e09d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58ea3b0 0x55afd5cdd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e0a59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cf9c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e09d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58ea3b0 0x55afd5cdd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51600/0 (socket says 172.21.15.1:51600) 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e09d9b640 1 -- 172.21.15.1:0/2801572099 learned_addr learned my addr 172.21.15.1:0/2801572099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cdd7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e09d9b640 1 -- 172.21.15.1:0/2801572099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 msgr2=0x55afd5cf9c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cf9c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e09d9b640 1 -- 172.21.15.1:0/2801572099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 msgr2=0x55afd5cdd7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cdd7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e09d9b640 1 -- 172.21.15.1:0/2801572099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd5b9d720 con 0x55afd58ea3b0 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cf9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58ea3b0 0x55afd5cdd2a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3e00035040 tx=0x7f3e0007d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22d380d64c1166e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3debfff640 1 -- 172.21.15.1:0/2801572099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00081690 con 0x55afd58ea3b0 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3debfff640 1 -- 172.21.15.1:0/2801572099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e00080f90 con 0x55afd58ea3b0 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.918+0000 7f3debfff640 1 -- 172.21.15.1:0/2801572099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e00089b10 con 0x55afd58ea3b0 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 -- 172.21.15.1:0/2801572099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58ea3b0 msgr2=0x55afd5cdd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58ea3b0 0x55afd5cdd2a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3e00035040 tx=0x7f3e0007d710 comp rx=0 tx=0).stop 2024-05-09T05:06:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 -- 172.21.15.1:0/2801572099 shutdown_connections 2024-05-09T05:06:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cdd7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58ea3b0 0x55afd5cdd2a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2801572099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cf9c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 -- 172.21.15.1:0/2801572099 >> 172.21.15.1:0/2801572099 conn(0x55afd59073c0 msgr2=0x55afd5cb78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 -- 172.21.15.1:0/2801572099 shutdown_connections 2024-05-09T05:06:38.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.919+0000 7f3e17e45740 1 -- 172.21.15.1:0/2801572099 wait complete. 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.920+0000 7f3e17e45740 1 -- start start 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.920+0000 7f3e17e45740 1 Processor -- start 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.921+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58ea3b0 0x55afd5cdefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.921+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58f2590 0x55afd5cfc950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.921+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cdcd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.921+0000 7f3e17e45740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55afd5cb7700 con 0x55afd58f2590 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.921+0000 7f3e17e45740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55afd5ba2760 con 0x55afd58ea3b0 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.921+0000 7f3e17e45740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55afd5c1b4e0 con 0x55afd5913270 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.921+0000 7f3e17e45740 10 client.0 _ll_register_callbacks cb 0x55afd5bc54b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58f2590 0x55afd5cfc950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58f2590 0x55afd5cfc950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51602/0 (socket says 172.21.15.1:51602) 2024-05-09T05:06:38.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 -- 172.21.15.1:0/442063774 learned_addr learned my addr 172.21.15.1:0/442063774 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58ea3b0 0x55afd5cdefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cdcd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 -- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 msgr2=0x55afd5cdcd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cdcd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 -- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58ea3b0 msgr2=0x55afd5cdefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58ea3b0 0x55afd5cdefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 -- 172.21.15.1:0/442063774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd5be3bd0 con 0x55afd58f2590 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58f2590 0x55afd5cfc950 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3e00035040 tx=0x7f3e00009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f58cffe621bac76 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3de97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3de97fa640 1 -- 172.21.15.1:0/442063774 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e000b3070 con 0x55afd58f2590 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3de97fa640 1 -- 172.21.15.1:0/442063774 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3e0007c900 con 0x55afd58f2590 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e17e45740 1 -- 172.21.15.1:0/442063774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e00080f90 con 0x55afd58f2590 2024-05-09T05:06:38.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3de97fa640 1 -- 172.21.15.1:0/442063774 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f3e00081690 con 0x55afd58f2590 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e17e45740 10 client.4734 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3e17e45740 1 -- 172.21.15.1:0/442063774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55afd5bc9280 con 0x55afd58f2590 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.922+0000 7f3de97fa640 1 -- 172.21.15.1:0/442063774 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e00080f90 con 0x55afd58f2590 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3de97fa640 1 -- 172.21.15.1:0/442063774 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7f3e0007cb00 con 0x55afd58f2590 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3e17e45740 10 client.4734 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3e17e45740 10 client.4734 resolve_mds: resolved name 'f' to daemon mds.f 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3e17e45740 20 client.4734 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3e17e45740 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] conn(0x55afd58d1e40 0x55afd5d0c360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3e17e45740 4 client.4734 mds_command: new command op to 34160 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3e17e45740 1 -- 172.21.15.1:0/442063774 --> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55afd5c290f0 con 0x55afd58d1e40 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] conn(0x55afd58d1e40 0x55afd5d0c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3de97fa640 10 client.4734 ms_handle_connect on v2:172.21.15.196:6836/3384416671 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.923+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] conn(0x55afd58d1e40 0x55afd5d0c360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3632ea9838c4947a server_cookie=c62ef14dcf0fc4ad in_seq=0 out_seq=0 2024-05-09T05:06:38.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3de97fa640 1 -- 172.21.15.1:0/442063774 <== mds.0 v2:172.21.15.196:6836/3384416671 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23648 (crc 0 0 0) 0x55afd5c290f0 con 0x55afd58d1e40 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3de97fa640 10 client.4734 handle_command_reply: tid=0 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 client.4734 shutdown 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 -- 172.21.15.1:0/442063774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58f2590 msgr2=0x55afd5cfc950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58f2590 0x55afd5cfc950 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f3e00035040 tx=0x7f3e00009190 comp rx=0 tx=0).stop 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 -- 172.21.15.1:0/442063774 shutdown_connections 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] conn(0x55afd58d1e40 0x55afd5d0c360 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd5913270 0x55afd5cdcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd58f2590 0x55afd5cfc950 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3de97fa640 0 client.4734 ms_handle_reset on v2:172.21.15.196:6836/3384416671 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 --2- 172.21.15.1:0/442063774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58ea3b0 0x55afd5cdefb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 -- 172.21.15.1:0/442063774 >> 172.21.15.1:0/442063774 conn(0x55afd59073c0 msgr2=0x55afd5cb33a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 -- 172.21.15.1:0/442063774 shutdown_connections 2024-05-09T05:06:38.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 1 -- 172.21.15.1:0/442063774 wait complete. 2024-05-09T05:06:38.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.924+0000 7f3e17e45740 20 client.4734 trim_cache size 0 max 16384 2024-05-09T05:06:38.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.944+0000 7f3e17e45740 1 Processor -- start 2024-05-09T05:06:38.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.944+0000 7f3e17e45740 1 -- start start 2024-05-09T05:06:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cf2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e17e45740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3e0c157690 con 0x55afd5913270 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e17e45740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55afd5ba2760 con 0x55afd58f2590 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e17e45740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55afd5cb7700 con 0x55afd58ea3b0 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51610/0 (socket says 172.21.15.1:51610) 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 -- 172.21.15.1:0/1778050174 learned_addr learned my addr 172.21.15.1:0/1778050174 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cf2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 -- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 msgr2=0x55afd5cf2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cf2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 -- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 msgr2=0x55afd5cb33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 -- 172.21.15.1:0/1778050174 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55afd597e850 con 0x55afd5913270 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cf2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0ce250 tx=0x7f3e04009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99a452d98011339 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3deb7fe640 1 -- 172.21.15.1:0/1778050174 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04003990 con 0x55afd5913270 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3deb7fe640 1 -- 172.21.15.1:0/1778050174 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e04003b30 con 0x55afd5913270 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.945+0000 7f3deb7fe640 1 -- 172.21.15.1:0/1778050174 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e040158f0 con 0x55afd5913270 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 -- 172.21.15.1:0/1778050174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 msgr2=0x55afd5cb38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c0ce250 tx=0x7f3e04009750 comp rx=0 tx=0).stop 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 -- 172.21.15.1:0/1778050174 shutdown_connections 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 --2- 172.21.15.1:0/1778050174 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cf2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 -- 172.21.15.1:0/1778050174 >> 172.21.15.1:0/1778050174 conn(0x55afd59073c0 msgr2=0x55afd5cdef90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 -- 172.21.15.1:0/1778050174 shutdown_connections 2024-05-09T05:06:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.946+0000 7f3e17e45740 1 -- 172.21.15.1:0/1778050174 wait complete. 2024-05-09T05:06:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.947+0000 7f3e17e45740 1 -- start start 2024-05-09T05:06:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.947+0000 7f3e17e45740 1 Processor -- start 2024-05-09T05:06:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.948+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cb27d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.948+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.948+0000 7f3e17e45740 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.948+0000 7f3e17e45740 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x55afd5cfd160 con 0x55afd5913270 2024-05-09T05:06:38.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.948+0000 7f3e17e45740 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x55afd5cf6a00 con 0x55afd58f2590 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.948+0000 7f3e17e45740 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x55afd5d0c8b0 con 0x55afd58ea3b0 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e17e45740 10 client.0 _ll_register_callbacks cb 0x7f3e04009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51626/0 (socket says 172.21.15.1:51626) 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 -- 172.21.15.1:0/2299037936 learned_addr learned my addr 172.21.15.1:0/2299037936 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e09d9b640 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cb27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 -- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 msgr2=0x55afd5cb27d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cb27d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 -- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 msgr2=0x55afd5cb33a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 -- 172.21.15.1:0/2299037936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55afd59f56b0 con 0x55afd5913270 2024-05-09T05:06:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e0ad9d640 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55afd5cba400 tx=0x7f3e04019040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c02cd617fe873ee7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3de97fa640 10 client.0 ms_handle_connect on v2:172.21.15.1:3300/0 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3de97fa640 1 -- 172.21.15.1:0/2299037936 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e04015d00 con 0x55afd5913270 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3de97fa640 1 -- 172.21.15.1:0/2299037936 <== mon.0 v2:172.21.15.1:3300/0 2 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3e04004030 con 0x55afd5913270 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.949+0000 7f3e17e45740 1 -- 172.21.15.1:0/2299037936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3e04003b30 con 0x55afd5913270 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3de97fa640 1 -- 172.21.15.1:0/2299037936 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) ==== 24+0+0 (secure 0 0 0) 0x7f3e04015ea0 con 0x55afd5913270 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e17e45740 10 client.4740 fetch_fsmap learned FSMap version 16 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e17e45740 1 -- 172.21.15.1:0/2299037936 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=2+}) -- 0x55afd5bd7630 con 0x55afd5913270 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3de97fa640 1 -- 172.21.15.1:0/2299037936 <== mon.0 v2:172.21.15.1:3300/0 4 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e04003b30 con 0x55afd5913270 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3de97fa640 1 -- 172.21.15.1:0/2299037936 <== mon.0 v2:172.21.15.1:3300/0 5 ==== fsmap(e 16) ==== 4784+0+0 (secure 0 0 0) 0x7f3e04004230 con 0x55afd5913270 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e17e45740 10 client.4740 fetch_fsmap finished waiting for FSMap version 16 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e17e45740 10 client.4740 resolve_mds: resolved name 'f' to daemon mds.f 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e17e45740 20 client.4740 populate_metadata read hostname 'smithi001' 2024-05-09T05:06:38.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] conn(0x55afd5cd9ae0 0x55afd5d829b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e17e45740 4 client.4740 mds_command: new command op to 34160 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e17e45740 1 -- 172.21.15.1:0/2299037936 --> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] -- command(tid 0: {"prefix": "status"}) -- 0x55afd5c22f60 con 0x55afd5cd9ae0 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] conn(0x55afd5cd9ae0 0x55afd5d829b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3de97fa640 10 client.4740 ms_handle_connect on v2:172.21.15.196:6836/3384416671 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.950+0000 7f3e0a59c640 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] conn(0x55afd5cd9ae0 0x55afd5d829b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e1dedab8545b1b02 server_cookie=cfa714a305dec59b in_seq=0 out_seq=0 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3de97fa640 1 -- 172.21.15.1:0/2299037936 <== mds.0 v2:172.21.15.196:6836/3384416671 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55afd5c22f60 con 0x55afd5cd9ae0 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3de97fa640 10 client.4740 handle_command_reply: tid=0 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 client.4740 shutdown 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 -- 172.21.15.1:0/2299037936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 msgr2=0x55afd5cb38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x55afd5cba400 tx=0x7f3e04019040 comp rx=0 tx=0).stop 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 -- 172.21.15.1:0/2299037936 shutdown_connections 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:6836/3384416671,v1:172.21.15.196:6837/3384416671] conn(0x55afd5cd9ae0 0x55afd5d829b0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x55afd58f2590 0x55afd5cb33a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x55afd5913270 0x55afd5cb38e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 --2- 172.21.15.1:0/2299037936 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x55afd58ea3b0 0x55afd5cb27d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3de97fa640 0 client.4740 ms_handle_reset on v2:172.21.15.196:6836/3384416671 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 -- 172.21.15.1:0/2299037936 >> 172.21.15.1:0/2299037936 conn(0x55afd59073c0 msgr2=0x55afd5cdeee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 -- 172.21.15.1:0/2299037936 shutdown_connections 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 1 -- 172.21.15.1:0/2299037936 wait complete. 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.951+0000 7f3e17e45740 20 client.4740 trim_cache size 0 max 16384 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stdout: "cluster_fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stdout: "whoami": 0, 2024-05-09T05:06:38.958 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 34160, 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout: "want_state": "up:active", 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "up:active", 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs", 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout: "rank_uptime": 33.909319455999999, 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mdsmap_epoch": 16, 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch": 21, 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap_epoch_barrier": 21, 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 39.924996352000001 2024-05-09T05:06:38.959 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 -- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3dec076300 msgr2=0x7f3dec078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3dec076300 0x7f3dec078810 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f3dfc007fd0 tx=0x7f3dfc007de0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 -- 172.21.15.1:0/831849471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0c15c9a0 msgr2=0x7f3e0c1f2570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0c15c9a0 0x7f3e0c1f2570 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3e000097a0 tx=0x7f3e000046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 -- 172.21.15.1:0/831849471 shutdown_connections 2024-05-09T05:06:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3dec076300 0x7f3dec078810 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e0c0d06a0 0x7f3e0c153100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e0c15c9a0 0x7f3e0c1f2570 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 --2- 172.21.15.1:0/831849471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e0c0b4570 0x7f3e0c159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 -- 172.21.15.1:0/831849471 >> 172.21.15.1:0/831849471 conn(0x7f3e0c06de10 msgr2=0x7f3e0c0d1590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 -- 172.21.15.1:0/831849471 shutdown_connections 2024-05-09T05:06:38.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:38.952+0000 7f3e10f8f640 1 -- 172.21.15.1:0/831849471 wait complete. 2024-05-09T05:06:38.963 INFO:teuthology.run_tasks:Running task sequential... 2024-05-09T05:06:38.973 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-09T05:06:38.974 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-09T05:06:38.974 INFO:teuthology.task.print:Enabling mgr modules 2024-05-09T05:06:38.974 INFO:teuthology.run_tasks:Running task ceph-fuse... 2024-05-09T05:06:39.012 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2024-05-09T05:06:39.012 INFO:tasks.ceph_fuse:config is {'client.0': {}} 2024-05-09T05:06:39.012 INFO:tasks.ceph_fuse:client.0 config is {} 2024-05-09T05:06:39.012 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T05:06:39.012 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:39.012 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:39.013 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-05-09T05:06:39.029 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:39.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-05-09T05:06:39.097 INFO:teuthology.orchestra.run.smithi001.stderr:Cannot find device "ceph-brx" 2024-05-09T05:06:39.098 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:06:39.098 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2024-05-09T05:06:39.099 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs ls 2024-05-09T05:06:39.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.253+0000 7f320668b640 1 Processor -- start 2024-05-09T05:06:39.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.253+0000 7f320668b640 1 -- start start 2024-05-09T05:06:39.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f320668b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f320006f440 0x7f3200156b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f320668b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32000d1530 0x7f320015ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f320668b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f320015ff70 con 0x7f32000ce4b0 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f320668b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3200157070 con 0x7f32000d1530 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f320668b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f32000d1940 con 0x7f320006f440 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f3204c28640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32000d1530 0x7f320015ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f3204c28640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32000d1530 0x7f320015ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51638/0 (socket says 172.21.15.1:51638) 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f3204c28640 1 -- 172.21.15.1:0/3313384456 learned_addr learned my addr 172.21.15.1:0/3313384456 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f31fffff640 1 --2- 172.21.15.1:0/3313384456 >> v2:172.21.15.196:3300/0 conn(0x7f320006f440 0x7f3200156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f3204c28640 1 -- 172.21.15.1:0/3313384456 >> v2:172.21.15.196:3300/0 conn(0x7f320006f440 msgr2=0x7f3200156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f3204c28640 1 --2- 172.21.15.1:0/3313384456 >> v2:172.21.15.196:3300/0 conn(0x7f320006f440 0x7f3200156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f3204c28640 1 -- 172.21.15.1:0/3313384456 >> v1:172.21.15.196:6789/0 conn(0x7f32000ce4b0 legacy=0x7f32000ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f3204c28640 1 -- 172.21.15.1:0/3313384456 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3200160250 con 0x7f32000d1530 2024-05-09T05:06:39.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.254+0000 7f31fffff640 1 --2- 172.21.15.1:0/3313384456 >> v2:172.21.15.196:3300/0 conn(0x7f320006f440 0x7f3200156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:39.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.255+0000 7f3204c28640 1 --2- 172.21.15.1:0/3313384456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000d1530 0x7f320015ca30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f32000b4390 tx=0x7f31f802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ae75f5b2cadc033 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:39.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.255+0000 7f31feffd640 1 -- 172.21.15.1:0/3313384456 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f8036070 con 0x7f32000d1530 2024-05-09T05:06:39.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.255+0000 7f31feffd640 1 -- 172.21.15.1:0/3313384456 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f8031600 con 0x7f32000d1530 2024-05-09T05:06:39.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.255+0000 7f31feffd640 1 -- 172.21.15.1:0/3313384456 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f80319d0 con 0x7f32000d1530 2024-05-09T05:06:39.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.256+0000 7f320668b640 1 -- 172.21.15.1:0/3313384456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000d1530 msgr2=0x7f320015ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:39.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.256+0000 7f320668b640 1 --2- 172.21.15.1:0/3313384456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000d1530 0x7f320015ca30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f32000b4390 tx=0x7f31f802a730 comp rx=0 tx=0).stop 2024-05-09T05:06:39.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.256+0000 7f320668b640 1 -- 172.21.15.1:0/3313384456 shutdown_connections 2024-05-09T05:06:39.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.256+0000 7f320668b640 1 --2- 172.21.15.1:0/3313384456 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000d1530 0x7f320015ca30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.256+0000 7f320668b640 1 --2- 172.21.15.1:0/3313384456 >> v2:172.21.15.196:3300/0 conn(0x7f320006f440 0x7f3200156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.256+0000 7f320668b640 1 -- 172.21.15.1:0/3313384456 >> 172.21.15.1:0/3313384456 conn(0x7f32000b3670 msgr2=0x7f32000b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:39.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.256+0000 7f320668b640 1 -- 172.21.15.1:0/3313384456 shutdown_connections 2024-05-09T05:06:39.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.256+0000 7f320668b640 1 -- 172.21.15.1:0/3313384456 wait complete. 2024-05-09T05:06:39.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.258+0000 7f320668b640 1 Processor -- start 2024-05-09T05:06:39.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.258+0000 7f320668b640 1 -- start start 2024-05-09T05:06:39.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f320668b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f320006f440 0x7f3200159e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:39.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f320668b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000ce4b0 0x7f32000cc370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:39.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f320668b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32000d1530 0x7f3200153d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:39.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f320668b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3200160810 con 0x7f32000ce4b0 2024-05-09T05:06:39.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f320668b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f32000cd180 con 0x7f320006f440 2024-05-09T05:06:39.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f320668b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f32000cf940 con 0x7f32000d1530 2024-05-09T05:06:39.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f31ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000ce4b0 0x7f32000cc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f31ff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000ce4b0 0x7f32000cc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51652/0 (socket says 172.21.15.1:51652) 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f31ff7fe640 1 -- 172.21.15.1:0/1832706331 learned_addr learned my addr 172.21.15.1:0/1832706331 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f31fffff640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f320006f440 0x7f3200159e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.259+0000 7f3204c28640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32000d1530 0x7f3200153d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31ff7fe640 1 -- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32000d1530 msgr2=0x7f3200153d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31ff7fe640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32000d1530 0x7f3200153d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31ff7fe640 1 -- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f320006f440 msgr2=0x7f3200159e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31ff7fe640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f320006f440 0x7f3200159e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31ff7fe640 1 -- 172.21.15.1:0/1832706331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32001f6470 con 0x7f32000ce4b0 2024-05-09T05:06:39.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31ff7fe640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000ce4b0 0x7f32000cc370 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f31f400c3f0 tx=0x7f31f400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:39.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31fcff9640 1 -- 172.21.15.1:0/1832706331 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f4017070 con 0x7f32000ce4b0 2024-05-09T05:06:39.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31fcff9640 1 -- 172.21.15.1:0/1832706331 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f31f40052d0 con 0x7f32000ce4b0 2024-05-09T05:06:39.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3200157570 con 0x7f32000ce4b0 2024-05-09T05:06:39.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f31fcff9640 1 -- 172.21.15.1:0/1832706331 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31f4003640 con 0x7f32000ce4b0 2024-05-09T05:06:39.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.260+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32001f2fb0 con 0x7f32000ce4b0 2024-05-09T05:06:39.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.261+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32000cf3f0 con 0x7f32000ce4b0 2024-05-09T05:06:39.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.262+0000 7f31fcff9640 1 -- 172.21.15.1:0/1832706331 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f31f401a070 con 0x7f32000ce4b0 2024-05-09T05:06:39.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.263+0000 7f31fcff9640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f31ec076300 0x7f31ec078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:39.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.263+0000 7f31fcff9640 1 -- 172.21.15.1:0/1832706331 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f31f4096850 con 0x7f32000ce4b0 2024-05-09T05:06:39.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.263+0000 7f31fffff640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f31ec076300 0x7f31ec078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:39.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.263+0000 7f31fffff640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f31ec076300 0x7f31ec078810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f31f00045f0 tx=0x7f31f0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:39.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.268+0000 7f31fcff9640 1 -- 172.21.15.1:0/1832706331 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f32000cf3f0 con 0x7f32000ce4b0 2024-05-09T05:06:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.503+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f3200050870 con 0x7f32000ce4b0 2024-05-09T05:06:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.504+0000 7f31fcff9640 1 -- 172.21.15.1:0/1832706331 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) ==== 53+0+73 (secure 0 0 0) 0x7f31f4068190 con 0x7f32000ce4b0 2024-05-09T05:06:39.505 INFO:teuthology.orchestra.run.smithi001.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-05-09T05:06:39.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f31ec076300 msgr2=0x7f31ec078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:39.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f31ec076300 0x7f31ec078810 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f31f00045f0 tx=0x7f31f0008040 comp rx=0 tx=0).stop 2024-05-09T05:06:39.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000ce4b0 msgr2=0x7f32000cc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:39.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000ce4b0 0x7f32000cc370 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f31f400c3f0 tx=0x7f31f400c8b0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 shutdown_connections 2024-05-09T05:06:39.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f31ec076300 0x7f31ec078810 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32000d1530 0x7f3200153d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32000ce4b0 0x7f32000cc370 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 --2- 172.21.15.1:0/1832706331 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f320006f440 0x7f3200159e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:39.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 >> 172.21.15.1:0/1832706331 conn(0x7f32000b3670 msgr2=0x7f320006e660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:39.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 shutdown_connections 2024-05-09T05:06:39.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39.506+0000 7f320668b640 1 -- 172.21.15.1:0/1832706331 wait complete. 2024-05-09T05:06:39.518 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T05:06:39.518 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T05:06:39.518 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi001.front.sepia.ceph.com 2024-05-09T05:06:39.518 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T05:06:39.518 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:39.519 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T05:06:39.519 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T05:06:39.519 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T05:06:39.519 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:06:39.519 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:39.519 DEBUG:teuthology.orchestra.run.smithi001:> ip addr 2024-05-09T05:06:39.535 INFO:teuthology.orchestra.run.smithi001.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T05:06:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T05:06:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T05:06:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:06:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 ::1/128 scope host 2024-05-09T05:06:39.535 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7a brd ff:ff:ff:ff:ff:ff 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f0 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:6b:f6:7b brd ff:ff:ff:ff:ff:ff 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp7s0f1 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e8 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f0 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: inet 172.21.15.1/20 brd 172.21.15.255 scope global dynamic noprefixroute ens1f0 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft 42706sec preferred_lft 42706sec 2024-05-09T05:06:39.536 INFO:teuthology.orchestra.run.smithi001.stdout: inet6 fe80::ec4:7aff:febd:15e8/64 scope link noprefixroute 2024-05-09T05:06:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: valid_lft forever preferred_lft forever 2024-05-09T05:06:39.537 INFO:teuthology.orchestra.run.smithi001.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T05:06:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: link/ether 0c:c4:7a:bd:15:e9 brd ff:ff:ff:ff:ff:ff 2024-05-09T05:06:39.537 INFO:teuthology.orchestra.run.smithi001.stdout: altname enp1s0f1 2024-05-09T05:06:39.537 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T05:06:39.537 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:06:39.537 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-05-09T05:06:39.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add name ceph-brx type bridge 2024-05-09T05:06:39.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr flush dev ceph-brx 2024-05-09T05:06:39.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set ceph-brx up 2024-05-09T05:06:39.538 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T05:06:39.538 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-05-09T05:06:39.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:06:39.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:06:39.730 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:39.730 DEBUG:teuthology.orchestra.run.smithi001:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T05:06:39.755 INFO:teuthology.orchestra.run.smithi001.stdout:1 2024-05-09T05:06:39.757 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:39.757 DEBUG:teuthology.orchestra.run.smithi001:> route 2024-05-09T05:06:39.831 INFO:teuthology.orchestra.run.smithi001.stdout:Kernel IP routing table 2024-05-09T05:06:39.831 INFO:teuthology.orchestra.run.smithi001.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T05:06:39.831 INFO:teuthology.orchestra.run.smithi001.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-05-09T05:06:39.831 INFO:teuthology.orchestra.run.smithi001.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-05-09T05:06:39.831 INFO:teuthology.orchestra.run.smithi001.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T05:06:39.831 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:06:39.832 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-05-09T05:06:39.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-05-09T05:06:39.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-05-09T05:06:39.832 DEBUG:teuthology.orchestra.run.smithi001:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-05-09T05:06:39.832 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-05-09T05:06:39.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:39 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:06:40.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:06:40.137 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:40.137 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-05-09T05:06:40.154 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:40.154 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list-id 2024-05-09T05:06:40.210 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:06:40.210 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-05-09T05:06:40.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:06:40.210 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-09T05:06:40.210 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-05-09T05:06:40.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:06:40.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:06:40.335 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-09T05:06:40.335 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:06:40.335 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-05-09T05:06:40.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-09T05:06:40.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T05:06:40.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-09T05:06:40.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-09T05:06:40.335 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-09T05:06:40.335 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-05-09T05:06:40.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:06:40.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:06:40.465 DEBUG:teuthology.orchestra.run.smithi001:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T05:06:40.465 DEBUG:teuthology.orchestra.run.smithi001:> set -e 2024-05-09T05:06:40.465 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set brx.0 up 2024-05-09T05:06:40.465 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T05:06:40.465 DEBUG:teuthology.orchestra.run.smithi001:> ') 2024-05-09T05:06:40.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:40 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T05:06:40.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:40 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:06:40.546 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2024-05-09T05:06:40.546 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:06:40.546 DEBUG:teuthology.orchestra.run.smithi001:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:40.561 INFO:teuthology.orchestra.run.smithi001.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T05:06:40.562 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T05:06:40.562 DEBUG:teuthology.orchestra.run.smithi001:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:40.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-05-09T05:06:40.685 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T05:06:40.742 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-05-09T05:06:40.742 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-05-09T05:06:40.742 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-05-09T05:06:40.742 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-05-09T05:06:40.742 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-05-09T05:06:40.742 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-05-09T05:06:40.742 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T05:06:40.743 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:06:40.744 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:06:40.745 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:40.745 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-05-09T05:06:40.798 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T05:06:40.798 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.0 --id 0) 2024-05-09T05:06:40.840 DEBUG:teuthology.orchestra.run.smithi001:> sudo modprobe fuse 2024-05-09T05:06:40.866 DEBUG:teuthology.orchestra.run.smithi001:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T05:06:40.923 INFO:teuthology.orchestra.run.smithi001.stdout:/proc 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/sys 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/dev 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/security 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/shm 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/pts 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/run 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/cgroup 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/pstore 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/bpf 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/ 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/selinux 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T05:06:40.924 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/mqueue 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/dev/hugepages 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/debug 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/tracing 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/fs/fuse/connections 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/sys/kernel/config 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/run/user/1000 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-0 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-1 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-2 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/var/lib/ceph/osd/ceph-3 2024-05-09T05:06:40.925 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns 2024-05-09T05:06:40.926 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:06:40.926 INFO:teuthology.orchestra.run.smithi001.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:06:40.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:40.926 DEBUG:teuthology.orchestra.run.smithi001:> ls /sys/fs/fuse/connections 2024-05-09T05:06:40.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:2024-05-09T05:06:40.948+0000 7fad13f3d480 -1 init, newargv = 0x7facf8005700 newargc=15 2024-05-09T05:06:40.948 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[43333]: starting ceph client 2024-05-09T05:06:40.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[43333]: starting fuse 2024-05-09T05:06:40.966 INFO:teuthology.orchestra.run.smithi001.stdout:41 2024-05-09T05:06:40.966 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [41] 2024-05-09T05:06:40.966 DEBUG:teuthology.orchestra.run.smithi001:> sudo stdin-killer -- python3 -c ' 2024-05-09T05:06:40.966 DEBUG:teuthology.orchestra.run.smithi001:> import glob 2024-05-09T05:06:40.966 DEBUG:teuthology.orchestra.run.smithi001:> import re 2024-05-09T05:06:40.966 DEBUG:teuthology.orchestra.run.smithi001:> import os 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> import subprocess 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> def _find_admin_socket(client_name): 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> files = glob.glob(asok_path) 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> mountpoint = "/home/ubuntu/cephtest/mnt.0" 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> # Given a non-glob path, it better be there 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> if "*" not in asok_path: 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> assert(len(files) == 1) 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> return files[0] 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> 2024-05-09T05:06:40.967 DEBUG:teuthology.orchestra.run.smithi001:> for f in files: 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> contents = proc_f.read() 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> if mountpoint in contents: 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> return f 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> print(_find_admin_socket("client.0")) 2024-05-09T05:06:40.968 DEBUG:teuthology.orchestra.run.smithi001:> ' 2024-05-09T05:06:41.087 INFO:teuthology.orchestra.run.smithi001.stdout:/var/run/ceph/ceph-client.0.43333.asok 2024-05-09T05:06:41.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T05:06:41.098 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.43333.asok 2024-05-09T05:06:41.098 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:41.098 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.43333.asok status 2024-05-09T05:06:41.254 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "metadata": { 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_sha1": "c9da4049f638c0f84bdd9b4e33c42e86046ac554", 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "ceph_version": "ceph version 19.0.0-3594-gc9da4049 (c9da4049f638c0f84bdd9b4e33c42e86046ac554) squid (dev)", 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "entity_id": "0", 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "hostname": "smithi001", 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "pid": "43333", 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "root": "/" 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_count": 0, 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "dentry_pinned_count": 0, 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "id": 4752, 2024-05-09T05:06:41.255 INFO:teuthology.orchestra.run.smithi001.stdout: "inst": { 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "name": { 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "client", 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4752 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 853583088 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": { 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "192.168.0.1:0", 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 853583088 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:41.256 INFO:teuthology.orchestra.run.smithi001.stdout: "inst_str": "client.4752 192.168.0.1:0/853583088", 2024-05-09T05:06:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "addr_str": "192.168.0.1:0/853583088", 2024-05-09T05:06:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "inode_count": 1, 2024-05-09T05:06:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "mds_epoch": 16, 2024-05-09T05:06:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch": 21, 2024-05-09T05:06:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "osd_epoch_barrier": 0, 2024-05-09T05:06:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "blocklisted": false, 2024-05-09T05:06:41.257 INFO:teuthology.orchestra.run.smithi001.stdout: "fs_name": "cephfs" 2024-05-09T05:06:41.257 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:41.263 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:41.263 DEBUG:teuthology.orchestra.run.smithi001:> stat --file-system '--printf=%T 2024-05-09T05:06:41.263 DEBUG:teuthology.orchestra.run.smithi001:> ' -- /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:41.280 INFO:teuthology.orchestra.run.smithi001.stdout:fuseblk 2024-05-09T05:06:41.280 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:41.280 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:06:41.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:41.349 INFO:teuthology.run_tasks:Running task mon_thrash... 2024-05-09T05:06:41.362 INFO:tasks.mon_thrash:Beginning mon_thrash... 2024-05-09T05:06:41.363 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:06:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.504+0000 7fa96d602640 1 Processor -- start 2024-05-09T05:06:41.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.504+0000 7fa96d602640 1 -- start start 2024-05-09T05:06:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.505+0000 7fa96d602640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa96815af00 0x7fa9680c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.505+0000 7fa96d602640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9680caee0 con 0x7fa9680ccee0 2024-05-09T05:06:41.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.505+0000 7fa96d602640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa9680cc120 con 0x7fa9680cf860 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.505+0000 7fa96d602640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa9680cb1c0 con 0x7fa96815af00 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.505+0000 7fa95ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa96815af00 0x7fa9680c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.505+0000 7fa95ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa96815af00 0x7fa9680c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53364/0 (socket says 172.21.15.1:53364) 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.505+0000 7fa95ffff640 1 -- 172.21.15.1:0/2986041969 learned_addr learned my addr 172.21.15.1:0/2986041969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.505+0000 7fa966ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa9680ccee0 0x7fa968150e80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59634/0 (socket says 172.21.15.1:59634) 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.506+0000 7fa95ffff640 1 -- 172.21.15.1:0/2986041969 >> v1:172.21.15.196:6789/0 conn(0x7fa9680cf860 legacy=0x7fa968164e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.506+0000 7fa95ffff640 1 -- 172.21.15.1:0/2986041969 >> v1:172.21.15.1:6789/0 conn(0x7fa9680ccee0 legacy=0x7fa968150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.506+0000 7fa95ffff640 1 -- 172.21.15.1:0/2986041969 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9680cc400 con 0x7fa96815af00 2024-05-09T05:06:41.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.506+0000 7fa9667fc640 1 -- 172.21.15.1:0/2986041969 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1877658883 0 0) 0x7fa9680caee0 con 0x7fa9680ccee0 2024-05-09T05:06:41.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.506+0000 7fa95ffff640 1 --2- 172.21.15.1:0/2986041969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680c4b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa9500098a0 tx=0x7fa95002a290 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7eb6d032ad6411f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:41.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa9667fc640 1 -- 172.21.15.1:0/2986041969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa950037070 con 0x7fa96815af00 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa9667fc640 1 -- 172.21.15.1:0/2986041969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa950032440 con 0x7fa96815af00 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa9667fc640 1 -- 172.21.15.1:0/2986041969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa950032810 con 0x7fa96815af00 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa96d602640 1 -- 172.21.15.1:0/2986041969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 msgr2=0x7fa9680c4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa96d602640 1 --2- 172.21.15.1:0/2986041969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680c4b60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa9500098a0 tx=0x7fa95002a290 comp rx=0 tx=0).stop 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa96d602640 1 -- 172.21.15.1:0/2986041969 shutdown_connections 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa96d602640 1 --2- 172.21.15.1:0/2986041969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680c4b60 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa96d602640 1 -- 172.21.15.1:0/2986041969 >> 172.21.15.1:0/2986041969 conn(0x7fa9680727f0 msgr2=0x7fa968072c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa96d602640 1 -- 172.21.15.1:0/2986041969 shutdown_connections 2024-05-09T05:06:41.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.507+0000 7fa96d602640 1 -- 172.21.15.1:0/2986041969 wait complete. 2024-05-09T05:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.509+0000 7fa96d602640 1 Processor -- start 2024-05-09T05:06:41.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.509+0000 7fa96d602640 1 -- start start 2024-05-09T05:06:41.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.510+0000 7fa96d602640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa9680ccee0 0x7fa9681f0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.510+0000 7fa96d602640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9680cf860 0x7fa9681f3f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.510+0000 7fa96d602640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680b49c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.510+0000 7fa96d602640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa9680c50a0 con 0x7fa96815af00 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.510+0000 7fa96d602640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa9680cca50 con 0x7fa9680cf860 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.510+0000 7fa96d602640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa9680caba0 con 0x7fa9680ccee0 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.510+0000 7fa9677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680b49c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa9677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680b49c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53366/0 (socket says 172.21.15.1:53366) 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa9677fe640 1 -- 172.21.15.1:0/2636503035 learned_addr learned my addr 172.21.15.1:0/2636503035 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa95ffff640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9680cf860 0x7fa9681f3f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa966ffd640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa9680ccee0 0x7fa9681f0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa9677fe640 1 -- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa9680ccee0 msgr2=0x7fa9681f0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa9677fe640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa9680ccee0 0x7fa9681f0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa9677fe640 1 -- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9680cf860 msgr2=0x7fa9681f3f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa9677fe640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9680cf860 0x7fa9681f3f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa9677fe640 1 -- 172.21.15.1:0/2636503035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9681f1100 con 0x7fa96815af00 2024-05-09T05:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa9677fe640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680b49c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa96000c700 tx=0x7fa96000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa95f7fe640 1 -- 172.21.15.1:0/2636503035 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa960003b70 con 0x7fa96815af00 2024-05-09T05:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa95f7fe640 1 -- 172.21.15.1:0/2636503035 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa960003d10 con 0x7fa96815af00 2024-05-09T05:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa95f7fe640 1 -- 172.21.15.1:0/2636503035 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa96000aa50 con 0x7fa96815af00 2024-05-09T05:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9681f44d0 con 0x7fa96815af00 2024-05-09T05:06:41.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.511+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9681e3c10 con 0x7fa96815af00 2024-05-09T05:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.512+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa968076f80 con 0x7fa96815af00 2024-05-09T05:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.513+0000 7fa95f7fe640 1 -- 172.21.15.1:0/2636503035 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa96000abf0 con 0x7fa96815af00 2024-05-09T05:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.513+0000 7fa95f7fe640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9480762d0 0x7fa9480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.514+0000 7fa95f7fe640 1 -- 172.21.15.1:0/2636503035 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fa960097430 con 0x7fa96815af00 2024-05-09T05:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.514+0000 7fa966ffd640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9480762d0 0x7fa9480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.514+0000 7fa966ffd640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9480762d0 0x7fa9480787e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa958000f60 tx=0x7fa958007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:41.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.518+0000 7fa95f7fe640 1 -- 172.21.15.1:0/2636503035 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa9681f1100 con 0x7fa96815af00 2024-05-09T05:06:41.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.710+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fa9680cb6e0 con 0x7fa96815af00 2024-05-09T05:06:41.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.712+0000 7fa95f7fe640 1 -- 172.21.15.1:0/2636503035 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) ==== 74+0+16545 (secure 0 0 0) 0x7fa960060610 con 0x7fa96815af00 2024-05-09T05:06:41.712 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:41.712 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":21,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:06:06.049536+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9480762d0 msgr2=0x7fa9480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9480762d0 0x7fa9480787e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fa958000f60 tx=0x7fa958007e40 comp rx=0 tx=0).stop 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 msgr2=0x7fa9680b49c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680b49c0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fa96000c700 tx=0x7fa96000cbc0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 shutdown_connections 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa9480762d0 0x7fa9480787e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa9680cf860 0x7fa9681f3f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa96815af00 0x7fa9680b49c0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 --2- 172.21.15.1:0/2636503035 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa9680ccee0 0x7fa9681f0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 >> 172.21.15.1:0/2636503035 conn(0x7fa9680727f0 msgr2=0x7fa9680ce6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 shutdown_connections 2024-05-09T05:06:41.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.714+0000 7fa96d602640 1 -- 172.21.15.1:0/2636503035 wait complete. 2024-05-09T05:06:41.726 INFO:tasks.mon_thrash.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T05:05:51.558449+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 8, 'score_stable': 8, 'optimal_score': 0.25, 'raw_score_acting': 2, 'raw_score_stable': 2, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T05:06:02.375743+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '20', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T05:06:03.392859+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 1, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T05:06:41.726 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T05:06:41.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.842+0000 7f09c493f640 1 Processor -- start 2024-05-09T05:06:41.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.842+0000 7f09c493f640 1 -- start start 2024-05-09T05:06:41.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09c493f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c0154530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09c493f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f09c0070f00 0x7f09c0071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09c493f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f09c00cbf90 con 0x7f09c00cae10 2024-05-09T05:06:41.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09c493f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f09c00ca3c0 con 0x7f09c0070f00 2024-05-09T05:06:41.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09c493f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f09c015a780 con 0x7f09c0072400 2024-05-09T05:06:41.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09be59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c0154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09bdd9b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f09c0070f00 0x7f09c0071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09bdd9b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f09c0070f00 0x7f09c0071310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:36244/0 (socket says 172.21.15.1:36244) 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09bed9d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f09c00cae10 0x7f09c015db60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:58672/0 (socket says 172.21.15.1:58672) 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09be59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c0154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:42450/0 (socket says 172.21.15.1:42450) 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.844+0000 7f09bdd9b640 1 -- 172.21.15.1:0/596844555 learned_addr learned my addr 172.21.15.1:0/596844555 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.845+0000 7f09bdd9b640 1 -- 172.21.15.1:0/596844555 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 msgr2=0x7f09c0154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.845+0000 7f09bdd9b640 1 --2- 172.21.15.1:0/596844555 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c0154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.845+0000 7f09bdd9b640 1 -- 172.21.15.1:0/596844555 >> v1:172.21.15.196:6789/0 conn(0x7f09c00cae10 legacy=0x7f09c015db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.845+0000 7f09bdd9b640 1 -- 172.21.15.1:0/596844555 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c00ca610 con 0x7f09c0070f00 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.845+0000 7f09bd59a640 1 -- 172.21.15.1:0/596844555 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 494911854 0 0) 0x7f09c00cbf90 con 0x7f09c00cae10 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.845+0000 7f09be59c640 1 --2- 172.21.15.1:0/596844555 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c0154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:41.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.845+0000 7f09bdd9b640 1 --2- 172.21.15.1:0/596844555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c0070f00 0x7f09c0071310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009870 tx=0x7f09a802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fae95e55f2bf1a5e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09bd59a640 1 -- 172.21.15.1:0/596844555 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8037070 con 0x7f09c0070f00 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09bd59a640 1 -- 172.21.15.1:0/596844555 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09a8032440 con 0x7f09c0070f00 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09bd59a640 1 -- 172.21.15.1:0/596844555 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09a8032810 con 0x7f09c0070f00 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09c493f640 1 -- 172.21.15.1:0/596844555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c0070f00 msgr2=0x7f09c0071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09c493f640 1 --2- 172.21.15.1:0/596844555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c0070f00 0x7f09c0071310 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f09a8009870 tx=0x7f09a802a270 comp rx=0 tx=0).stop 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09c493f640 1 -- 172.21.15.1:0/596844555 shutdown_connections 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09c493f640 1 --2- 172.21.15.1:0/596844555 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c0070f00 0x7f09c0071310 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09c493f640 1 --2- 172.21.15.1:0/596844555 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c0154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09c493f640 1 -- 172.21.15.1:0/596844555 >> 172.21.15.1:0/596844555 conn(0x7f09c00d0470 msgr2=0x7f09c00cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09c493f640 1 -- 172.21.15.1:0/596844555 shutdown_connections 2024-05-09T05:06:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.846+0000 7f09c493f640 1 -- 172.21.15.1:0/596844555 wait complete. 2024-05-09T05:06:41.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.848+0000 7f09c493f640 1 Processor -- start 2024-05-09T05:06:41.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.848+0000 7f09c493f640 1 -- start start 2024-05-09T05:06:41.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.849+0000 7f09c493f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09c0070f00 0x7f09c014f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.849+0000 7f09c493f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c01502a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09c493f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c00cae10 0x7f09c00b45e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09c493f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f09c00b27b0 con 0x7f09c0070f00 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09c493f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f09c00b2630 con 0x7f09c00cae10 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09c493f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f09c00cb910 con 0x7f09c0072400 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09c0070f00 0x7f09c014f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09c0070f00 0x7f09c014f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53370/0 (socket says 172.21.15.1:53370) 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 -- 172.21.15.1:0/3704566082 learned_addr learned my addr 172.21.15.1:0/3704566082 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09bed9d640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c00cae10 0x7f09c00b45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09bdd9b640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c01502a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 -- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 msgr2=0x7f09c01502a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c01502a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 -- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c00cae10 msgr2=0x7f09c00b45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c00cae10 0x7f09c00b45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:41.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 -- 172.21.15.1:0/3704566082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c01507e0 con 0x7f09c0070f00 2024-05-09T05:06:41.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09bed9d640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c00cae10 0x7f09c00b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:41.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09be59c640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09c0070f00 0x7f09c014f290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f09b800e960 tx=0x7f09b800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:41.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09b6ffd640 1 -- 172.21.15.1:0/3704566082 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8017070 con 0x7f09c0070f00 2024-05-09T05:06:41.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09b6ffd640 1 -- 172.21.15.1:0/3704566082 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f09b800ca00 con 0x7f09c0070f00 2024-05-09T05:06:41.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09b6ffd640 1 -- 172.21.15.1:0/3704566082 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b800cdd0 con 0x7f09c0070f00 2024-05-09T05:06:41.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.850+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c01f05f0 con 0x7f09c0070f00 2024-05-09T05:06:41.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.851+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c01f0b90 con 0x7f09c0070f00 2024-05-09T05:06:41.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.852+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09c0050870 con 0x7f09c0070f00 2024-05-09T05:06:41.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.852+0000 7f09b6ffd640 1 -- 172.21.15.1:0/3704566082 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f09b801a070 con 0x7f09c0070f00 2024-05-09T05:06:41.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.853+0000 7f09b6ffd640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09a0076300 0x7f09a0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:41.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.853+0000 7f09b6ffd640 1 -- 172.21.15.1:0/3704566082 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f09b80973f0 con 0x7f09c0070f00 2024-05-09T05:06:41.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.853+0000 7f09bdd9b640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09a0076300 0x7f09a0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:41.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.853+0000 7f09bdd9b640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09a0076300 0x7f09a0078810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f09a8034000 tx=0x7f09a802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:41.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:41.857+0000 7f09b6ffd640 1 -- 172.21.15.1:0/3704566082 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f09b8060700 con 0x7f09c0070f00 2024-05-09T05:06:42.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.047+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f09c00cbcd0 con 0x7f09c0070f00 2024-05-09T05:06:42.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.047+0000 7f09b6ffd640 1 -- 172.21.15.1:0/3704566082 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v21) ==== 93+0+10 (secure 0 0 0) 0x7f09b8064a00 con 0x7f09c0070f00 2024-05-09T05:06:42.048 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 1 2024-05-09T05:06:42.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09a0076300 msgr2=0x7f09a0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09a0076300 0x7f09a0078810 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f09a8034000 tx=0x7f09a802e000 comp rx=0 tx=0).stop 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09c0070f00 msgr2=0x7f09c014f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09c0070f00 0x7f09c014f290 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f09b800e960 tx=0x7f09b800ee20 comp rx=0 tx=0).stop 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 shutdown_connections 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f09a0076300 0x7f09a0078810 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f09c00cae10 0x7f09c00b45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f09c0072400 0x7f09c01502a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 --2- 172.21.15.1:0/3704566082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f09c0070f00 0x7f09c014f290 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.049+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 >> 172.21.15.1:0/3704566082 conn(0x7f09c00d0470 msgr2=0x7f09c00c5880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.050+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 shutdown_connections 2024-05-09T05:06:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.050+0000 7f09c493f640 1 -- 172.21.15.1:0/3704566082 wait complete. 2024-05-09T05:06:42.061 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T05:06:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.179+0000 7fd8dce7e640 1 Processor -- start 2024-05-09T05:06:42.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.179+0000 7fd8dce7e640 1 -- start start 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8dce7e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd8d80cf360 0x7fd8d80b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8dce7e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8d80b4570 0x7fd8d8156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8dce7e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd8d80ce5c0 con 0x7fd8d815c9a0 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8dce7e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd8d815d4c0 con 0x7fd8d80b4570 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8dce7e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd8d815d7a0 con 0x7fd8d80cf360 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d5d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd8d80b4570 0x7fd8d8156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd8d815c9a0 0x7fd8d80cd3f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59638/0 (socket says 172.21.15.1:59638) 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d6d9d640 1 -- 172.21.15.1:0/2902255380 learned_addr learned my addr 172.21.15.1:0/2902255380 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:42.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d659c640 1 --2- 172.21.15.1:0/2902255380 >> v2:172.21.15.196:3300/0 conn(0x7fd8d80cf360 0x7fd8d80b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d5d9b640 1 -- 172.21.15.1:0/2902255380 >> v2:172.21.15.196:3300/0 conn(0x7fd8d80cf360 msgr2=0x7fd8d80b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d5d9b640 1 --2- 172.21.15.1:0/2902255380 >> v2:172.21.15.196:3300/0 conn(0x7fd8d80cf360 0x7fd8d80b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d5d9b640 1 -- 172.21.15.1:0/2902255380 >> v1:172.21.15.1:6789/0 conn(0x7fd8d815c9a0 legacy=0x7fd8d80cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d5d9b640 1 -- 172.21.15.1:0/2902255380 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d80ce880 con 0x7fd8d80b4570 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.181+0000 7fd8d559a640 1 -- 172.21.15.1:0/2902255380 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2600808065 0 0) 0x7fd8d80ce5c0 con 0x7fd8d815c9a0 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.182+0000 7fd8d5d9b640 1 --2- 172.21.15.1:0/2902255380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d80b4570 0x7fd8d8156b50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd8d808f300 tx=0x7fd8c002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=443edde965bbacf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.182+0000 7fd8d559a640 1 -- 172.21.15.1:0/2902255380 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c0037070 con 0x7fd8d80b4570 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.182+0000 7fd8d559a640 1 -- 172.21.15.1:0/2902255380 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c0032440 con 0x7fd8d80b4570 2024-05-09T05:06:42.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.182+0000 7fd8d559a640 1 -- 172.21.15.1:0/2902255380 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c0032810 con 0x7fd8d80b4570 2024-05-09T05:06:42.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.183+0000 7fd8dce7e640 1 -- 172.21.15.1:0/2902255380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d80b4570 msgr2=0x7fd8d8156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.183+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/2902255380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d80b4570 0x7fd8d8156b50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd8d808f300 tx=0x7fd8c002a230 comp rx=0 tx=0).stop 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.183+0000 7fd8dce7e640 1 -- 172.21.15.1:0/2902255380 shutdown_connections 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.183+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/2902255380 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d80b4570 0x7fd8d8156b50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.183+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/2902255380 >> v2:172.21.15.196:3300/0 conn(0x7fd8d80cf360 0x7fd8d80b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.183+0000 7fd8dce7e640 1 -- 172.21.15.1:0/2902255380 >> 172.21.15.1:0/2902255380 conn(0x7fd8d806de10 msgr2=0x7fd8d806e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.183+0000 7fd8dce7e640 1 -- 172.21.15.1:0/2902255380 shutdown_connections 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.183+0000 7fd8dce7e640 1 -- 172.21.15.1:0/2902255380 wait complete. 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.185+0000 7fd8dce7e640 1 Processor -- start 2024-05-09T05:06:42.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.185+0000 7fd8dce7e640 1 -- start start 2024-05-09T05:06:42.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.186+0000 7fd8dce7e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd8d80b4570 0x7fd8d81e2d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.186+0000 7fd8dce7e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8d815c9a0 0x7fd8d8153200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.186+0000 7fd8dce7e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d815bd50 0x7fd8d8153740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.186+0000 7fd8dce7e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d80b5090 con 0x7fd8d815bd50 2024-05-09T05:06:42.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.186+0000 7fd8dce7e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd8d80cee40 con 0x7fd8d815c9a0 2024-05-09T05:06:42.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.186+0000 7fd8dce7e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd8d8157090 con 0x7fd8d80b4570 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.186+0000 7fd8d6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d815bd50 0x7fd8d8153740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d815bd50 0x7fd8d8153740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53398/0 (socket says 172.21.15.1:53398) 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d6d9d640 1 -- 172.21.15.1:0/4216662878 learned_addr learned my addr 172.21.15.1:0/4216662878 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d5d9b640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8d815c9a0 0x7fd8d8153200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d659c640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd8d80b4570 0x7fd8d81e2d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d6d9d640 1 -- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd8d80b4570 msgr2=0x7fd8d81e2d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d6d9d640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd8d80b4570 0x7fd8d81e2d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d6d9d640 1 -- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8d815c9a0 msgr2=0x7fd8d8153200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d6d9d640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8d815c9a0 0x7fd8d8153200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d6d9d640 1 -- 172.21.15.1:0/4216662878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8d81f33e0 con 0x7fd8d815bd50 2024-05-09T05:06:42.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8d6d9d640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d815bd50 0x7fd8d8153740 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c6b0 tx=0x7fd8c800cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8ceffd640 1 -- 172.21.15.1:0/4216662878 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c8018070 con 0x7fd8d815bd50 2024-05-09T05:06:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8ceffd640 1 -- 172.21.15.1:0/4216662878 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd8c8004da0 con 0x7fd8d815bd50 2024-05-09T05:06:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8d81e3290 con 0x7fd8d815bd50 2024-05-09T05:06:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8ceffd640 1 -- 172.21.15.1:0/4216662878 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8c800a880 con 0x7fd8d815bd50 2024-05-09T05:06:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.187+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8d81f2b40 con 0x7fd8d815bd50 2024-05-09T05:06:42.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.188+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8d8050870 con 0x7fd8d815bd50 2024-05-09T05:06:42.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.189+0000 7fd8ceffd640 1 -- 172.21.15.1:0/4216662878 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd8c801b070 con 0x7fd8d815bd50 2024-05-09T05:06:42.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.189+0000 7fd8ceffd640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd8b80762d0 0x7fd8b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.190+0000 7fd8ceffd640 1 -- 172.21.15.1:0/4216662878 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fd8c8096a40 con 0x7fd8d815bd50 2024-05-09T05:06:42.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.190+0000 7fd8d659c640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd8b80762d0 0x7fd8b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.190+0000 7fd8d659c640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd8b80762d0 0x7fd8b80787e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009aa0 tx=0x7fd8d0009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.194+0000 7fd8ceffd640 1 -- 172.21.15.1:0/4216662878 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd8c805fde0 con 0x7fd8d815bd50 2024-05-09T05:06:42.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.386+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd8d80cdf30 con 0x7fd8d815bd50 2024-05-09T05:06:42.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.386+0000 7fd8ceffd640 1 -- 172.21.15.1:0/4216662878 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v21) ==== 104+0+11 (secure 0 0 0) 0x7fd8c8068420 con 0x7fd8d815bd50 2024-05-09T05:06:42.387 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 32 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd8b80762d0 msgr2=0x7fd8b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd8b80762d0 0x7fd8b80787e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fd8d0009aa0 tx=0x7fd8d0009190 comp rx=0 tx=0).stop 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d815bd50 msgr2=0x7fd8d8153740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d815bd50 0x7fd8d8153740 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd8c800c6b0 tx=0x7fd8c800cb70 comp rx=0 tx=0).stop 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 shutdown_connections 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd8b80762d0 0x7fd8b80787e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd8d815c9a0 0x7fd8d8153200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd8d815bd50 0x7fd8d8153740 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 --2- 172.21.15.1:0/4216662878 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd8d80b4570 0x7fd8d81e2d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 >> 172.21.15.1:0/4216662878 conn(0x7fd8d806de10 msgr2=0x7fd8d806e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 shutdown_connections 2024-05-09T05:06:42.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.389+0000 7fd8dce7e640 1 -- 172.21.15.1:0/4216662878 wait complete. 2024-05-09T05:06:42.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T05:06:42.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.517+0000 7f8ac2884640 1 Processor -- start 2024-05-09T05:06:42.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.518+0000 7f8ac2884640 1 -- start start 2024-05-09T05:06:42.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8ac2884640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8abc0cae10 0x7f8abc153730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8ac2884640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0c67b0 0x7f8abc0c6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8ac2884640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8abc0c8a50 con 0x7f8abc0ca970 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8ac2884640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8abc0c9d00 con 0x7f8abc0cae10 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8ac2884640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8abc0c9870 con 0x7f8abc0c67b0 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8abbfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8abc0ca970 0x7f8abc15c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59654/0 (socket says 172.21.15.1:59654) 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8abbfff640 1 -- 172.21.15.1:0/2537305796 learned_addr learned my addr 172.21.15.1:0/2537305796 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8ac0e21640 1 --2- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0c67b0 0x7f8abc0c6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8ab3fff640 1 --2- 172.21.15.1:0/2537305796 >> v2:172.21.15.196:3300/0 conn(0x7f8abc0cae10 0x7f8abc153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8abb7fe640 1 -- 172.21.15.1:0/2537305796 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2075466296 0 0) 0x7f8abc0c8a50 con 0x7f8abc0ca970 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.519+0000 7f8abb7fe640 1 -- 172.21.15.1:0/2537305796 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8aa8003520 con 0x7f8abc0ca970 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.520+0000 7f8ab3fff640 1 -- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0c67b0 msgr2=0x7f8abc0c6bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.520+0000 7f8ab3fff640 1 --2- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0c67b0 0x7f8abc0c6bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.520+0000 7f8ab3fff640 1 -- 172.21.15.1:0/2537305796 >> v1:172.21.15.1:6789/0 conn(0x7f8abc0ca970 legacy=0x7f8abc15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.520+0000 7f8ab3fff640 1 -- 172.21.15.1:0/2537305796 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc0c8d10 con 0x7f8abc0cae10 2024-05-09T05:06:42.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.520+0000 7f8ac0e21640 1 --2- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0c67b0 0x7f8abc0c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:06:42.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.520+0000 7f8ab3fff640 1 --2- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 0x7f8abc153730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40097a0 tx=0x7f8aa402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9faa055fa7e81d12 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8abb7fe640 1 -- 172.21.15.1:0/2537305796 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa4037070 con 0x7f8abc0cae10 2024-05-09T05:06:42.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8abb7fe640 1 -- 172.21.15.1:0/2537305796 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aa4032440 con 0x7f8abc0cae10 2024-05-09T05:06:42.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8abb7fe640 1 -- 172.21.15.1:0/2537305796 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aa40327f0 con 0x7f8abc0cae10 2024-05-09T05:06:42.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8ac2884640 1 -- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 msgr2=0x7f8abc153730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 0x7f8abc153730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f8aa40097a0 tx=0x7f8aa402a180 comp rx=0 tx=0).stop 2024-05-09T05:06:42.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8ac2884640 1 -- 172.21.15.1:0/2537305796 shutdown_connections 2024-05-09T05:06:42.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0c67b0 0x7f8abc0c6bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2537305796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 0x7f8abc153730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8ac2884640 1 -- 172.21.15.1:0/2537305796 >> 172.21.15.1:0/2537305796 conn(0x7f8abc0ce910 msgr2=0x7f8abc0ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:42.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8ac2884640 1 -- 172.21.15.1:0/2537305796 shutdown_connections 2024-05-09T05:06:42.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.521+0000 7f8ac2884640 1 -- 172.21.15.1:0/2537305796 wait complete. 2024-05-09T05:06:42.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.523+0000 7f8ac2884640 1 Processor -- start 2024-05-09T05:06:42.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.523+0000 7f8ac2884640 1 -- start start 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.524+0000 7f8ac2884640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc0c67b0 0x7f8abc0cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.524+0000 7f8ac2884640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0ca970 0x7f8abc165360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.524+0000 7f8ac2884640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 0x7f8abc0cc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.524+0000 7f8ac2884640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc0c60f0 con 0x7f8abc0c67b0 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.524+0000 7f8ac2884640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8abc0c7100 con 0x7f8abc0cae10 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.524+0000 7f8ac2884640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8abc0c63d0 con 0x7f8abc0ca970 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc0c67b0 0x7f8abc0cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc0c67b0 0x7f8abc0cdd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53408/0 (socket says 172.21.15.1:53408) 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 -- 172.21.15.1:0/2594593145 learned_addr learned my addr 172.21.15.1:0/2594593145 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8ac0e21640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 0x7f8abc0cc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8ab3fff640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0ca970 0x7f8abc165360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 -- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0ca970 msgr2=0x7f8abc165360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0ca970 0x7f8abc165360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 -- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 msgr2=0x7f8abc0cc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 0x7f8abc0cc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 -- 172.21.15.1:0/2594593145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8abc1658a0 con 0x7f8abc0c67b0 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8ac0e21640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 0x7f8abc0cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8abbfff640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc0c67b0 0x7f8abc0cdd00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00c440 tx=0x7f8aac00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8ab97fa640 1 -- 172.21.15.1:0/2594593145 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac017070 con 0x7f8abc0c67b0 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8ab97fa640 1 -- 172.21.15.1:0/2594593145 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8aac005ce0 con 0x7f8abc0c67b0 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8abc134dd0 con 0x7f8abc0c67b0 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8ab97fa640 1 -- 172.21.15.1:0/2594593145 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8aac007270 con 0x7f8abc0c67b0 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.525+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8abc135390 con 0x7f8abc0c67b0 2024-05-09T05:06:42.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.526+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8abc050870 con 0x7f8abc0c67b0 2024-05-09T05:06:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.527+0000 7f8ab97fa640 1 -- 172.21.15.1:0/2594593145 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8aac01a070 con 0x7f8abc0c67b0 2024-05-09T05:06:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.528+0000 7f8ab97fa640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a9c0762d0 0x7f8a9c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.528+0000 7f8ab97fa640 1 -- 172.21.15.1:0/2594593145 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f8aac096c90 con 0x7f8abc0c67b0 2024-05-09T05:06:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.528+0000 7f8ab3fff640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a9c0762d0 0x7f8a9c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.528+0000 7f8ab3fff640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a9c0762d0 0x7f8a9c0787e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4034000 tx=0x7f8aa402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.532+0000 7f8ab97fa640 1 -- 172.21.15.1:0/2594593145 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8aac060010 con 0x7f8abc0c67b0 2024-05-09T05:06:42.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.722+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8abc14d7e0 con 0x7f8abc0c67b0 2024-05-09T05:06:42.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.723+0000 7f8ab97fa640 1 -- 172.21.15.1:0/2594593145 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v21) ==== 100+0+11 (secure 0 0 0) 0x7f8aac064310 con 0x7f8abc0c67b0 2024-05-09T05:06:42.724 INFO:teuthology.orchestra.run.smithi001.stdout:pg_num: 64 2024-05-09T05:06:42.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a9c0762d0 msgr2=0x7f8a9c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a9c0762d0 0x7f8a9c0787e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8aa4034000 tx=0x7f8aa402e000 comp rx=0 tx=0).stop 2024-05-09T05:06:42.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc0c67b0 msgr2=0x7f8abc0cdd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc0c67b0 0x7f8abc0cdd00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8aac00c440 tx=0x7f8aac00c900 comp rx=0 tx=0).stop 2024-05-09T05:06:42.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 shutdown_connections 2024-05-09T05:06:42.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a9c0762d0 0x7f8a9c0787e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8abc0cae10 0x7f8abc0cc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8abc0ca970 0x7f8abc165360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 --2- 172.21.15.1:0/2594593145 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8abc0c67b0 0x7f8abc0cdd00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 >> 172.21.15.1:0/2594593145 conn(0x7f8abc0ce910 msgr2=0x7f8abc140640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:42.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 shutdown_connections 2024-05-09T05:06:42.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.725+0000 7f8ac2884640 1 -- 172.21.15.1:0/2594593145 wait complete. 2024-05-09T05:06:42.737 INFO:tasks.mon_thrash.ceph_manager:waiting for clean 2024-05-09T05:06:42.737 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:06:42.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.851+0000 7fb68998c640 1 Processor -- start 2024-05-09T05:06:42.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.851+0000 7fb68998c640 1 -- start start 2024-05-09T05:06:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.852+0000 7fb68998c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb6840cc4c0 0x7fb6840c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.852+0000 7fb68998c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb6840c7500 0x7fb6841569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.852+0000 7fb68998c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb68415d3f0 con 0x7fb6840c6a80 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.852+0000 7fb68998c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb6840c82d0 con 0x7fb6840cc4c0 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.852+0000 7fb68998c640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb6840c8ca0 con 0x7fb6840c7500 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb682ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb6840cc4c0 0x7fb6840c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb6837fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb6840c6a80 0x7fb6840ce550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59662/0 (socket says 172.21.15.1:59662) 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb6837fe640 1 -- 172.21.15.1:0/533767052 learned_addr learned my addr 172.21.15.1:0/533767052 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb6827fc640 1 --2- 172.21.15.1:0/533767052 >> v2:172.21.15.196:3300/0 conn(0x7fb6840c7500 0x7fb6841569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb682ffd640 1 -- 172.21.15.1:0/533767052 >> v2:172.21.15.196:3300/0 conn(0x7fb6840c7500 msgr2=0x7fb6841569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb682ffd640 1 --2- 172.21.15.1:0/533767052 >> v2:172.21.15.196:3300/0 conn(0x7fb6840c7500 0x7fb6841569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb682ffd640 1 -- 172.21.15.1:0/533767052 >> v1:172.21.15.1:6789/0 conn(0x7fb6840c6a80 legacy=0x7fb6840ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb682ffd640 1 -- 172.21.15.1:0/533767052 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6840c7c80 con 0x7fb6840cc4c0 2024-05-09T05:06:42.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.853+0000 7fb681ffb640 1 -- 172.21.15.1:0/533767052 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 360687175 0 0) 0x7fb68415d3f0 con 0x7fb6840c6a80 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.854+0000 7fb682ffd640 1 --2- 172.21.15.1:0/533767052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840cc4c0 0x7fb6840c8760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb68408f500 tx=0x7fb67402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4cbcf6a8e268a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.854+0000 7fb681ffb640 1 -- 172.21.15.1:0/533767052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674037070 con 0x7fb6840cc4c0 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.854+0000 7fb681ffb640 1 -- 172.21.15.1:0/533767052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb674032440 con 0x7fb6840cc4c0 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.854+0000 7fb681ffb640 1 -- 172.21.15.1:0/533767052 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674032810 con 0x7fb6840cc4c0 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.855+0000 7fb68998c640 1 -- 172.21.15.1:0/533767052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840cc4c0 msgr2=0x7fb6840c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.855+0000 7fb68998c640 1 --2- 172.21.15.1:0/533767052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840cc4c0 0x7fb6840c8760 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fb68408f500 tx=0x7fb67402a230 comp rx=0 tx=0).stop 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.855+0000 7fb68998c640 1 -- 172.21.15.1:0/533767052 shutdown_connections 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.855+0000 7fb68998c640 1 --2- 172.21.15.1:0/533767052 >> v2:172.21.15.196:3300/0 conn(0x7fb6840c7500 0x7fb6841569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.855+0000 7fb68998c640 1 --2- 172.21.15.1:0/533767052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840cc4c0 0x7fb6840c8760 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.855+0000 7fb68998c640 1 -- 172.21.15.1:0/533767052 >> 172.21.15.1:0/533767052 conn(0x7fb684068650 msgr2=0x7fb684068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:42.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.855+0000 7fb68998c640 1 -- 172.21.15.1:0/533767052 shutdown_connections 2024-05-09T05:06:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.855+0000 7fb68998c640 1 -- 172.21.15.1:0/533767052 wait complete. 2024-05-09T05:06:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.856+0000 7fb68998c640 1 Processor -- start 2024-05-09T05:06:42.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.857+0000 7fb68998c640 1 -- start start 2024-05-09T05:06:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb68998c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840c6a80 0x7fb684155f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb68998c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb6840c7500 0x7fb6841f2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb68998c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6840cc4c0 0x7fb684159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb68998c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb68415d6d0 con 0x7fb6840c6a80 2024-05-09T05:06:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb68998c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb684156f00 con 0x7fb6840cc4c0 2024-05-09T05:06:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb68998c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb6840c9cd0 con 0x7fb6840c7500 2024-05-09T05:06:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840c6a80 0x7fb684155f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840c6a80 0x7fb684155f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53424/0 (socket says 172.21.15.1:53424) 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 -- 172.21.15.1:0/3849026019 learned_addr learned my addr 172.21.15.1:0/3849026019 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb6837fe640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6840cc4c0 0x7fb684159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb6827fc640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb6840c7500 0x7fb6841f2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 -- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb6840c7500 msgr2=0x7fb6841f2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb6840c7500 0x7fb6841f2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 -- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6840cc4c0 msgr2=0x7fb684159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6840cc4c0 0x7fb684159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 -- 172.21.15.1:0/3849026019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6841f3220 con 0x7fb6840c6a80 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb6827fc640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb6840c7500 0x7fb6841f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.858+0000 7fb682ffd640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840c6a80 0x7fb684155f90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb674035040 tx=0x7fb6740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.859+0000 7fb67bfff640 1 -- 172.21.15.1:0/3849026019 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674037050 con 0x7fb6840c6a80 2024-05-09T05:06:42.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.859+0000 7fb67bfff640 1 -- 172.21.15.1:0/3849026019 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb674038040 con 0x7fb6840c6a80 2024-05-09T05:06:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.859+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb68415dbe0 con 0x7fb6840c6a80 2024-05-09T05:06:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.859+0000 7fb67bfff640 1 -- 172.21.15.1:0/3849026019 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb674004470 con 0x7fb6840c6a80 2024-05-09T05:06:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.859+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb68415e1a0 con 0x7fb6840c6a80 2024-05-09T05:06:42.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.860+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb684050870 con 0x7fb6840c6a80 2024-05-09T05:06:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.860+0000 7fb67bfff640 1 -- 172.21.15.1:0/3849026019 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb674003610 con 0x7fb6840c6a80 2024-05-09T05:06:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.861+0000 7fb67bfff640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6640762d0 0x7fb6640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.861+0000 7fb67bfff640 1 -- 172.21.15.1:0/3849026019 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fb6740b7ef0 con 0x7fb6840c6a80 2024-05-09T05:06:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.861+0000 7fb6827fc640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6640762d0 0x7fb6640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:42.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.862+0000 7fb6827fc640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6640762d0 0x7fb6640787e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb68415aed0 tx=0x7fb66c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:42.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:42.866+0000 7fb67bfff640 1 -- 172.21.15.1:0/3849026019 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb67404c020 con 0x7fb6840c6a80 2024-05-09T05:06:43.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.053+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fb6840cbbf0 con 0x7fb6640762d0 2024-05-09T05:06:43.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.059+0000 7fb67bfff640 1 -- 172.21.15.1:0/3849026019 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252011 (secure 0 0 0) 0x7fb6840cbbf0 con 0x7fb6640762d0 2024-05-09T05:06:43.062 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:43.064 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:06:43.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6640762d0 msgr2=0x7fb6640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6640762d0 0x7fb6640787e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb68415aed0 tx=0x7fb66c008040 comp rx=0 tx=0).stop 2024-05-09T05:06:43.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840c6a80 msgr2=0x7fb684155f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840c6a80 0x7fb684155f90 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fb674035040 tx=0x7fb6740046c0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 shutdown_connections 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6640762d0 0x7fb6640787e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6840cc4c0 0x7fb684159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb6840c7500 0x7fb6841f2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 --2- 172.21.15.1:0/3849026019 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6840c6a80 0x7fb684155f90 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 >> 172.21.15.1:0/3849026019 conn(0x7fb684068650 msgr2=0x7fb6840ce550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 shutdown_connections 2024-05-09T05:06:43.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.063+0000 7fb68998c640 1 -- 172.21.15.1:0/3849026019 wait complete. 2024-05-09T05:06:43.075 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-05-09T05:06:41.551980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":305,"num_write_kb":1690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239976,"kb_used_data":3048,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734744,"statfs":{"total":773094113280,"available":772848377856,"internally_reserved":0,"allocated":3121152,"data_stored":1785036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2140,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002734"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737187+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:06:06.737187+0000","last_peered":"2024-05-09T05:06:06.737187+0000","last_clean":"2024-05-09T05:06:06.737187+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:06:06.737187+0000","last_undegraded":"2024-05-09T05:06:06.737187+0000","last_fullsized":"2024-05-09T05:06:06.737187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058883+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:06:06.058883+0000","last_peered":"2024-05-09T05:06:06.058883+0000","last_clean":"2024-05-09T05:06:06.058883+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:06:06.058883+0000","last_undegraded":"2024-05-09T05:06:06.058883+0000","last_fullsized":"2024-05-09T05:06:06.058883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736013+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:06:06.736013+0000","last_peered":"2024-05-09T05:06:06.736013+0000","last_clean":"2024-05-09T05:06:06.736013+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:06:06.736013+0000","last_undegraded":"2024-05-09T05:06:06.736013+0000","last_fullsized":"2024-05-09T05:06:06.736013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:48.770365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062625+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:06:06.062625+0000","last_peered":"2024-05-09T05:06:06.062625+0000","last_clean":"2024-05-09T05:06:06.062625+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:06:06.062625+0000","last_undegraded":"2024-05-09T05:06:06.062625+0000","last_fullsized":"2024-05-09T05:06:06.062625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737255+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:06:06.737255+0000","last_peered":"2024-05-09T05:06:06.737255+0000","last_clean":"2024-05-09T05:06:06.737255+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:06:06.737255+0000","last_undegraded":"2024-05-09T05:06:06.737255+0000","last_fullsized":"2024-05-09T05:06:06.737255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062081+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:06:06.062081+0000","last_peered":"2024-05-09T05:06:06.062081+0000","last_clean":"2024-05-09T05:06:06.062081+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:06:06.062081+0000","last_undegraded":"2024-05-09T05:06:06.062081+0000","last_fullsized":"2024-05-09T05:06:06.062081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736570+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:06:06.736570+0000","last_peered":"2024-05-09T05:06:06.736570+0000","last_clean":"2024-05-09T05:06:06.736570+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:06:06.736570+0000","last_undegraded":"2024-05-09T05:06:06.736570+0000","last_fullsized":"2024-05-09T05:06:06.736570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735634+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:06:06.735634+0000","last_peered":"2024-05-09T05:06:06.735634+0000","last_clean":"2024-05-09T05:06:06.735634+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:06:06.735634+0000","last_undegraded":"2024-05-09T05:06:06.735634+0000","last_fullsized":"2024-05-09T05:06:06.735634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737253+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:06:06.737253+0000","last_peered":"2024-05-09T05:06:06.737253+0000","last_clean":"2024-05-09T05:06:06.737253+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:06:06.737253+0000","last_undegraded":"2024-05-09T05:06:06.737253+0000","last_fullsized":"2024-05-09T05:06:06.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736506+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:06:06.736506+0000","last_peered":"2024-05-09T05:06:06.736506+0000","last_clean":"2024-05-09T05:06:06.736506+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:06:06.736506+0000","last_undegraded":"2024-05-09T05:06:06.736506+0000","last_fullsized":"2024-05-09T05:06:06.736506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736352+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:06:06.736352+0000","last_peered":"2024-05-09T05:06:06.736352+0000","last_clean":"2024-05-09T05:06:06.736352+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:06:06.736352+0000","last_undegraded":"2024-05-09T05:06:06.736352+0000","last_fullsized":"2024-05-09T05:06:06.736352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722788+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:06:06.722788+0000","last_peered":"2024-05-09T05:06:06.722788+0000","last_clean":"2024-05-09T05:06:06.722788+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:06:06.722788+0000","last_undegraded":"2024-05-09T05:06:06.722788+0000","last_fullsized":"2024-05-09T05:06:06.722788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736075+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:06:06.736075+0000","last_peered":"2024-05-09T05:06:06.736075+0000","last_clean":"2024-05-09T05:06:06.736075+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:06:06.736075+0000","last_undegraded":"2024-05-09T05:06:06.736075+0000","last_fullsized":"2024-05-09T05:06:06.736075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736071+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:06:06.736071+0000","last_peered":"2024-05-09T05:06:06.736071+0000","last_clean":"2024-05-09T05:06:06.736071+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:06:06.736071+0000","last_undegraded":"2024-05-09T05:06:06.736071+0000","last_fullsized":"2024-05-09T05:06:06.736071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:37.258923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058421+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:06:06.058421+0000","last_peered":"2024-05-09T05:06:06.058421+0000","last_clean":"2024-05-09T05:06:06.058421+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:06:06.058421+0000","last_undegraded":"2024-05-09T05:06:06.058421+0000","last_fullsized":"2024-05-09T05:06:06.058421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062588+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:06:06.062588+0000","last_peered":"2024-05-09T05:06:06.062588+0000","last_clean":"2024-05-09T05:06:06.062588+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:06:06.062588+0000","last_undegraded":"2024-05-09T05:06:06.062588+0000","last_fullsized":"2024-05-09T05:06:06.062588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062864+0000","last_change":"2024-05-09T05:06:05.071898+0000","last_active":"2024-05-09T05:06:06.062864+0000","last_peered":"2024-05-09T05:06:06.062864+0000","last_clean":"2024-05-09T05:06:06.062864+0000","last_became_active":"2024-05-09T05:06:05.071618+0000","last_became_peered":"2024-05-09T05:06:05.071618+0000","last_unstale":"2024-05-09T05:06:06.062864+0000","last_undegraded":"2024-05-09T05:06:06.062864+0000","last_fullsized":"2024-05-09T05:06:06.062864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060053+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:06:06.060053+0000","last_peered":"2024-05-09T05:06:06.060053+0000","last_clean":"2024-05-09T05:06:06.060053+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:06:06.060053+0000","last_undegraded":"2024-05-09T05:06:06.060053+0000","last_fullsized":"2024-05-09T05:06:06.060053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737104+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:06:06.737104+0000","last_peered":"2024-05-09T05:06:06.737104+0000","last_clean":"2024-05-09T05:06:06.737104+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:06:06.737104+0000","last_undegraded":"2024-05-09T05:06:06.737104+0000","last_fullsized":"2024-05-09T05:06:06.737104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736090+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:06:06.736090+0000","last_peered":"2024-05-09T05:06:06.736090+0000","last_clean":"2024-05-09T05:06:06.736090+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:06:06.736090+0000","last_undegraded":"2024-05-09T05:06:06.736090+0000","last_fullsized":"2024-05-09T05:06:06.736090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736595+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:06:06.736595+0000","last_peered":"2024-05-09T05:06:06.736595+0000","last_clean":"2024-05-09T05:06:06.736595+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:06:06.736595+0000","last_undegraded":"2024-05-09T05:06:06.736595+0000","last_fullsized":"2024-05-09T05:06:06.736595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:02.665344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062276+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:06:06.062276+0000","last_peered":"2024-05-09T05:06:06.062276+0000","last_clean":"2024-05-09T05:06:06.062276+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:06:06.062276+0000","last_undegraded":"2024-05-09T05:06:06.062276+0000","last_fullsized":"2024-05-09T05:06:06.062276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735814+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:06:06.735814+0000","last_peered":"2024-05-09T05:06:06.735814+0000","last_clean":"2024-05-09T05:06:06.735814+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:06:06.735814+0000","last_undegraded":"2024-05-09T05:06:06.735814+0000","last_fullsized":"2024-05-09T05:06:06.735814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062254+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062254+0000","last_peered":"2024-05-09T05:06:06.062254+0000","last_clean":"2024-05-09T05:06:06.062254+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:06:06.062254+0000","last_undegraded":"2024-05-09T05:06:06.062254+0000","last_fullsized":"2024-05-09T05:06:06.062254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736447+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:06:06.736447+0000","last_peered":"2024-05-09T05:06:06.736447+0000","last_clean":"2024-05-09T05:06:06.736447+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:06:06.736447+0000","last_undegraded":"2024-05-09T05:06:06.736447+0000","last_fullsized":"2024-05-09T05:06:06.736447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:25:28.166958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736699+0000","last_change":"2024-05-09T05:06:05.067878+0000","last_active":"2024-05-09T05:06:06.736699+0000","last_peered":"2024-05-09T05:06:06.736699+0000","last_clean":"2024-05-09T05:06:06.736699+0000","last_became_active":"2024-05-09T05:06:05.067426+0000","last_became_peered":"2024-05-09T05:06:05.067426+0000","last_unstale":"2024-05-09T05:06:06.736699+0000","last_undegraded":"2024-05-09T05:06:06.736699+0000","last_fullsized":"2024-05-09T05:06:06.736699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058528+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:06:06.058528+0000","last_peered":"2024-05-09T05:06:06.058528+0000","last_clean":"2024-05-09T05:06:06.058528+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:06:06.058528+0000","last_undegraded":"2024-05-09T05:06:06.058528+0000","last_fullsized":"2024-05-09T05:06:06.058528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736532+0000","last_change":"2024-05-09T05:06:05.063359+0000","last_active":"2024-05-09T05:06:06.736532+0000","last_peered":"2024-05-09T05:06:06.736532+0000","last_clean":"2024-05-09T05:06:06.736532+0000","last_became_active":"2024-05-09T05:06:05.063026+0000","last_became_peered":"2024-05-09T05:06:05.063026+0000","last_unstale":"2024-05-09T05:06:06.736532+0000","last_undegraded":"2024-05-09T05:06:06.736532+0000","last_fullsized":"2024-05-09T05:06:06.736532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736550+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:06:06.736550+0000","last_peered":"2024-05-09T05:06:06.736550+0000","last_clean":"2024-05-09T05:06:06.736550+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:06:06.736550+0000","last_undegraded":"2024-05-09T05:06:06.736550+0000","last_fullsized":"2024-05-09T05:06:06.736550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062682+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:06:06.062682+0000","last_peered":"2024-05-09T05:06:06.062682+0000","last_clean":"2024-05-09T05:06:06.062682+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:06:06.062682+0000","last_undegraded":"2024-05-09T05:06:06.062682+0000","last_fullsized":"2024-05-09T05:06:06.062682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059766+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:06:06.059766+0000","last_peered":"2024-05-09T05:06:06.059766+0000","last_clean":"2024-05-09T05:06:06.059766+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:06:06.059766+0000","last_undegraded":"2024-05-09T05:06:06.059766+0000","last_fullsized":"2024-05-09T05:06:06.059766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736152+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:06:06.736152+0000","last_peered":"2024-05-09T05:06:06.736152+0000","last_clean":"2024-05-09T05:06:06.736152+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:06:06.736152+0000","last_undegraded":"2024-05-09T05:06:06.736152+0000","last_fullsized":"2024-05-09T05:06:06.736152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722480+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:06:06.722480+0000","last_peered":"2024-05-09T05:06:06.722480+0000","last_clean":"2024-05-09T05:06:06.722480+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:06:06.722480+0000","last_undegraded":"2024-05-09T05:06:06.722480+0000","last_fullsized":"2024-05-09T05:06:06.722480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.099849+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:06:15.099849+0000","last_peered":"2024-05-09T05:06:15.099849+0000","last_clean":"2024-05-09T05:06:15.099849+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:06:15.099849+0000","last_undegraded":"2024-05-09T05:06:15.099849+0000","last_fullsized":"2024-05-09T05:06:15.099849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736786+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:06:06.736786+0000","last_peered":"2024-05-09T05:06:06.736786+0000","last_clean":"2024-05-09T05:06:06.736786+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:06:06.736786+0000","last_undegraded":"2024-05-09T05:06:06.736786+0000","last_fullsized":"2024-05-09T05:06:06.736786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135868+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:06:27.135868+0000","last_peered":"2024-05-09T05:06:27.135868+0000","last_clean":"2024-05-09T05:06:27.135868+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:06:27.135868+0000","last_undegraded":"2024-05-09T05:06:27.135868+0000","last_fullsized":"2024-05-09T05:06:27.135868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:21.118094+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:06:21.118094+0000","last_peered":"2024-05-09T05:06:21.118094+0000","last_clean":"2024-05-09T05:06:21.118094+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:06:21.118094+0000","last_undegraded":"2024-05-09T05:06:21.118094+0000","last_fullsized":"2024-05-09T05:06:21.118094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735754+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:06:06.735754+0000","last_peered":"2024-05-09T05:06:06.735754+0000","last_clean":"2024-05-09T05:06:06.735754+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:06:06.735754+0000","last_undegraded":"2024-05-09T05:06:06.735754+0000","last_fullsized":"2024-05-09T05:06:06.735754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.137356+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:06:27.137356+0000","last_peered":"2024-05-09T05:06:27.137356+0000","last_clean":"2024-05-09T05:06:27.137356+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:06:27.137356+0000","last_undegraded":"2024-05-09T05:06:27.137356+0000","last_fullsized":"2024-05-09T05:06:27.137356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058418+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:06:06.058418+0000","last_peered":"2024-05-09T05:06:06.058418+0000","last_clean":"2024-05-09T05:06:06.058418+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:06:06.058418+0000","last_undegraded":"2024-05-09T05:06:06.058418+0000","last_fullsized":"2024-05-09T05:06:06.058418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.094128+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:06:15.094128+0000","last_peered":"2024-05-09T05:06:15.094128+0000","last_clean":"2024-05-09T05:06:15.094128+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:06:15.094128+0000","last_undegraded":"2024-05-09T05:06:15.094128+0000","last_fullsized":"2024-05-09T05:06:15.094128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062383+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:06:06.062383+0000","last_peered":"2024-05-09T05:06:06.062383+0000","last_clean":"2024-05-09T05:06:06.062383+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:06:06.062383+0000","last_undegraded":"2024-05-09T05:06:06.062383+0000","last_fullsized":"2024-05-09T05:06:06.062383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161228+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:06:33.161228+0000","last_peered":"2024-05-09T05:06:33.161228+0000","last_clean":"2024-05-09T05:06:33.161228+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:06:33.161228+0000","last_undegraded":"2024-05-09T05:06:33.161228+0000","last_fullsized":"2024-05-09T05:06:33.161228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736605+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:06:06.736605+0000","last_peered":"2024-05-09T05:06:06.736605+0000","last_clean":"2024-05-09T05:06:06.736605+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:06:06.736605+0000","last_undegraded":"2024-05-09T05:06:06.736605+0000","last_fullsized":"2024-05-09T05:06:06.736605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:26.607004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.099386+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:06:15.099386+0000","last_peered":"2024-05-09T05:06:15.099386+0000","last_clean":"2024-05-09T05:06:15.099386+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:06:15.099386+0000","last_undegraded":"2024-05-09T05:06:15.099386+0000","last_fullsized":"2024-05-09T05:06:15.099386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722806+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:06:06.722806+0000","last_peered":"2024-05-09T05:06:06.722806+0000","last_clean":"2024-05-09T05:06:06.722806+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:06:06.722806+0000","last_undegraded":"2024-05-09T05:06:06.722806+0000","last_fullsized":"2024-05-09T05:06:06.722806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.159438+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:06:33.159438+0000","last_peered":"2024-05-09T05:06:33.159438+0000","last_clean":"2024-05-09T05:06:33.159438+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:06:33.159438+0000","last_undegraded":"2024-05-09T05:06:33.159438+0000","last_fullsized":"2024-05-09T05:06:33.159438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736736+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:06:06.736736+0000","last_peered":"2024-05-09T05:06:06.736736+0000","last_clean":"2024-05-09T05:06:06.736736+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:06:06.736736+0000","last_undegraded":"2024-05-09T05:06:06.736736+0000","last_fullsized":"2024-05-09T05:06:06.736736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161901+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:06:33.161901+0000","last_peered":"2024-05-09T05:06:33.161901+0000","last_clean":"2024-05-09T05:06:33.161901+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:06:33.161901+0000","last_undegraded":"2024-05-09T05:06:33.161901+0000","last_fullsized":"2024-05-09T05:06:33.161901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736396+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:06:06.736396+0000","last_peered":"2024-05-09T05:06:06.736396+0000","last_clean":"2024-05-09T05:06:06.736396+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:06:06.736396+0000","last_undegraded":"2024-05-09T05:06:06.736396+0000","last_fullsized":"2024-05-09T05:06:06.736396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:41.283927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.137358+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:06:27.137358+0000","last_peered":"2024-05-09T05:06:27.137358+0000","last_clean":"2024-05-09T05:06:27.137358+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:06:27.137358+0000","last_undegraded":"2024-05-09T05:06:27.137358+0000","last_fullsized":"2024-05-09T05:06:27.137358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737118+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:06:06.737118+0000","last_peered":"2024-05-09T05:06:06.737118+0000","last_clean":"2024-05-09T05:06:06.737118+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:06:06.737118+0000","last_undegraded":"2024-05-09T05:06:06.737118+0000","last_fullsized":"2024-05-09T05:06:06.737118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135525+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:06:27.135525+0000","last_peered":"2024-05-09T05:06:27.135525+0000","last_clean":"2024-05-09T05:06:27.135525+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:06:27.135525+0000","last_undegraded":"2024-05-09T05:06:27.135525+0000","last_fullsized":"2024-05-09T05:06:27.135525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736621+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:06:06.736621+0000","last_peered":"2024-05-09T05:06:06.736621+0000","last_clean":"2024-05-09T05:06:06.736621+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:06:06.736621+0000","last_undegraded":"2024-05-09T05:06:06.736621+0000","last_fullsized":"2024-05-09T05:06:06.736621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":31,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.134077+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:06:27.134077+0000","last_peered":"2024-05-09T05:06:27.134077+0000","last_clean":"2024-05-09T05:06:27.134077+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:06:27.134077+0000","last_undegraded":"2024-05-09T05:06:27.134077+0000","last_fullsized":"2024-05-09T05:06:27.134077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736921+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:06:06.736921+0000","last_peered":"2024-05-09T05:06:06.736921+0000","last_clean":"2024-05-09T05:06:06.736921+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:06:06.736921+0000","last_undegraded":"2024-05-09T05:06:06.736921+0000","last_fullsized":"2024-05-09T05:06:06.736921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.052196+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:06:10.052196+0000","last_peered":"2024-05-09T05:06:10.052196+0000","last_clean":"2024-05-09T05:06:10.052196+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:06:10.052196+0000","last_undegraded":"2024-05-09T05:06:10.052196+0000","last_fullsized":"2024-05-09T05:06:10.052196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737109+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:06:06.737109+0000","last_peered":"2024-05-09T05:06:06.737109+0000","last_clean":"2024-05-09T05:06:06.737109+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:06:06.737109+0000","last_undegraded":"2024-05-09T05:06:06.737109+0000","last_fullsized":"2024-05-09T05:06:06.737109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161496+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:06:33.161496+0000","last_peered":"2024-05-09T05:06:33.161496+0000","last_clean":"2024-05-09T05:06:33.161496+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:06:33.161496+0000","last_undegraded":"2024-05-09T05:06:33.161496+0000","last_fullsized":"2024-05-09T05:06:33.161496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737391+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:06:06.737391+0000","last_peered":"2024-05-09T05:06:06.737391+0000","last_clean":"2024-05-09T05:06:06.737391+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:06:06.737391+0000","last_undegraded":"2024-05-09T05:06:06.737391+0000","last_fullsized":"2024-05-09T05:06:06.737391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135984+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:06:27.135984+0000","last_peered":"2024-05-09T05:06:27.135984+0000","last_clean":"2024-05-09T05:06:27.135984+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:06:27.135984+0000","last_undegraded":"2024-05-09T05:06:27.135984+0000","last_fullsized":"2024-05-09T05:06:27.135984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:06:06.062865+0000","last_peered":"2024-05-09T05:06:06.062865+0000","last_clean":"2024-05-09T05:06:06.062865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:06:06.062865+0000","last_undegraded":"2024-05-09T05:06:06.062865+0000","last_fullsized":"2024-05-09T05:06:06.062865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.140130+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:06:27.140130+0000","last_peered":"2024-05-09T05:06:27.140130+0000","last_clean":"2024-05-09T05:06:27.140130+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:06:27.140130+0000","last_undegraded":"2024-05-09T05:06:27.140130+0000","last_fullsized":"2024-05-09T05:06:27.140130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736885+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:06:06.736885+0000","last_peered":"2024-05-09T05:06:06.736885+0000","last_clean":"2024-05-09T05:06:06.736885+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:06:06.736885+0000","last_undegraded":"2024-05-09T05:06:06.736885+0000","last_fullsized":"2024-05-09T05:06:06.736885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736326+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:06:06.736326+0000","last_peered":"2024-05-09T05:06:06.736326+0000","last_clean":"2024-05-09T05:06:06.736326+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:06:06.736326+0000","last_undegraded":"2024-05-09T05:06:06.736326+0000","last_fullsized":"2024-05-09T05:06:06.736326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.139983+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:06:27.139983+0000","last_peered":"2024-05-09T05:06:27.139983+0000","last_clean":"2024-05-09T05:06:27.139983+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:06:27.139983+0000","last_undegraded":"2024-05-09T05:06:27.139983+0000","last_fullsized":"2024-05-09T05:06:27.139983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735899+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:06:06.735899+0000","last_peered":"2024-05-09T05:06:06.735899+0000","last_clean":"2024-05-09T05:06:06.735899+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:06:06.735899+0000","last_undegraded":"2024-05-09T05:06:06.735899+0000","last_fullsized":"2024-05-09T05:06:06.735899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:21.117843+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:06:21.117843+0000","last_peered":"2024-05-09T05:06:21.117843+0000","last_clean":"2024-05-09T05:06:21.117843+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:06:21.117843+0000","last_undegraded":"2024-05-09T05:06:21.117843+0000","last_fullsized":"2024-05-09T05:06:21.117843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736254+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:06:06.736254+0000","last_peered":"2024-05-09T05:06:06.736254+0000","last_clean":"2024-05-09T05:06:06.736254+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:06:06.736254+0000","last_undegraded":"2024-05-09T05:06:06.736254+0000","last_fullsized":"2024-05-09T05:06:06.736254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:23.362012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:21.117677+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:06:21.117677+0000","last_peered":"2024-05-09T05:06:21.117677+0000","last_clean":"2024-05-09T05:06:21.117677+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:06:21.117677+0000","last_undegraded":"2024-05-09T05:06:21.117677+0000","last_fullsized":"2024-05-09T05:06:21.117677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058422+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:06:06.058422+0000","last_peered":"2024-05-09T05:06:06.058422+0000","last_clean":"2024-05-09T05:06:06.058422+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:06:06.058422+0000","last_undegraded":"2024-05-09T05:06:06.058422+0000","last_fullsized":"2024-05-09T05:06:06.058422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161352+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:06:33.161352+0000","last_peered":"2024-05-09T05:06:33.161352+0000","last_clean":"2024-05-09T05:06:33.161352+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:06:33.161352+0000","last_undegraded":"2024-05-09T05:06:33.161352+0000","last_fullsized":"2024-05-09T05:06:33.161352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722592+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:06:06.722592+0000","last_peered":"2024-05-09T05:06:06.722592+0000","last_clean":"2024-05-09T05:06:06.722592+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:06:06.722592+0000","last_undegraded":"2024-05-09T05:06:06.722592+0000","last_fullsized":"2024-05-09T05:06:06.722592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.555529+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:06:06.555529+0000","last_peered":"2024-05-09T05:06:06.555529+0000","last_clean":"2024-05-09T05:06:06.555529+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:06:06.555529+0000","last_undegraded":"2024-05-09T05:06:06.555529+0000","last_fullsized":"2024-05-09T05:06:06.555529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062852+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:06:06.062852+0000","last_peered":"2024-05-09T05:06:06.062852+0000","last_clean":"2024-05-09T05:06:06.062852+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:06:06.062852+0000","last_undegraded":"2024-05-09T05:06:06.062852+0000","last_fullsized":"2024-05-09T05:06:06.062852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135199+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:06:27.135199+0000","last_peered":"2024-05-09T05:06:27.135199+0000","last_clean":"2024-05-09T05:06:27.135199+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:06:27.135199+0000","last_undegraded":"2024-05-09T05:06:27.135199+0000","last_fullsized":"2024-05-09T05:06:27.135199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058622+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:06:06.058622+0000","last_peered":"2024-05-09T05:06:06.058622+0000","last_clean":"2024-05-09T05:06:06.058622+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:06:06.058622+0000","last_undegraded":"2024-05-09T05:06:06.058622+0000","last_fullsized":"2024-05-09T05:06:06.058622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.140453+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:06:27.140453+0000","last_peered":"2024-05-09T05:06:27.140453+0000","last_clean":"2024-05-09T05:06:27.140453+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:06:27.140453+0000","last_undegraded":"2024-05-09T05:06:27.140453+0000","last_fullsized":"2024-05-09T05:06:27.140453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736837+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:06:06.736837+0000","last_peered":"2024-05-09T05:06:06.736837+0000","last_clean":"2024-05-09T05:06:06.736837+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:06:06.736837+0000","last_undegraded":"2024-05-09T05:06:06.736837+0000","last_fullsized":"2024-05-09T05:06:06.736837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:33.840344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.160660+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:06:33.160660+0000","last_peered":"2024-05-09T05:06:33.160660+0000","last_clean":"2024-05-09T05:06:33.160660+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:06:33.160660+0000","last_undegraded":"2024-05-09T05:06:33.160660+0000","last_fullsized":"2024-05-09T05:06:33.160660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736183+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:06:06.736183+0000","last_peered":"2024-05-09T05:06:06.736183+0000","last_clean":"2024-05-09T05:06:06.736183+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:06:06.736183+0000","last_undegraded":"2024-05-09T05:06:06.736183+0000","last_fullsized":"2024-05-09T05:06:06.736183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:50.236344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.095074+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:06:15.095074+0000","last_peered":"2024-05-09T05:06:15.095074+0000","last_clean":"2024-05-09T05:06:15.095074+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:06:15.095074+0000","last_undegraded":"2024-05-09T05:06:15.095074+0000","last_fullsized":"2024-05-09T05:06:15.095074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722580+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:06:06.722580+0000","last_peered":"2024-05-09T05:06:06.722580+0000","last_clean":"2024-05-09T05:06:06.722580+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:06:06.722580+0000","last_undegraded":"2024-05-09T05:06:06.722580+0000","last_fullsized":"2024-05-09T05:06:06.722580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.160577+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:06:33.160577+0000","last_peered":"2024-05-09T05:06:33.160577+0000","last_clean":"2024-05-09T05:06:33.160577+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:06:33.160577+0000","last_undegraded":"2024-05-09T05:06:33.160577+0000","last_fullsized":"2024-05-09T05:06:33.160577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736386+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:06:06.736386+0000","last_peered":"2024-05-09T05:06:06.736386+0000","last_clean":"2024-05-09T05:06:06.736386+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:06:06.736386+0000","last_undegraded":"2024-05-09T05:06:06.736386+0000","last_fullsized":"2024-05-09T05:06:06.736386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135899+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:06:27.135899+0000","last_peered":"2024-05-09T05:06:27.135899+0000","last_clean":"2024-05-09T05:06:27.135899+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:06:27.135899+0000","last_undegraded":"2024-05-09T05:06:27.135899+0000","last_fullsized":"2024-05-09T05:06:27.135899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736096+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:06:06.736096+0000","last_peered":"2024-05-09T05:06:06.736096+0000","last_clean":"2024-05-09T05:06:06.736096+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:06:06.736096+0000","last_undegraded":"2024-05-09T05:06:06.736096+0000","last_fullsized":"2024-05-09T05:06:06.736096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:58:35.651549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.091272+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:06:15.091272+0000","last_peered":"2024-05-09T05:06:15.091272+0000","last_clean":"2024-05-09T05:06:15.091272+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:06:15.091272+0000","last_undegraded":"2024-05-09T05:06:15.091272+0000","last_fullsized":"2024-05-09T05:06:15.091272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736285+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:06:06.736285+0000","last_peered":"2024-05-09T05:06:06.736285+0000","last_clean":"2024-05-09T05:06:06.736285+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:06:06.736285+0000","last_undegraded":"2024-05-09T05:06:06.736285+0000","last_fullsized":"2024-05-09T05:06:06.736285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062736+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062736+0000","last_peered":"2024-05-09T05:06:06.062736+0000","last_clean":"2024-05-09T05:06:06.062736+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:06:06.062736+0000","last_undegraded":"2024-05-09T05:06:06.062736+0000","last_fullsized":"2024-05-09T05:06:06.062736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.160862+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:06:33.160862+0000","last_peered":"2024-05-09T05:06:33.160862+0000","last_clean":"2024-05-09T05:06:33.160862+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:06:33.160862+0000","last_undegraded":"2024-05-09T05:06:33.160862+0000","last_fullsized":"2024-05-09T05:06:33.160862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062769+0000","last_change":"2024-05-09T05:06:05.063337+0000","last_active":"2024-05-09T05:06:06.062769+0000","last_peered":"2024-05-09T05:06:06.062769+0000","last_clean":"2024-05-09T05:06:06.062769+0000","last_became_active":"2024-05-09T05:06:05.063016+0000","last_became_peered":"2024-05-09T05:06:05.063016+0000","last_unstale":"2024-05-09T05:06:06.062769+0000","last_undegraded":"2024-05-09T05:06:06.062769+0000","last_fullsized":"2024-05-09T05:06:06.062769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'11","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161376+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:06:33.161376+0000","last_peered":"2024-05-09T05:06:33.161376+0000","last_clean":"2024-05-09T05:06:33.161376+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:06:33.161376+0000","last_undegraded":"2024-05-09T05:06:33.161376+0000","last_fullsized":"2024-05-09T05:06:33.161376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736394+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:06:06.736394+0000","last_peered":"2024-05-09T05:06:06.736394+0000","last_clean":"2024-05-09T05:06:06.736394+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:06:06.736394+0000","last_undegraded":"2024-05-09T05:06:06.736394+0000","last_fullsized":"2024-05-09T05:06:06.736394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:42.457145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053935+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:06:10.053935+0000","last_peered":"2024-05-09T05:06:10.053935+0000","last_clean":"2024-05-09T05:06:10.053935+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:06:10.053935+0000","last_undegraded":"2024-05-09T05:06:10.053935+0000","last_fullsized":"2024-05-09T05:06:10.053935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736091+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:06:06.736091+0000","last_peered":"2024-05-09T05:06:06.736091+0000","last_clean":"2024-05-09T05:06:06.736091+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:06:06.736091+0000","last_undegraded":"2024-05-09T05:06:06.736091+0000","last_fullsized":"2024-05-09T05:06:06.736091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:32:29.331565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.160480+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:06:33.160480+0000","last_peered":"2024-05-09T05:06:33.160480+0000","last_clean":"2024-05-09T05:06:33.160480+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:06:33.160480+0000","last_undegraded":"2024-05-09T05:06:33.160480+0000","last_fullsized":"2024-05-09T05:06:33.160480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509451,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":665997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509452,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":258048,"data_stored":77461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542156,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":75931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":74859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":75965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542158,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":74991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542157,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":75061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542156,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36148,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335692,"statfs":{"total":96636764160,"available":96599748608,"internally_reserved":0,"allocated":831488,"data_stored":664771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:06:43.079 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:06:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.230+0000 7fc6d4b08640 1 Processor -- start 2024-05-09T05:06:43.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.230+0000 7fc6d4b08640 1 -- start start 2024-05-09T05:06:43.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.231+0000 7fc6d4b08640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6d00cd2b0 0x7fc6d0168190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.231+0000 7fc6d4b08640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d00cd0a0 con 0x7fc6d006dd70 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.231+0000 7fc6d4b08640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6d00cce00 con 0x7fc6d00cd7f0 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.231+0000 7fc6d4b08640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc6d00c56a0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.232+0000 7fc6ced9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6d00cd2b0 0x7fc6d0168190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.232+0000 7fc6ced9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6d00cd2b0 0x7fc6d0168190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53434/0 (socket says 172.21.15.1:53434) 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.232+0000 7fc6ce59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc6d006dd70 0x7fc6d00cbff0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59674/0 (socket says 172.21.15.1:59674) 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.232+0000 7fc6ced9d640 1 -- 172.21.15.1:0/134559298 learned_addr learned my addr 172.21.15.1:0/134559298 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.232+0000 7fc6ced9d640 1 -- 172.21.15.1:0/134559298 >> v1:172.21.15.196:6789/0 conn(0x7fc6d00cd7f0 legacy=0x7fc6d00c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.232+0000 7fc6ced9d640 1 -- 172.21.15.1:0/134559298 >> v1:172.21.15.1:6789/0 conn(0x7fc6d006dd70 legacy=0x7fc6d00cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.232+0000 7fc6ced9d640 1 -- 172.21.15.1:0/134559298 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d015d320 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.232+0000 7fc6cdd9b640 1 -- 172.21.15.1:0/134559298 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3066682411 0 0) 0x7fc6d00cd0a0 con 0x7fc6d006dd70 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.233+0000 7fc6ced9d640 1 --2- 172.21.15.1:0/134559298 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d0168190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009e50 tx=0x7fc6c802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d73634fff2cdbf3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.233+0000 7fc6cdd9b640 1 -- 172.21.15.1:0/134559298 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c8036070 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.233+0000 7fc6cdd9b640 1 -- 172.21.15.1:0/134559298 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6c8031600 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.233+0000 7fc6cdd9b640 1 -- 172.21.15.1:0/134559298 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c80319f0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.234+0000 7fc6d4b08640 1 -- 172.21.15.1:0/134559298 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 msgr2=0x7fc6d0168190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.234+0000 7fc6d4b08640 1 --2- 172.21.15.1:0/134559298 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d0168190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fc6c8009e50 tx=0x7fc6c802a7c0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.234+0000 7fc6d4b08640 1 -- 172.21.15.1:0/134559298 shutdown_connections 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.234+0000 7fc6d4b08640 1 --2- 172.21.15.1:0/134559298 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d0168190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.234+0000 7fc6d4b08640 1 -- 172.21.15.1:0/134559298 >> 172.21.15.1:0/134559298 conn(0x7fc6d00ce110 msgr2=0x7fc6d00ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.234+0000 7fc6d4b08640 1 -- 172.21.15.1:0/134559298 shutdown_connections 2024-05-09T05:06:43.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.234+0000 7fc6d4b08640 1 -- 172.21.15.1:0/134559298 wait complete. 2024-05-09T05:06:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.236+0000 7fc6d4b08640 1 Processor -- start 2024-05-09T05:06:43.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.236+0000 7fc6d4b08640 1 -- start start 2024-05-09T05:06:43.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6d4b08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d01f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6d4b08640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6d00cd7f0 0x7fc6d0143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6d4b08640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6d0147340 0x7fc6d0157010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6d4b08640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d0074ba0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6d4b08640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc6d0074a20 con 0x7fc6d0147340 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6d4b08640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc6d00c7750 con 0x7fc6d00cd7f0 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d01f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d01f5fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53440/0 (socket says 172.21.15.1:53440) 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 -- 172.21.15.1:0/447571774 learned_addr learned my addr 172.21.15.1:0/447571774 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ced9d640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6d0147340 0x7fc6d0157010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6c7fff640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6d00cd7f0 0x7fc6d0143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 -- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6d00cd7f0 msgr2=0x7fc6d0143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6d00cd7f0 0x7fc6d0143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 -- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6d0147340 msgr2=0x7fc6d0157010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6d0147340 0x7fc6d0157010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 -- 172.21.15.1:0/447571774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6d01f64f0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.237+0000 7fc6ce59c640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d01f5fb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000c3d0 tx=0x7fc6c000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.238+0000 7fc6c6ffd640 1 -- 172.21.15.1:0/447571774 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c00074e0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.238+0000 7fc6c6ffd640 1 -- 172.21.15.1:0/447571774 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6c00047e0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.238+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6d00c99b0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.238+0000 7fc6c6ffd640 1 -- 172.21.15.1:0/447571774 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6c0004bd0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.238+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6d015ae20 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.239+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc6d0074770 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.240+0000 7fc6c6ffd640 1 -- 172.21.15.1:0/447571774 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc6c0011070 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.240+0000 7fc6c6ffd640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6b00762d0 0x7fc6b00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.240+0000 7fc6c6ffd640 1 -- 172.21.15.1:0/447571774 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fc6c00970f0 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.240+0000 7fc6c7fff640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6b00762d0 0x7fc6b00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.241+0000 7fc6c7fff640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6b00762d0 0x7fc6b00787e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00cc520 tx=0x7fc6b8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.245+0000 7fc6c6ffd640 1 -- 172.21.15.1:0/447571774 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc6c0060370 con 0x7fc6d00cd2b0 2024-05-09T05:06:43.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.432+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fc6d0156250 con 0x7fc6b00762d0 2024-05-09T05:06:43.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.439+0000 7fc6c6ffd640 1 -- 172.21.15.1:0/447571774 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252011 (secure 0 0 0) 0x7fc6d0156250 con 0x7fc6b00762d0 2024-05-09T05:06:43.442 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:43.443 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:06:43.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6b00762d0 msgr2=0x7fc6b00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6b00762d0 0x7fc6b00787e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fc6d00cc520 tx=0x7fc6b8009040 comp rx=0 tx=0).stop 2024-05-09T05:06:43.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 msgr2=0x7fc6d01f5fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d01f5fb0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fc6c000c3d0 tx=0x7fc6c000c890 comp rx=0 tx=0).stop 2024-05-09T05:06:43.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 shutdown_connections 2024-05-09T05:06:43.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6b00762d0 0x7fc6b00787e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6d0147340 0x7fc6d0157010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6d00cd7f0 0x7fc6d0143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 --2- 172.21.15.1:0/447571774 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6d00cd2b0 0x7fc6d01f5fb0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 >> 172.21.15.1:0/447571774 conn(0x7fc6d00ce110 msgr2=0x7fc6d00ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 shutdown_connections 2024-05-09T05:06:43.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.442+0000 7fc6d4b08640 1 -- 172.21.15.1:0/447571774 wait complete. 2024-05-09T05:06:43.453 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":39,"stamp":"2024-05-09T05:06:41.551980+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":598402,"num_objects":99,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":99,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":305,"num_write_kb":1690,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":250,"ondisk_log_size":250,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":239976,"kb_used_data":3048,"kb_used_omap":12,"kb_used_meta":236851,"kb_avail":754734744,"statfs":{"total":773094113280,"available":772848377856,"internally_reserved":0,"allocated":3121152,"data_stored":1785036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":242535985},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2140,"num_objects":32,"num_object_clones":0,"num_object_copies":64,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":32,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":68,"num_write_kb":22,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":22,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002734"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737187+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:06:06.737187+0000","last_peered":"2024-05-09T05:06:06.737187+0000","last_clean":"2024-05-09T05:06:06.737187+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:06:06.737187+0000","last_undegraded":"2024-05-09T05:06:06.737187+0000","last_fullsized":"2024-05-09T05:06:06.737187+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058883+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:06:06.058883+0000","last_peered":"2024-05-09T05:06:06.058883+0000","last_clean":"2024-05-09T05:06:06.058883+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:06:06.058883+0000","last_undegraded":"2024-05-09T05:06:06.058883+0000","last_fullsized":"2024-05-09T05:06:06.058883+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736013+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:06:06.736013+0000","last_peered":"2024-05-09T05:06:06.736013+0000","last_clean":"2024-05-09T05:06:06.736013+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:06:06.736013+0000","last_undegraded":"2024-05-09T05:06:06.736013+0000","last_fullsized":"2024-05-09T05:06:06.736013+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:48.770365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062625+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:06:06.062625+0000","last_peered":"2024-05-09T05:06:06.062625+0000","last_clean":"2024-05-09T05:06:06.062625+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:06:06.062625+0000","last_undegraded":"2024-05-09T05:06:06.062625+0000","last_fullsized":"2024-05-09T05:06:06.062625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737255+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:06:06.737255+0000","last_peered":"2024-05-09T05:06:06.737255+0000","last_clean":"2024-05-09T05:06:06.737255+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:06:06.737255+0000","last_undegraded":"2024-05-09T05:06:06.737255+0000","last_fullsized":"2024-05-09T05:06:06.737255+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062081+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:06:06.062081+0000","last_peered":"2024-05-09T05:06:06.062081+0000","last_clean":"2024-05-09T05:06:06.062081+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:06:06.062081+0000","last_undegraded":"2024-05-09T05:06:06.062081+0000","last_fullsized":"2024-05-09T05:06:06.062081+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736570+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:06:06.736570+0000","last_peered":"2024-05-09T05:06:06.736570+0000","last_clean":"2024-05-09T05:06:06.736570+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:06:06.736570+0000","last_undegraded":"2024-05-09T05:06:06.736570+0000","last_fullsized":"2024-05-09T05:06:06.736570+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735634+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:06:06.735634+0000","last_peered":"2024-05-09T05:06:06.735634+0000","last_clean":"2024-05-09T05:06:06.735634+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:06:06.735634+0000","last_undegraded":"2024-05-09T05:06:06.735634+0000","last_fullsized":"2024-05-09T05:06:06.735634+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737253+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:06:06.737253+0000","last_peered":"2024-05-09T05:06:06.737253+0000","last_clean":"2024-05-09T05:06:06.737253+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:06:06.737253+0000","last_undegraded":"2024-05-09T05:06:06.737253+0000","last_fullsized":"2024-05-09T05:06:06.737253+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736506+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:06:06.736506+0000","last_peered":"2024-05-09T05:06:06.736506+0000","last_clean":"2024-05-09T05:06:06.736506+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:06:06.736506+0000","last_undegraded":"2024-05-09T05:06:06.736506+0000","last_fullsized":"2024-05-09T05:06:06.736506+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736352+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:06:06.736352+0000","last_peered":"2024-05-09T05:06:06.736352+0000","last_clean":"2024-05-09T05:06:06.736352+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:06:06.736352+0000","last_undegraded":"2024-05-09T05:06:06.736352+0000","last_fullsized":"2024-05-09T05:06:06.736352+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722788+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:06:06.722788+0000","last_peered":"2024-05-09T05:06:06.722788+0000","last_clean":"2024-05-09T05:06:06.722788+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:06:06.722788+0000","last_undegraded":"2024-05-09T05:06:06.722788+0000","last_fullsized":"2024-05-09T05:06:06.722788+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736075+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:06:06.736075+0000","last_peered":"2024-05-09T05:06:06.736075+0000","last_clean":"2024-05-09T05:06:06.736075+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:06:06.736075+0000","last_undegraded":"2024-05-09T05:06:06.736075+0000","last_fullsized":"2024-05-09T05:06:06.736075+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736071+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:06:06.736071+0000","last_peered":"2024-05-09T05:06:06.736071+0000","last_clean":"2024-05-09T05:06:06.736071+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:06:06.736071+0000","last_undegraded":"2024-05-09T05:06:06.736071+0000","last_fullsized":"2024-05-09T05:06:06.736071+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:37.258923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058421+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:06:06.058421+0000","last_peered":"2024-05-09T05:06:06.058421+0000","last_clean":"2024-05-09T05:06:06.058421+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:06:06.058421+0000","last_undegraded":"2024-05-09T05:06:06.058421+0000","last_fullsized":"2024-05-09T05:06:06.058421+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062588+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:06:06.062588+0000","last_peered":"2024-05-09T05:06:06.062588+0000","last_clean":"2024-05-09T05:06:06.062588+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:06:06.062588+0000","last_undegraded":"2024-05-09T05:06:06.062588+0000","last_fullsized":"2024-05-09T05:06:06.062588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062864+0000","last_change":"2024-05-09T05:06:05.071898+0000","last_active":"2024-05-09T05:06:06.062864+0000","last_peered":"2024-05-09T05:06:06.062864+0000","last_clean":"2024-05-09T05:06:06.062864+0000","last_became_active":"2024-05-09T05:06:05.071618+0000","last_became_peered":"2024-05-09T05:06:05.071618+0000","last_unstale":"2024-05-09T05:06:06.062864+0000","last_undegraded":"2024-05-09T05:06:06.062864+0000","last_fullsized":"2024-05-09T05:06:06.062864+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.060053+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:06:06.060053+0000","last_peered":"2024-05-09T05:06:06.060053+0000","last_clean":"2024-05-09T05:06:06.060053+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:06:06.060053+0000","last_undegraded":"2024-05-09T05:06:06.060053+0000","last_fullsized":"2024-05-09T05:06:06.060053+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737104+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:06:06.737104+0000","last_peered":"2024-05-09T05:06:06.737104+0000","last_clean":"2024-05-09T05:06:06.737104+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:06:06.737104+0000","last_undegraded":"2024-05-09T05:06:06.737104+0000","last_fullsized":"2024-05-09T05:06:06.737104+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736090+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:06:06.736090+0000","last_peered":"2024-05-09T05:06:06.736090+0000","last_clean":"2024-05-09T05:06:06.736090+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:06:06.736090+0000","last_undegraded":"2024-05-09T05:06:06.736090+0000","last_fullsized":"2024-05-09T05:06:06.736090+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736595+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:06:06.736595+0000","last_peered":"2024-05-09T05:06:06.736595+0000","last_clean":"2024-05-09T05:06:06.736595+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:06:06.736595+0000","last_undegraded":"2024-05-09T05:06:06.736595+0000","last_fullsized":"2024-05-09T05:06:06.736595+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:02.665344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062276+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:06:06.062276+0000","last_peered":"2024-05-09T05:06:06.062276+0000","last_clean":"2024-05-09T05:06:06.062276+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:06:06.062276+0000","last_undegraded":"2024-05-09T05:06:06.062276+0000","last_fullsized":"2024-05-09T05:06:06.062276+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735814+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:06:06.735814+0000","last_peered":"2024-05-09T05:06:06.735814+0000","last_clean":"2024-05-09T05:06:06.735814+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:06:06.735814+0000","last_undegraded":"2024-05-09T05:06:06.735814+0000","last_fullsized":"2024-05-09T05:06:06.735814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062254+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062254+0000","last_peered":"2024-05-09T05:06:06.062254+0000","last_clean":"2024-05-09T05:06:06.062254+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:06:06.062254+0000","last_undegraded":"2024-05-09T05:06:06.062254+0000","last_fullsized":"2024-05-09T05:06:06.062254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736447+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:06:06.736447+0000","last_peered":"2024-05-09T05:06:06.736447+0000","last_clean":"2024-05-09T05:06:06.736447+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:06:06.736447+0000","last_undegraded":"2024-05-09T05:06:06.736447+0000","last_fullsized":"2024-05-09T05:06:06.736447+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:25:28.166958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736699+0000","last_change":"2024-05-09T05:06:05.067878+0000","last_active":"2024-05-09T05:06:06.736699+0000","last_peered":"2024-05-09T05:06:06.736699+0000","last_clean":"2024-05-09T05:06:06.736699+0000","last_became_active":"2024-05-09T05:06:05.067426+0000","last_became_peered":"2024-05-09T05:06:05.067426+0000","last_unstale":"2024-05-09T05:06:06.736699+0000","last_undegraded":"2024-05-09T05:06:06.736699+0000","last_fullsized":"2024-05-09T05:06:06.736699+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058528+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:06:06.058528+0000","last_peered":"2024-05-09T05:06:06.058528+0000","last_clean":"2024-05-09T05:06:06.058528+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:06:06.058528+0000","last_undegraded":"2024-05-09T05:06:06.058528+0000","last_fullsized":"2024-05-09T05:06:06.058528+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736532+0000","last_change":"2024-05-09T05:06:05.063359+0000","last_active":"2024-05-09T05:06:06.736532+0000","last_peered":"2024-05-09T05:06:06.736532+0000","last_clean":"2024-05-09T05:06:06.736532+0000","last_became_active":"2024-05-09T05:06:05.063026+0000","last_became_peered":"2024-05-09T05:06:05.063026+0000","last_unstale":"2024-05-09T05:06:06.736532+0000","last_undegraded":"2024-05-09T05:06:06.736532+0000","last_fullsized":"2024-05-09T05:06:06.736532+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736550+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:06:06.736550+0000","last_peered":"2024-05-09T05:06:06.736550+0000","last_clean":"2024-05-09T05:06:06.736550+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:06:06.736550+0000","last_undegraded":"2024-05-09T05:06:06.736550+0000","last_fullsized":"2024-05-09T05:06:06.736550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062682+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:06:06.062682+0000","last_peered":"2024-05-09T05:06:06.062682+0000","last_clean":"2024-05-09T05:06:06.062682+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:06:06.062682+0000","last_undegraded":"2024-05-09T05:06:06.062682+0000","last_fullsized":"2024-05-09T05:06:06.062682+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.059766+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:06:06.059766+0000","last_peered":"2024-05-09T05:06:06.059766+0000","last_clean":"2024-05-09T05:06:06.059766+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:06:06.059766+0000","last_undegraded":"2024-05-09T05:06:06.059766+0000","last_fullsized":"2024-05-09T05:06:06.059766+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736152+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:06:06.736152+0000","last_peered":"2024-05-09T05:06:06.736152+0000","last_clean":"2024-05-09T05:06:06.736152+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:06:06.736152+0000","last_undegraded":"2024-05-09T05:06:06.736152+0000","last_fullsized":"2024-05-09T05:06:06.736152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722480+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:06:06.722480+0000","last_peered":"2024-05-09T05:06:06.722480+0000","last_clean":"2024-05-09T05:06:06.722480+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:06:06.722480+0000","last_undegraded":"2024-05-09T05:06:06.722480+0000","last_fullsized":"2024-05-09T05:06:06.722480+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.099849+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:06:15.099849+0000","last_peered":"2024-05-09T05:06:15.099849+0000","last_clean":"2024-05-09T05:06:15.099849+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:06:15.099849+0000","last_undegraded":"2024-05-09T05:06:15.099849+0000","last_fullsized":"2024-05-09T05:06:15.099849+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736786+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:06:06.736786+0000","last_peered":"2024-05-09T05:06:06.736786+0000","last_clean":"2024-05-09T05:06:06.736786+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:06:06.736786+0000","last_undegraded":"2024-05-09T05:06:06.736786+0000","last_fullsized":"2024-05-09T05:06:06.736786+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135868+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:06:27.135868+0000","last_peered":"2024-05-09T05:06:27.135868+0000","last_clean":"2024-05-09T05:06:27.135868+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:06:27.135868+0000","last_undegraded":"2024-05-09T05:06:27.135868+0000","last_fullsized":"2024-05-09T05:06:27.135868+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:21.118094+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:06:21.118094+0000","last_peered":"2024-05-09T05:06:21.118094+0000","last_clean":"2024-05-09T05:06:21.118094+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:06:21.118094+0000","last_undegraded":"2024-05-09T05:06:21.118094+0000","last_fullsized":"2024-05-09T05:06:21.118094+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735754+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:06:06.735754+0000","last_peered":"2024-05-09T05:06:06.735754+0000","last_clean":"2024-05-09T05:06:06.735754+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:06:06.735754+0000","last_undegraded":"2024-05-09T05:06:06.735754+0000","last_fullsized":"2024-05-09T05:06:06.735754+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.137356+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:06:27.137356+0000","last_peered":"2024-05-09T05:06:27.137356+0000","last_clean":"2024-05-09T05:06:27.137356+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:06:27.137356+0000","last_undegraded":"2024-05-09T05:06:27.137356+0000","last_fullsized":"2024-05-09T05:06:27.137356+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058418+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:06:06.058418+0000","last_peered":"2024-05-09T05:06:06.058418+0000","last_clean":"2024-05-09T05:06:06.058418+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:06:06.058418+0000","last_undegraded":"2024-05-09T05:06:06.058418+0000","last_fullsized":"2024-05-09T05:06:06.058418+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.094128+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:06:15.094128+0000","last_peered":"2024-05-09T05:06:15.094128+0000","last_clean":"2024-05-09T05:06:15.094128+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:06:15.094128+0000","last_undegraded":"2024-05-09T05:06:15.094128+0000","last_fullsized":"2024-05-09T05:06:15.094128+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062383+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:06:06.062383+0000","last_peered":"2024-05-09T05:06:06.062383+0000","last_clean":"2024-05-09T05:06:06.062383+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:06:06.062383+0000","last_undegraded":"2024-05-09T05:06:06.062383+0000","last_fullsized":"2024-05-09T05:06:06.062383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161228+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:06:33.161228+0000","last_peered":"2024-05-09T05:06:33.161228+0000","last_clean":"2024-05-09T05:06:33.161228+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:06:33.161228+0000","last_undegraded":"2024-05-09T05:06:33.161228+0000","last_fullsized":"2024-05-09T05:06:33.161228+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736605+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:06:06.736605+0000","last_peered":"2024-05-09T05:06:06.736605+0000","last_clean":"2024-05-09T05:06:06.736605+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:06:06.736605+0000","last_undegraded":"2024-05-09T05:06:06.736605+0000","last_fullsized":"2024-05-09T05:06:06.736605+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:26.607004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.099386+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:06:15.099386+0000","last_peered":"2024-05-09T05:06:15.099386+0000","last_clean":"2024-05-09T05:06:15.099386+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:06:15.099386+0000","last_undegraded":"2024-05-09T05:06:15.099386+0000","last_fullsized":"2024-05-09T05:06:15.099386+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722806+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:06:06.722806+0000","last_peered":"2024-05-09T05:06:06.722806+0000","last_clean":"2024-05-09T05:06:06.722806+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:06:06.722806+0000","last_undegraded":"2024-05-09T05:06:06.722806+0000","last_fullsized":"2024-05-09T05:06:06.722806+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"21'10","reported_seq":24,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.159438+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:06:33.159438+0000","last_peered":"2024-05-09T05:06:33.159438+0000","last_clean":"2024-05-09T05:06:33.159438+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:06:33.159438+0000","last_undegraded":"2024-05-09T05:06:33.159438+0000","last_fullsized":"2024-05-09T05:06:33.159438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736736+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:06:06.736736+0000","last_peered":"2024-05-09T05:06:06.736736+0000","last_clean":"2024-05-09T05:06:06.736736+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:06:06.736736+0000","last_undegraded":"2024-05-09T05:06:06.736736+0000","last_fullsized":"2024-05-09T05:06:06.736736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"21'9","reported_seq":23,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161901+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:06:33.161901+0000","last_peered":"2024-05-09T05:06:33.161901+0000","last_clean":"2024-05-09T05:06:33.161901+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:06:33.161901+0000","last_undegraded":"2024-05-09T05:06:33.161901+0000","last_fullsized":"2024-05-09T05:06:33.161901+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736396+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:06:06.736396+0000","last_peered":"2024-05-09T05:06:06.736396+0000","last_clean":"2024-05-09T05:06:06.736396+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:06:06.736396+0000","last_undegraded":"2024-05-09T05:06:06.736396+0000","last_fullsized":"2024-05-09T05:06:06.736396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:41.283927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.137358+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:06:27.137358+0000","last_peered":"2024-05-09T05:06:27.137358+0000","last_clean":"2024-05-09T05:06:27.137358+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:06:27.137358+0000","last_undegraded":"2024-05-09T05:06:27.137358+0000","last_fullsized":"2024-05-09T05:06:27.137358+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737118+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:06:06.737118+0000","last_peered":"2024-05-09T05:06:06.737118+0000","last_clean":"2024-05-09T05:06:06.737118+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:06:06.737118+0000","last_undegraded":"2024-05-09T05:06:06.737118+0000","last_fullsized":"2024-05-09T05:06:06.737118+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135525+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:06:27.135525+0000","last_peered":"2024-05-09T05:06:27.135525+0000","last_clean":"2024-05-09T05:06:27.135525+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:06:27.135525+0000","last_undegraded":"2024-05-09T05:06:27.135525+0000","last_fullsized":"2024-05-09T05:06:27.135525+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736621+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:06:06.736621+0000","last_peered":"2024-05-09T05:06:06.736621+0000","last_clean":"2024-05-09T05:06:06.736621+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:06:06.736621+0000","last_undegraded":"2024-05-09T05:06:06.736621+0000","last_fullsized":"2024-05-09T05:06:06.736621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"21'17","reported_seq":31,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.134077+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:06:27.134077+0000","last_peered":"2024-05-09T05:06:27.134077+0000","last_clean":"2024-05-09T05:06:27.134077+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:06:27.134077+0000","last_undegraded":"2024-05-09T05:06:27.134077+0000","last_fullsized":"2024-05-09T05:06:27.134077+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736921+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:06:06.736921+0000","last_peered":"2024-05-09T05:06:06.736921+0000","last_clean":"2024-05-09T05:06:06.736921+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:06:06.736921+0000","last_undegraded":"2024-05-09T05:06:06.736921+0000","last_fullsized":"2024-05-09T05:06:06.736921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.052196+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:06:10.052196+0000","last_peered":"2024-05-09T05:06:10.052196+0000","last_clean":"2024-05-09T05:06:10.052196+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:06:10.052196+0000","last_undegraded":"2024-05-09T05:06:10.052196+0000","last_fullsized":"2024-05-09T05:06:10.052196+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737109+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:06:06.737109+0000","last_peered":"2024-05-09T05:06:06.737109+0000","last_clean":"2024-05-09T05:06:06.737109+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:06:06.737109+0000","last_undegraded":"2024-05-09T05:06:06.737109+0000","last_fullsized":"2024-05-09T05:06:06.737109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161496+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:06:33.161496+0000","last_peered":"2024-05-09T05:06:33.161496+0000","last_clean":"2024-05-09T05:06:33.161496+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:06:33.161496+0000","last_undegraded":"2024-05-09T05:06:33.161496+0000","last_fullsized":"2024-05-09T05:06:33.161496+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.737391+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:06:06.737391+0000","last_peered":"2024-05-09T05:06:06.737391+0000","last_clean":"2024-05-09T05:06:06.737391+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:06:06.737391+0000","last_undegraded":"2024-05-09T05:06:06.737391+0000","last_fullsized":"2024-05-09T05:06:06.737391+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135984+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:06:27.135984+0000","last_peered":"2024-05-09T05:06:27.135984+0000","last_clean":"2024-05-09T05:06:27.135984+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:06:27.135984+0000","last_undegraded":"2024-05-09T05:06:27.135984+0000","last_fullsized":"2024-05-09T05:06:27.135984+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:06:06.062865+0000","last_peered":"2024-05-09T05:06:06.062865+0000","last_clean":"2024-05-09T05:06:06.062865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:06:06.062865+0000","last_undegraded":"2024-05-09T05:06:06.062865+0000","last_fullsized":"2024-05-09T05:06:06.062865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.140130+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:06:27.140130+0000","last_peered":"2024-05-09T05:06:27.140130+0000","last_clean":"2024-05-09T05:06:27.140130+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:06:27.140130+0000","last_undegraded":"2024-05-09T05:06:27.140130+0000","last_fullsized":"2024-05-09T05:06:27.140130+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736885+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:06:06.736885+0000","last_peered":"2024-05-09T05:06:06.736885+0000","last_clean":"2024-05-09T05:06:06.736885+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:06:06.736885+0000","last_undegraded":"2024-05-09T05:06:06.736885+0000","last_fullsized":"2024-05-09T05:06:06.736885+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":110,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736326+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:06:06.736326+0000","last_peered":"2024-05-09T05:06:06.736326+0000","last_clean":"2024-05-09T05:06:06.736326+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:06:06.736326+0000","last_undegraded":"2024-05-09T05:06:06.736326+0000","last_fullsized":"2024-05-09T05:06:06.736326+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"21'6","reported_seq":20,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.139983+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:06:27.139983+0000","last_peered":"2024-05-09T05:06:27.139983+0000","last_clean":"2024-05-09T05:06:27.139983+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:06:27.139983+0000","last_undegraded":"2024-05-09T05:06:27.139983+0000","last_fullsized":"2024-05-09T05:06:27.139983+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.735899+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:06:06.735899+0000","last_peered":"2024-05-09T05:06:06.735899+0000","last_clean":"2024-05-09T05:06:06.735899+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:06:06.735899+0000","last_undegraded":"2024-05-09T05:06:06.735899+0000","last_fullsized":"2024-05-09T05:06:06.735899+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:21.117843+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:06:21.117843+0000","last_peered":"2024-05-09T05:06:21.117843+0000","last_clean":"2024-05-09T05:06:21.117843+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:06:21.117843+0000","last_undegraded":"2024-05-09T05:06:21.117843+0000","last_fullsized":"2024-05-09T05:06:21.117843+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736254+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:06:06.736254+0000","last_peered":"2024-05-09T05:06:06.736254+0000","last_clean":"2024-05-09T05:06:06.736254+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:06:06.736254+0000","last_undegraded":"2024-05-09T05:06:06.736254+0000","last_fullsized":"2024-05-09T05:06:06.736254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:23.362012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:21.117677+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:06:21.117677+0000","last_peered":"2024-05-09T05:06:21.117677+0000","last_clean":"2024-05-09T05:06:21.117677+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:06:21.117677+0000","last_undegraded":"2024-05-09T05:06:21.117677+0000","last_fullsized":"2024-05-09T05:06:21.117677+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058422+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:06:06.058422+0000","last_peered":"2024-05-09T05:06:06.058422+0000","last_clean":"2024-05-09T05:06:06.058422+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:06:06.058422+0000","last_undegraded":"2024-05-09T05:06:06.058422+0000","last_fullsized":"2024-05-09T05:06:06.058422+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161352+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:06:33.161352+0000","last_peered":"2024-05-09T05:06:33.161352+0000","last_clean":"2024-05-09T05:06:33.161352+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:06:33.161352+0000","last_undegraded":"2024-05-09T05:06:33.161352+0000","last_fullsized":"2024-05-09T05:06:33.161352+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722592+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:06:06.722592+0000","last_peered":"2024-05-09T05:06:06.722592+0000","last_clean":"2024-05-09T05:06:06.722592+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:06:06.722592+0000","last_undegraded":"2024-05-09T05:06:06.722592+0000","last_fullsized":"2024-05-09T05:06:06.722592+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"21'1","reported_seq":15,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.555529+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:06:06.555529+0000","last_peered":"2024-05-09T05:06:06.555529+0000","last_clean":"2024-05-09T05:06:06.555529+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:06:06.555529+0000","last_undegraded":"2024-05-09T05:06:06.555529+0000","last_fullsized":"2024-05-09T05:06:06.555529+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062852+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:06:06.062852+0000","last_peered":"2024-05-09T05:06:06.062852+0000","last_clean":"2024-05-09T05:06:06.062852+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:06:06.062852+0000","last_undegraded":"2024-05-09T05:06:06.062852+0000","last_fullsized":"2024-05-09T05:06:06.062852+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135199+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:06:27.135199+0000","last_peered":"2024-05-09T05:06:27.135199+0000","last_clean":"2024-05-09T05:06:27.135199+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:06:27.135199+0000","last_undegraded":"2024-05-09T05:06:27.135199+0000","last_fullsized":"2024-05-09T05:06:27.135199+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.058622+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:06:06.058622+0000","last_peered":"2024-05-09T05:06:06.058622+0000","last_clean":"2024-05-09T05:06:06.058622+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:06:06.058622+0000","last_undegraded":"2024-05-09T05:06:06.058622+0000","last_fullsized":"2024-05-09T05:06:06.058622+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.140453+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:06:27.140453+0000","last_peered":"2024-05-09T05:06:27.140453+0000","last_clean":"2024-05-09T05:06:27.140453+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:06:27.140453+0000","last_undegraded":"2024-05-09T05:06:27.140453+0000","last_fullsized":"2024-05-09T05:06:27.140453+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736837+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:06:06.736837+0000","last_peered":"2024-05-09T05:06:06.736837+0000","last_clean":"2024-05-09T05:06:06.736837+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:06:06.736837+0000","last_undegraded":"2024-05-09T05:06:06.736837+0000","last_fullsized":"2024-05-09T05:06:06.736837+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:33.840344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.160660+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:06:33.160660+0000","last_peered":"2024-05-09T05:06:33.160660+0000","last_clean":"2024-05-09T05:06:33.160660+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:06:33.160660+0000","last_undegraded":"2024-05-09T05:06:33.160660+0000","last_fullsized":"2024-05-09T05:06:33.160660+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736183+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:06:06.736183+0000","last_peered":"2024-05-09T05:06:06.736183+0000","last_clean":"2024-05-09T05:06:06.736183+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:06:06.736183+0000","last_undegraded":"2024-05-09T05:06:06.736183+0000","last_fullsized":"2024-05-09T05:06:06.736183+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:50.236344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.095074+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:06:15.095074+0000","last_peered":"2024-05-09T05:06:15.095074+0000","last_clean":"2024-05-09T05:06:15.095074+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:06:15.095074+0000","last_undegraded":"2024-05-09T05:06:15.095074+0000","last_fullsized":"2024-05-09T05:06:15.095074+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.722580+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:06:06.722580+0000","last_peered":"2024-05-09T05:06:06.722580+0000","last_clean":"2024-05-09T05:06:06.722580+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:06:06.722580+0000","last_undegraded":"2024-05-09T05:06:06.722580+0000","last_fullsized":"2024-05-09T05:06:06.722580+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"21'7","reported_seq":21,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.160577+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:06:33.160577+0000","last_peered":"2024-05-09T05:06:33.160577+0000","last_clean":"2024-05-09T05:06:33.160577+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:06:33.160577+0000","last_undegraded":"2024-05-09T05:06:33.160577+0000","last_fullsized":"2024-05-09T05:06:33.160577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736386+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:06:06.736386+0000","last_peered":"2024-05-09T05:06:06.736386+0000","last_clean":"2024-05-09T05:06:06.736386+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:06:06.736386+0000","last_undegraded":"2024-05-09T05:06:06.736386+0000","last_fullsized":"2024-05-09T05:06:06.736386+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"21'3","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:27.135899+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:06:27.135899+0000","last_peered":"2024-05-09T05:06:27.135899+0000","last_clean":"2024-05-09T05:06:27.135899+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:06:27.135899+0000","last_undegraded":"2024-05-09T05:06:27.135899+0000","last_fullsized":"2024-05-09T05:06:27.135899+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736096+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:06:06.736096+0000","last_peered":"2024-05-09T05:06:06.736096+0000","last_clean":"2024-05-09T05:06:06.736096+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:06:06.736096+0000","last_undegraded":"2024-05-09T05:06:06.736096+0000","last_fullsized":"2024-05-09T05:06:06.736096+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:58:35.651549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"21'4","reported_seq":18,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:15.091272+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:06:15.091272+0000","last_peered":"2024-05-09T05:06:15.091272+0000","last_clean":"2024-05-09T05:06:15.091272+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:06:15.091272+0000","last_undegraded":"2024-05-09T05:06:15.091272+0000","last_fullsized":"2024-05-09T05:06:15.091272+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736285+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:06:06.736285+0000","last_peered":"2024-05-09T05:06:06.736285+0000","last_clean":"2024-05-09T05:06:06.736285+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:06:06.736285+0000","last_undegraded":"2024-05-09T05:06:06.736285+0000","last_fullsized":"2024-05-09T05:06:06.736285+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062736+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:06:06.062736+0000","last_peered":"2024-05-09T05:06:06.062736+0000","last_clean":"2024-05-09T05:06:06.062736+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:06:06.062736+0000","last_undegraded":"2024-05-09T05:06:06.062736+0000","last_fullsized":"2024-05-09T05:06:06.062736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"21'5","reported_seq":19,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.160862+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:06:33.160862+0000","last_peered":"2024-05-09T05:06:33.160862+0000","last_clean":"2024-05-09T05:06:33.160862+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:06:33.160862+0000","last_undegraded":"2024-05-09T05:06:33.160862+0000","last_fullsized":"2024-05-09T05:06:33.160862+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.062769+0000","last_change":"2024-05-09T05:06:05.063337+0000","last_active":"2024-05-09T05:06:06.062769+0000","last_peered":"2024-05-09T05:06:06.062769+0000","last_clean":"2024-05-09T05:06:06.062769+0000","last_became_active":"2024-05-09T05:06:05.063016+0000","last_became_peered":"2024-05-09T05:06:05.063016+0000","last_unstale":"2024-05-09T05:06:06.062769+0000","last_undegraded":"2024-05-09T05:06:06.062769+0000","last_fullsized":"2024-05-09T05:06:06.062769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1d","version":"21'11","reported_seq":25,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.161376+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:06:33.161376+0000","last_peered":"2024-05-09T05:06:33.161376+0000","last_clean":"2024-05-09T05:06:33.161376+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:06:33.161376+0000","last_undegraded":"2024-05-09T05:06:33.161376+0000","last_fullsized":"2024-05-09T05:06:33.161376+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736394+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:06:06.736394+0000","last_peered":"2024-05-09T05:06:06.736394+0000","last_clean":"2024-05-09T05:06:06.736394+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:06:06.736394+0000","last_undegraded":"2024-05-09T05:06:06.736394+0000","last_fullsized":"2024-05-09T05:06:06.736394+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:42.457145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"21'2","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:10.053935+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:06:10.053935+0000","last_peered":"2024-05-09T05:06:10.053935+0000","last_clean":"2024-05-09T05:06:10.053935+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:06:10.053935+0000","last_undegraded":"2024-05-09T05:06:10.053935+0000","last_fullsized":"2024-05-09T05:06:10.053935+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":12,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:06.736091+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:06:06.736091+0000","last_peered":"2024-05-09T05:06:06.736091+0000","last_clean":"2024-05-09T05:06:06.736091+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:06:06.736091+0000","last_undegraded":"2024-05-09T05:06:06.736091+0000","last_fullsized":"2024-05-09T05:06:06.736091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:32:29.331565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"21'13","reported_seq":27,"reported_epoch":21,"state":"active+clean","last_fresh":"2024-05-09T05:06:33.160480+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:06:33.160480+0000","last_peered":"2024-05-09T05:06:33.160480+0000","last_clean":"2024-05-09T05:06:33.160480+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:06:33.160480+0000","last_undegraded":"2024-05-09T05:06:33.160480+0000","last_fullsized":"2024-05-09T05:06:33.160480+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":8034,"num_objects":97,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":97,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":204,"num_write_kb":62,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":65,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":262144,"data_stored":16068,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":178,"ondisk_log_size":178,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509451,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27836,"kb_used_data":820,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344004,"statfs":{"total":96636764160,"available":96608260096,"internally_reserved":0,"allocated":839680,"data_stored":665997,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509452,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27268,"kb_used_data":252,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344572,"statfs":{"total":96636764160,"available":96608841728,"internally_reserved":0,"allocated":258048,"data_stored":77461,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542156,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27248,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344592,"statfs":{"total":96636764160,"available":96608862208,"internally_reserved":0,"allocated":237568,"data_stored":75931,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542157,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27244,"kb_used_data":228,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344596,"statfs":{"total":96636764160,"available":96608866304,"internally_reserved":0,"allocated":233472,"data_stored":74859,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542157,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":75965,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":31914440},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542158,"num_pgs":13,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31412,"kb_used_data":236,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340428,"statfs":{"total":96636764160,"available":96604598272,"internally_reserved":0,"allocated":241664,"data_stored":74991,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542157,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":31408,"kb_used_data":232,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":94340432,"statfs":{"total":96636764160,"available":96604602368,"internally_reserved":0,"allocated":237568,"data_stored":75061,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542156,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":36148,"kb_used_data":812,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":94335692,"statfs":{"total":96636764160,"available":96599748608,"internally_reserved":0,"allocated":831488,"data_stored":664771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":874,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":1462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2436,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":1330,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":3932,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2402,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":2100,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:06:43.458 INFO:tasks.mon_thrash.ceph_manager:clean! 2024-05-09T05:06:43.459 DEBUG:tasks.mon_thrash:Yielding 2024-05-09T05:06:43.459 INFO:teuthology.run_tasks:Running task workunit... 2024-05-09T05:06:43.472 INFO:tasks.workunit:Pulling workunits from ref c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T05:06:43.473 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-09T05:06:43.473 INFO:tasks.workunit:timeout=3h 2024-05-09T05:06:43.473 INFO:tasks.workunit:cleanup=True 2024-05-09T05:06:43.473 DEBUG:teuthology.orchestra.run.smithi001:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:43.473 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:06:43.511 INFO:teuthology.orchestra.run.smithi001.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:43.511 INFO:teuthology.orchestra.run.smithi001.stdout: Size: 0 Blocks: 1 IO Block: 4194304 directory 2024-05-09T05:06:43.511 INFO:teuthology.orchestra.run.smithi001.stdout:Device: 29h/41d Inode: 1 Links: 2 2024-05-09T05:06:43.511 INFO:teuthology.orchestra.run.smithi001.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T05:06:43.511 INFO:teuthology.orchestra.run.smithi001.stdout:Context: system_u:object_r:fusefs_t:s0 2024-05-09T05:06:43.511 INFO:teuthology.orchestra.run.smithi001.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-05-09T05:06:43.511 INFO:teuthology.orchestra.run.smithi001.stdout:Modify: 2024-05-09 05:06:10.044091531 +0000 2024-05-09T05:06:43.512 INFO:teuthology.orchestra.run.smithi001.stdout:Change: 2024-05-09 05:06:41.346439625 +0000 2024-05-09T05:06:43.512 INFO:teuthology.orchestra.run.smithi001.stdout: Birth: - 2024-05-09T05:06:43.512 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-09T05:06:43.512 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-09T05:06:43.587 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout c9da4049f638c0f84bdd9b4e33c42e86046ac554 2024-05-09T05:06:43.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.610+0000 7fad5ffb2640 1 Processor -- start 2024-05-09T05:06:43.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.611+0000 7fad5ffb2640 1 -- start start 2024-05-09T05:06:43.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5ffb2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad5806f440 0x7fad58156b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5ffb2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 0x7fad580ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5ffb2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad580cee00 con 0x7fad5815ca30 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5ffb2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fad58157070 con 0x7fad5806f440 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5ffb2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fad580cd180 con 0x7fad580ce4b0 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5dd4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad5806f440 0x7fad58156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5dd4e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad5806f440 0x7fad58156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53448/0 (socket says 172.21.15.1:53448) 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5dd4e640 1 -- 172.21.15.1:0/1763281832 learned_addr learned my addr 172.21.15.1:0/1763281832 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.612+0000 7fad5e54f640 1 --1- 172.21.15.1:0/1763281832 >> v1:172.21.15.1:6789/0 conn(0x7fad5815ca30 0x7fad5815ce40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59678/0 (socket says 172.21.15.1:59678) 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.613+0000 7fad5d54d640 1 --2- 172.21.15.1:0/1763281832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 0x7fad580ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.613+0000 7fad5dd4e640 1 -- 172.21.15.1:0/1763281832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 msgr2=0x7fad580ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.613+0000 7fad5dd4e640 1 --2- 172.21.15.1:0/1763281832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 0x7fad580ce8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.613+0000 7fad5dd4e640 1 -- 172.21.15.1:0/1763281832 >> v1:172.21.15.1:6789/0 conn(0x7fad5815ca30 legacy=0x7fad5815ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.613+0000 7fad5cd4c640 1 -- 172.21.15.1:0/1763281832 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1337818043 0 0) 0x7fad50001110 con 0x7fad5815ca30 2024-05-09T05:06:43.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.613+0000 7fad5dd4e640 1 -- 172.21.15.1:0/1763281832 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad5815ff70 con 0x7fad5806f440 2024-05-09T05:06:43.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.614+0000 7fad5dd4e640 1 --2- 172.21.15.1:0/1763281832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad58156b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fad54009ed0 tx=0x7fad5402a7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=affd61fe08660f22 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:43.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.614+0000 7fad5cd4c640 1 -- 172.21.15.1:0/1763281832 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54036070 con 0x7fad5806f440 2024-05-09T05:06:43.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.614+0000 7fad5cd4c640 1 -- 172.21.15.1:0/1763281832 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad54031600 con 0x7fad5806f440 2024-05-09T05:06:43.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.614+0000 7fad5cd4c640 1 -- 172.21.15.1:0/1763281832 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad540319d0 con 0x7fad5806f440 2024-05-09T05:06:43.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.615+0000 7fad5ffb2640 1 -- 172.21.15.1:0/1763281832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 msgr2=0x7fad58156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.615+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/1763281832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad58156b30 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fad54009ed0 tx=0x7fad5402a7d0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.615+0000 7fad5ffb2640 1 -- 172.21.15.1:0/1763281832 shutdown_connections 2024-05-09T05:06:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.615+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/1763281832 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 0x7fad580ce8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.615+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/1763281832 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad58156b30 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.615+0000 7fad5ffb2640 1 -- 172.21.15.1:0/1763281832 >> 172.21.15.1:0/1763281832 conn(0x7fad580b3670 msgr2=0x7fad580b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.615+0000 7fad5ffb2640 1 -- 172.21.15.1:0/1763281832 shutdown_connections 2024-05-09T05:06:43.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.615+0000 7fad5ffb2640 1 -- 172.21.15.1:0/1763281832 wait complete. 2024-05-09T05:06:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.617+0000 7fad5ffb2640 1 Processor -- start 2024-05-09T05:06:43.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.617+0000 7fad5ffb2640 1 -- start start 2024-05-09T05:06:43.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5ffb2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad581f6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5ffb2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 0x7fad580b58c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5ffb2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad5815ca30 0x7fad581e95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5ffb2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fad58160810 con 0x7fad5806f440 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5ffb2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fad58160530 con 0x7fad5815ca30 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5ffb2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fad580cf940 con 0x7fad580ce4b0 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5dd4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad581f6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5dd4e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad581f6490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53456/0 (socket says 172.21.15.1:53456) 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5dd4e640 1 -- 172.21.15.1:0/2744049898 learned_addr learned my addr 172.21.15.1:0/2744049898 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5d54d640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 0x7fad580b58c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5e54f640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad5815ca30 0x7fad581e95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5dd4e640 1 -- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 msgr2=0x7fad580b58c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5dd4e640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 0x7fad580b58c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5dd4e640 1 -- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad5815ca30 msgr2=0x7fad581e95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5dd4e640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad5815ca30 0x7fad581e95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.618+0000 7fad5dd4e640 1 -- 172.21.15.1:0/2744049898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad58159cc0 con 0x7fad5806f440 2024-05-09T05:06:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.619+0000 7fad5dd4e640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad581f6490 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fad54034040 tx=0x7fad54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.619+0000 7fad4e7fc640 1 -- 172.21.15.1:0/2744049898 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54036050 con 0x7fad5806f440 2024-05-09T05:06:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.619+0000 7fad4e7fc640 1 -- 172.21.15.1:0/2744049898 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad54037040 con 0x7fad5806f440 2024-05-09T05:06:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.619+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad581f69d0 con 0x7fad5806f440 2024-05-09T05:06:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.619+0000 7fad4e7fc640 1 -- 172.21.15.1:0/2744049898 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad54009510 con 0x7fad5806f440 2024-05-09T05:06:43.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.619+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad58159f30 con 0x7fad5806f440 2024-05-09T05:06:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.620+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad58050870 con 0x7fad5806f440 2024-05-09T05:06:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.621+0000 7fad4e7fc640 1 -- 172.21.15.1:0/2744049898 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fad54003990 con 0x7fad5806f440 2024-05-09T05:06:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.621+0000 7fad4e7fc640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad3c076300 0x7fad3c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.621+0000 7fad4e7fc640 1 -- 172.21.15.1:0/2744049898 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7fad540b8100 con 0x7fad5806f440 2024-05-09T05:06:43.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.622+0000 7fad5d54d640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad3c076300 0x7fad3c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:43.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.622+0000 7fad5d54d640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad3c076300 0x7fad3c078810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad58160220 tx=0x7fad48008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:43.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.627+0000 7fad4e7fc640 1 -- 172.21.15.1:0/2744049898 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fad54081400 con 0x7fad5806f440 2024-05-09T05:06:43.628 INFO:tasks.workunit.client.0.smithi001.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-09T05:06:43.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.860+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fad580008d0 con 0x7fad5806f440 2024-05-09T05:06:43.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.860+0000 7fad4e7fc640 1 -- 172.21.15.1:0/2744049898 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6915 (secure 0 0 0) 0x7fad54085700 con 0x7fad5806f440 2024-05-09T05:06:43.861 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:06:43.861 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:33.214091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34160,"mds_1":34157,"mds_2":4311,"mds_3":4287,"mds_4":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4311":{"gid":4311,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[4273,34160,4287,4311,34157]},"id":1}]} 2024-05-09T05:06:43.861 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-05-09T05:06:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad3c076300 msgr2=0x7fad3c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad3c076300 0x7fad3c078810 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fad58160220 tx=0x7fad48008040 comp rx=0 tx=0).stop 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 msgr2=0x7fad581f6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad581f6490 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fad54034040 tx=0x7fad54004a40 comp rx=0 tx=0).stop 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 shutdown_connections 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad3c076300 0x7fad3c078810 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad5815ca30 0x7fad581e95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad580ce4b0 0x7fad580b58c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 --2- 172.21.15.1:0/2744049898 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad5806f440 0x7fad581f6490 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 >> 172.21.15.1:0/2744049898 conn(0x7fad580b3670 msgr2=0x7fad580c9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 shutdown_connections 2024-05-09T05:06:43.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:43.863+0000 7fad5ffb2640 1 -- 172.21.15.1:0/2744049898 wait complete. 2024-05-09T05:06:43.875 INFO:tasks.mon_thrash.mon_thrasher:start thrashing 2024-05-09T05:06:43.875 INFO:tasks.mon_thrash.mon_thrasher:seed: 1715231203, revive delay: 20.0, thrash delay: 10.0 thrash many: False, maintain quorum: True store thrash: False, probability: 50 freeze mon: prob 10.0 duration 15.0 2024-05-09T05:06:43.876 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:06:43.876 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:06:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.031+0000 7f2d612ee640 1 Processor -- start 2024-05-09T05:06:44.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.031+0000 7f2d612ee640 1 -- start start 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.032+0000 7f2d612ee640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2d5c153830 0x7f2d5c070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.032+0000 7f2d612ee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d5c0c4430 con 0x7f2d5c159440 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.032+0000 7f2d612ee640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2d5c0c3b90 con 0x7f2d5c070530 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.032+0000 7f2d612ee640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2d5c0c3e70 con 0x7f2d5c153830 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.032+0000 7f2d5b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2d5c153830 0x7f2d5c070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.032+0000 7f2d5affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2d5c159440 0x7f2d5c15d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59690/0 (socket says 172.21.15.1:59690) 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.032+0000 7f2d5affd640 1 -- 172.21.15.1:0/1283745684 learned_addr learned my addr 172.21.15.1:0/1283745684 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.033+0000 7f2d5b7fe640 1 -- 172.21.15.1:0/1283745684 >> v1:172.21.15.196:6789/0 conn(0x7f2d5c070530 legacy=0x7f2d5c0cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.033+0000 7f2d5b7fe640 1 -- 172.21.15.1:0/1283745684 >> v1:172.21.15.1:6789/0 conn(0x7f2d5c159440 legacy=0x7f2d5c15d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.033+0000 7f2d5a7fc640 1 -- 172.21.15.1:0/1283745684 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2997549326 0 0) 0x7f2d5c0c3b90 con 0x7f2d5c070530 2024-05-09T05:06:44.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.033+0000 7f2d5b7fe640 1 -- 172.21.15.1:0/1283745684 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5c0c4150 con 0x7f2d5c153830 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.033+0000 7f2d5b7fe640 1 --2- 172.21.15.1:0/1283745684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c153830 0x7f2d5c070ae0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009d90 tx=0x7f2d5402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54e94e9f6eadb64d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.033+0000 7f2d5a7fc640 1 -- 172.21.15.1:0/1283745684 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54007740 con 0x7f2d5c153830 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d5a7fc640 1 -- 172.21.15.1:0/1283745684 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d540078e0 con 0x7f2d5c153830 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d5a7fc640 1 -- 172.21.15.1:0/1283745684 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54007cb0 con 0x7f2d5c153830 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d612ee640 1 -- 172.21.15.1:0/1283745684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c153830 msgr2=0x7f2d5c070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d612ee640 1 --2- 172.21.15.1:0/1283745684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c153830 0x7f2d5c070ae0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009d90 tx=0x7f2d5402a680 comp rx=0 tx=0).stop 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d612ee640 1 -- 172.21.15.1:0/1283745684 shutdown_connections 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d612ee640 1 --2- 172.21.15.1:0/1283745684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c153830 0x7f2d5c070ae0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d612ee640 1 -- 172.21.15.1:0/1283745684 >> 172.21.15.1:0/1283745684 conn(0x7f2d5c071b40 msgr2=0x7f2d5c071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d612ee640 1 -- 172.21.15.1:0/1283745684 shutdown_connections 2024-05-09T05:06:44.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.034+0000 7f2d612ee640 1 -- 172.21.15.1:0/1283745684 wait complete. 2024-05-09T05:06:44.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.036+0000 7f2d612ee640 1 Processor -- start 2024-05-09T05:06:44.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.036+0000 7f2d612ee640 1 -- start start 2024-05-09T05:06:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d612ee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d5c070530 0x7f2d5c1f6cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d612ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2d5c153830 0x7f2d5c15beb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d612ee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c159440 0x7f2d5c1e5dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d612ee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c0c3870 con 0x7f2d5c159440 2024-05-09T05:06:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d612ee640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2d5c15ceb0 con 0x7f2d5c070530 2024-05-09T05:06:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d612ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2d5c06fad0 con 0x7f2d5c153830 2024-05-09T05:06:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d5b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c159440 0x7f2d5c1e5dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d5b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c159440 0x7f2d5c1e5dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53478/0 (socket says 172.21.15.1:53478) 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.037+0000 7f2d5b7fe640 1 -- 172.21.15.1:0/1281680096 learned_addr learned my addr 172.21.15.1:0/1281680096 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d5affd640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d5c070530 0x7f2d5c1f6cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d53fff640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2d5c153830 0x7f2d5c15beb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d5b7fe640 1 -- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2d5c153830 msgr2=0x7f2d5c15beb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d5b7fe640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2d5c153830 0x7f2d5c15beb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d5b7fe640 1 -- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d5c070530 msgr2=0x7f2d5c1f6cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d5b7fe640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d5c070530 0x7f2d5c1f6cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d5b7fe640 1 -- 172.21.15.1:0/1281680096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d5c1f6210 con 0x7f2d5c159440 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d5b7fe640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c159440 0x7f2d5c1e5dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009eb0 tx=0x7f2d54004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d537fe640 1 -- 172.21.15.1:0/1281680096 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54041070 con 0x7f2d5c159440 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d537fe640 1 -- 172.21.15.1:0/1281680096 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2d54031c40 con 0x7f2d5c159440 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d5c140ab0 con 0x7f2d5c159440 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d537fe640 1 -- 172.21.15.1:0/1281680096 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d54003b90 con 0x7f2d5c159440 2024-05-09T05:06:44.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.038+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d5c1f64d0 con 0x7f2d5c159440 2024-05-09T05:06:44.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.039+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d5c0c4ed0 con 0x7f2d5c159440 2024-05-09T05:06:44.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.040+0000 7f2d537fe640 1 -- 172.21.15.1:0/1281680096 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2d54044070 con 0x7f2d5c159440 2024-05-09T05:06:44.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.041+0000 7f2d537fe640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2d3c076300 0x7f2d3c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.041+0000 7f2d537fe640 1 -- 172.21.15.1:0/1281680096 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2d540b8bb0 con 0x7f2d5c159440 2024-05-09T05:06:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.041+0000 7f2d5affd640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2d3c076300 0x7f2d3c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.042+0000 7f2d5affd640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2d3c076300 0x7f2d3c078810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c000f60 tx=0x7f2d4c007c60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.046+0000 7f2d537fe640 1 -- 172.21.15.1:0/1281680096 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2d5c0c4ed0 con 0x7f2d5c159440 2024-05-09T05:06:44.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.315+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2d5c077020 con 0x7f2d5c159440 2024-05-09T05:06:44.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.316+0000 7f2d537fe640 1 -- 172.21.15.1:0/1281680096 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f2d5c077020 con 0x7f2d5c159440 2024-05-09T05:06:44.316 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":62,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:06:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2d3c076300 msgr2=0x7f2d3c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2d3c076300 0x7f2d3c078810 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f2d4c000f60 tx=0x7f2d4c007c60 comp rx=0 tx=0).stop 2024-05-09T05:06:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c159440 msgr2=0x7f2d5c1e5dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c159440 0x7f2d5c1e5dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009eb0 tx=0x7f2d54004a40 comp rx=0 tx=0).stop 2024-05-09T05:06:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 shutdown_connections 2024-05-09T05:06:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2d3c076300 0x7f2d3c078810 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2d5c159440 0x7f2d5c1e5dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2d5c153830 0x7f2d5c15beb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 --2- 172.21.15.1:0/1281680096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2d5c070530 0x7f2d5c1f6cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 >> 172.21.15.1:0/1281680096 conn(0x7f2d5c071b40 msgr2=0x7f2d5c159900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 shutdown_connections 2024-05-09T05:06:44.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.318+0000 7f2d612ee640 1 -- 172.21.15.1:0/1281680096 wait complete. 2024-05-09T05:06:44.330 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:06:44.330 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:06:44.330 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:06:44.330 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:06:44.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.447+0000 7f412776b640 1 Processor -- start 2024-05-09T05:06:44.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.447+0000 7f412776b640 1 -- start start 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.448+0000 7f412776b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f4120072400 0x7f4120154530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.448+0000 7f412776b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4120071310 0x7f4120164ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.448+0000 7f412776b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f41200cbf90 con 0x7f4120070f00 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.448+0000 7f412776b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f41200ca3c0 con 0x7f4120072400 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.448+0000 7f412776b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f412015a780 con 0x7f4120071310 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.448+0000 7f4124d06640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4120070f00 0x7f41200cae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59704/0 (socket says 172.21.15.1:59704) 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.448+0000 7f4124d06640 1 -- 172.21.15.1:0/2694386915 learned_addr learned my addr 172.21.15.1:0/2694386915 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.448+0000 7f4125507640 1 --2- 172.21.15.1:0/2694386915 >> v2:172.21.15.196:3300/0 conn(0x7f4120072400 0x7f4120154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4125d08640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4120071310 0x7f4120164ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2378933759 0 0) 0x7f41200cbf90 con 0x7f4120070f00 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f410c003520 con 0x7f4120070f00 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2207723786 0 0) 0x7f410c003520 con 0x7f4120070f00 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4120071310 msgr2=0x7f4120164ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4120071310 0x7f4120164ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 >> v2:172.21.15.196:3300/0 conn(0x7f4120072400 msgr2=0x7f4120154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 --2- 172.21.15.1:0/2694386915 >> v2:172.21.15.196:3300/0 conn(0x7f4120072400 0x7f4120154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41200ca610 con 0x7f4120070f00 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f41100016f0 con 0x7f4120070f00 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 >> v1:172.21.15.1:6789/0 conn(0x7f4120070f00 legacy=0x7f41200cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4120071310 0x7f410c003a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c003ff0 0x7f410c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f410c004d40 0x7f410c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f41200ca3c0 con 0x7f410c003ff0 2024-05-09T05:06:44.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f412015a780 con 0x7f410c004d40 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f410c00baf0 con 0x7f4120071310 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4124d06640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c003ff0 0x7f410c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4125507640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4120071310 0x7f410c003a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4125d08640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f410c004d40 0x7f410c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4124d06640 1 -- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4120071310 msgr2=0x7f410c003a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4124d06640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4120071310 0x7f410c003a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4124d06640 1 -- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f410c004d40 msgr2=0x7f410c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4124d06640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f410c004d40 0x7f410c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.449+0000 7f4124d06640 1 -- 172.21.15.1:0/2694386915 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f410c003520 con 0x7f410c003ff0 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.450+0000 7f4124d06640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c003ff0 0x7f410c004800 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f411000c210 tx=0x7f410c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79a3e78988de0f43 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.450+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110007d30 con 0x7f410c003ff0 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.450+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411002e070 con 0x7f410c003ff0 2024-05-09T05:06:44.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.450+0000 7f4117fff640 1 -- 172.21.15.1:0/2694386915 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110005d30 con 0x7f410c003ff0 2024-05-09T05:06:44.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.451+0000 7f412776b640 1 -- 172.21.15.1:0/2694386915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c003ff0 msgr2=0x7f410c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.451+0000 7f412776b640 1 --2- 172.21.15.1:0/2694386915 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f410c003ff0 0x7f410c004800 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f411000c210 tx=0x7f410c005830 comp rx=0 tx=0).stop 2024-05-09T05:06:44.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.451+0000 7f4125d08640 1 -- 172.21.15.1:0/2694386915 reap_dead start 2024-05-09T05:06:44.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.451+0000 7f412776b640 1 -- 172.21.15.1:0/2694386915 shutdown_connections 2024-05-09T05:06:44.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.451+0000 7f412776b640 1 -- 172.21.15.1:0/2694386915 >> 172.21.15.1:0/2694386915 conn(0x7f41200d0470 msgr2=0x7f41200cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.451+0000 7f412776b640 1 -- 172.21.15.1:0/2694386915 shutdown_connections 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.451+0000 7f412776b640 1 -- 172.21.15.1:0/2694386915 wait complete. 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.453+0000 7f412776b640 1 Processor -- start 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.453+0000 7f412776b640 1 -- start start 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f412776b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f410c003ff0 0x7f4120158710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f412776b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41201e47c0 0x7f41200cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f412776b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cf260 0x7f41201e6160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f412776b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f41200b2630 con 0x7f41200cf260 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f412776b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4120071720 con 0x7f41201e47c0 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f412776b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f41200b27b0 con 0x7f410c003ff0 2024-05-09T05:06:44.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f4125d08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cf260 0x7f41201e6160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f4125d08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cf260 0x7f41201e6160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53498/0 (socket says 172.21.15.1:53498) 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.454+0000 7f4125d08640 1 -- 172.21.15.1:0/2646856602 learned_addr learned my addr 172.21.15.1:0/2646856602 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4124d06640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41201e47c0 0x7f41200cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4125507640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f410c003ff0 0x7f4120158710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4125d08640 1 -- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f410c003ff0 msgr2=0x7f4120158710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4125d08640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f410c003ff0 0x7f4120158710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4125d08640 1 -- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41201e47c0 msgr2=0x7f41200cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4125d08640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f41201e47c0 0x7f41200cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4125d08640 1 -- 172.21.15.1:0/2646856602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41200cdd60 con 0x7f41200cf260 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4125d08640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cf260 0x7f41201e6160 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f41200b3ec0 tx=0x7f411800f1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411800fc80 con 0x7f41200cf260 2024-05-09T05:06:44.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f411800fe20 con 0x7f41200cf260 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41201f0400 con 0x7f41200cf260 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4118015670 con 0x7f41200cf260 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.455+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41201f09c0 con 0x7f41200cf260 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.456+0000 7f412776b640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cdf30 0x7f412015d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.456+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4120050870 con 0x7f41200cdf30 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.456+0000 7f4125507640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cdf30 0x7f412015d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4125507640 1 -- 172.21.15.1:0/2646856602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4120050dc0 con 0x7f41200cdf30 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4125507640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cdf30 0x7f412015d980 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f411c014a50 tx=0x7f411c014f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f411c022070 con 0x7f41200cdf30 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f411c012db0 con 0x7f41200cdf30 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cdf30 msgr2=0x7f412015d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4115ffb640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cdf30 0x7f412015d980 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f411c014a50 tx=0x7f411c014f10 comp rx=0 tx=0).stop 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cdf30 msgr2=0x7f412015d980 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4115ffb640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cdf30 0x7f412015d980 secure :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f411c014a50 tx=0x7f411c014f10 comp rx=0 tx=0).stop 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.457+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4118015810 con 0x7f41200cf260 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.458+0000 7f4115ffb640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4104075de0 0x7f41040782f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.458+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f4118098170 con 0x7f41200cf260 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.458+0000 7f4125507640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4104075de0 0x7f41040782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.459+0000 7f4125507640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4104075de0 0x7f41040782f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f411c012860 tx=0x7f411c019000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.464+0000 7f412776b640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412015dec0 0x7f41201f3620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.464+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f412015e2d0 con 0x7f412015dec0 2024-05-09T05:06:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.464+0000 7f4124d06640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412015dec0 0x7f41201f3620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.464+0000 7f4124d06640 1 -- 172.21.15.1:0/2646856602 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f41200cbcd0 con 0x7f412015dec0 2024-05-09T05:06:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.464+0000 7f4124d06640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412015dec0 0x7f41201f3620 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4110001eb0 tx=0x7f412015a950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4110044020 con 0x7f412015dec0 2024-05-09T05:06:44.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f411002e050 con 0x7f412015dec0 2024-05-09T05:06:44.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412015dec0 msgr2=0x7f41201f3620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f4115ffb640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412015dec0 0x7f41201f3620 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f4110001eb0 tx=0x7f412015a950 comp rx=0 tx=0).stop 2024-05-09T05:06:44.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f4115ffb640 1 -- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412015dec0 msgr2=0x7f41201f3620 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:06:44.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f4115ffb640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f412015dec0 0x7f41201f3620 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.466 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 8, 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 68590, 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 62, 2024-05-09T05:06:44.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.468 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.469 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:06:44.470 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:06:44.471 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:06:44.471 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:06:44.471 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:06:44.471 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.472 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.476 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.478 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.479 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.480 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:06:44.481 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4104075de0 msgr2=0x7f41040782f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4104075de0 0x7f41040782f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f411c012860 tx=0x7f411c019000 comp rx=0 tx=0).stop 2024-05-09T05:06:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cf260 msgr2=0x7f41201e6160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f4125d08640 1 -- 172.21.15.1:0/2646856602 reap_dead start 2024-05-09T05:06:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cf260 0x7f41201e6160 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f41200b3ec0 tx=0x7f411800f1b0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 shutdown_connections 2024-05-09T05:06:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 --2- 172.21.15.1:0/2646856602 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f41200cf260 0x7f41201e6160 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 >> 172.21.15.1:0/2646856602 conn(0x7f41200d0470 msgr2=0x7f4120153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 shutdown_connections 2024-05-09T05:06:44.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.465+0000 7f412776b640 1 -- 172.21.15.1:0/2646856602 wait complete. 2024-05-09T05:06:44.483 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:06:44.483 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:06:44.484 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:06:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.598+0000 7f514be3b640 1 Processor -- start 2024-05-09T05:06:44.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.598+0000 7f514be3b640 1 -- start start 2024-05-09T05:06:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f514be3b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f51440cd610 0x7f514415ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f514be3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51440cdaf0 0x7f514415d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f514be3b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5144160160 con 0x7f5144156c20 2024-05-09T05:06:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f514be3b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f5144160ce0 con 0x7f51440cd610 2024-05-09T05:06:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f514be3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5144160440 con 0x7f51440cdaf0 2024-05-09T05:06:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5149bd7640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5144156c20 0x7f51440cea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59720/0 (socket says 172.21.15.1:59720) 2024-05-09T05:06:44.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5149bd7640 1 -- 172.21.15.1:0/2286355093 learned_addr learned my addr 172.21.15.1:0/2286355093 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f514a3d8640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51440cdaf0 0x7f514415d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f51493d6640 1 --2- 172.21.15.1:0/2286355093 >> v2:172.21.15.196:3300/0 conn(0x7f51440cd610 0x7f514415ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2319842905 0 0) 0x7f5144160160 con 0x7f5144156c20 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f512c003520 con 0x7f5144156c20 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 92205445 0 0) 0x7f512c003520 con 0x7f5144156c20 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51440cdaf0 msgr2=0x7f514415d350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5148bd5640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51440cdaf0 0x7f514415d350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 >> v2:172.21.15.196:3300/0 conn(0x7f51440cd610 msgr2=0x7f514415ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.600+0000 7f5148bd5640 1 --2- 172.21.15.1:0/2286355093 >> v2:172.21.15.196:3300/0 conn(0x7f51440cd610 0x7f514415ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5144160720 con 0x7f5144156c20 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f513c0016f0 con 0x7f5144156c20 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 >> v1:172.21.15.1:6789/0 conn(0x7f5144156c20 legacy=0x7f51440cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f512c003a00 0x7f512c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f512c004800 0x7f512c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51440cdaf0 0x7f512c005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5144160ce0 con 0x7f512c004800 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5144160440 con 0x7f512c003a00 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f512c00baf0 con 0x7f51440cdaf0 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f51493d6640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f512c004800 0x7f512c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f5149bd7640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f512c003a00 0x7f512c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f514a3d8640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51440cdaf0 0x7f512c005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f51493d6640 1 -- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51440cdaf0 msgr2=0x7f512c005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f51493d6640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51440cdaf0 0x7f512c005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f51493d6640 1 -- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f512c003a00 msgr2=0x7f512c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f51493d6640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f512c003a00 0x7f512c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.601+0000 7f51493d6640 1 -- 172.21.15.1:0/2286355093 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f512c003520 con 0x7f512c004800 2024-05-09T05:06:44.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.602+0000 7f51493d6640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f512c004800 0x7f512c004c10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5138009ac0 tx=0x7f512c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ff4b0f41349bd9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.602+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138036070 con 0x7f512c004800 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.602+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5138031400 con 0x7f512c004800 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.602+0000 7f5148bd5640 1 -- 172.21.15.1:0/2286355093 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138009690 con 0x7f512c004800 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.603+0000 7f514be3b640 1 -- 172.21.15.1:0/2286355093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f512c004800 msgr2=0x7f512c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.603+0000 7f514be3b640 1 --2- 172.21.15.1:0/2286355093 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f512c004800 0x7f512c004c10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f5138009ac0 tx=0x7f512c005830 comp rx=0 tx=0).stop 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.603+0000 7f514a3d8640 1 -- 172.21.15.1:0/2286355093 reap_dead start 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.603+0000 7f514be3b640 1 -- 172.21.15.1:0/2286355093 shutdown_connections 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.603+0000 7f514be3b640 1 -- 172.21.15.1:0/2286355093 >> 172.21.15.1:0/2286355093 conn(0x7f5144068830 msgr2=0x7f5144068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.603+0000 7f514be3b640 1 -- 172.21.15.1:0/2286355093 shutdown_connections 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.603+0000 7f514be3b640 1 -- 172.21.15.1:0/2286355093 wait complete. 2024-05-09T05:06:44.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.605+0000 7f514be3b640 1 Processor -- start 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.605+0000 7f514be3b640 1 -- start start 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514be3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f512c004800 0x7f514415e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514be3b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e85f0 0x7f5144150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514be3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51441f60b0 0x7f51441f64c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514be3b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f51440ce2a0 con 0x7f51441f60b0 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514be3b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f514415c8f0 con 0x7f51441e85f0 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514be3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f51440cf130 con 0x7f512c004800 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514a3d8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51441f60b0 0x7f51441f64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514a3d8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51441f60b0 0x7f51441f64c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53546/0 (socket says 172.21.15.1:53546) 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514a3d8640 1 -- 172.21.15.1:0/1675925502 learned_addr learned my addr 172.21.15.1:0/1675925502 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f51493d6640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e85f0 0x7f5144150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f5149bd7640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f512c004800 0x7f514415e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514a3d8640 1 -- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f512c004800 msgr2=0x7f514415e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514a3d8640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f512c004800 0x7f514415e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514a3d8640 1 -- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e85f0 msgr2=0x7f5144150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514a3d8640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e85f0 0x7f5144150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.606+0000 7f514a3d8640 1 -- 172.21.15.1:0/1675925502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514415e550 con 0x7f51441f60b0 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.607+0000 7f514a3d8640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51441f60b0 0x7f51441f64c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f514408fd60 tx=0x7f513400f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.607+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134011e60 con 0x7f51441f60b0 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.607+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5134015430 con 0x7f51441f60b0 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.607+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514414fd50 con 0x7f51441f60b0 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.607+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5134015800 con 0x7f51441f60b0 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.607+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51440cbbe0 con 0x7f51441f60b0 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.608+0000 7f514be3b640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514415e720 0x7f51441f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.608+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f51441f6f40 con 0x7f514415e720 2024-05-09T05:06:44.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.608+0000 7f5149bd7640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514415e720 0x7f51441f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.608+0000 7f5149bd7640 1 -- 172.21.15.1:0/1675925502 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f51441e5eb0 con 0x7f514415e720 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.608+0000 7f5149bd7640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514415e720 0x7f51441f6a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f513c0072f0 tx=0x7f513c0077b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c012a10 con 0x7f514415e720 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f51340159a0 con 0x7f51441f60b0 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f51300761d0 0x7f51300786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f5134098200 con 0x7f51441f60b0 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f513c01a070 con 0x7f514415e720 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514415e720 msgr2=0x7f51441f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514415e720 0x7f51441f6a00 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f513c0072f0 tx=0x7f513c0077b0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514415e720 msgr2=0x7f51441f6a00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51427fc640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f514415e720 0x7f51441f6a00 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.609+0000 7f51493d6640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f51300761d0 0x7f51300786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.610+0000 7f51493d6640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f51300761d0 0x7f51300786e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5138009ac0 tx=0x7f5138008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.616+0000 7f514be3b640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e7820 0x7f514414e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.616+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f514406f2f0 con 0x7f51441e7820 2024-05-09T05:06:44.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.616+0000 7f5149bd7640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e7820 0x7f514414e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.617+0000 7f5149bd7640 1 -- 172.21.15.1:0/1675925502 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f51440ce420 con 0x7f51441e7820 2024-05-09T05:06:44.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.617+0000 7f5149bd7640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e7820 0x7f514414e020 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f513c0072f0 tx=0x7f51440b4b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.617+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c0159e0 con 0x7f51441e7820 2024-05-09T05:06:44.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.617+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7f513c0159e0 con 0x7f51441e7820 2024-05-09T05:06:44.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.617+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e7820 msgr2=0x7f514414e020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.617+0000 7f51427fc640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e7820 0x7f514414e020 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f513c0072f0 tx=0x7f51440b4b00 comp rx=0 tx=0).stop 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.617+0000 7f51427fc640 1 -- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e7820 msgr2=0x7f514414e020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.617+0000 7f51427fc640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f51441e7820 0x7f514414e020 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 8, 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 68707, 2024-05-09T05:06:44.619 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 62, 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.621 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:06:44.622 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.623 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:06:44.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.630 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:06:44.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.633 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:06:44.633 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f51300761d0 msgr2=0x7f51300786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514be3b640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f51300761d0 0x7f51300786e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f5138009ac0 tx=0x7f5138008030 comp rx=0 tx=0).stop 2024-05-09T05:06:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51441f60b0 msgr2=0x7f51441f64c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514be3b640 1 --2- 172.21.15.1:0/1675925502 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51441f60b0 0x7f51441f64c0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f514408fd60 tx=0x7f513400f580 comp rx=0 tx=0).stop 2024-05-09T05:06:44.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514a3d8640 1 -- 172.21.15.1:0/1675925502 reap_dead start 2024-05-09T05:06:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514a3d8640 1 -- 172.21.15.1:0/1675925502 reap_dead start 2024-05-09T05:06:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 shutdown_connections 2024-05-09T05:06:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 >> 172.21.15.1:0/1675925502 conn(0x7f5144068830 msgr2=0x7f51440b6440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 shutdown_connections 2024-05-09T05:06:44.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.618+0000 7f514be3b640 1 -- 172.21.15.1:0/1675925502 wait complete. 2024-05-09T05:06:44.635 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:06:44.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:06:44.635 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:06:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.750+0000 7f37fb32d640 1 Processor -- start 2024-05-09T05:06:44.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.750+0000 7f37fb32d640 1 -- start start 2024-05-09T05:06:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37fb32d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f37f40ce010 0x7f37f40c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37fb32d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f37f40c77c0 0x7f37f40c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37fb32d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f37f415c7b0 con 0x7f37f40b4ec0 2024-05-09T05:06:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37fb32d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f37f415d0c0 con 0x7f37f40ce010 2024-05-09T05:06:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37fb32d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f37f40cca50 con 0x7f37f40c77c0 2024-05-09T05:06:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f88c8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f37f40ce010 0x7f37f40c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f88c8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f37f40ce010 0x7f37f40c7280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53556/0 (socket says 172.21.15.1:53556) 2024-05-09T05:06:44.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f88c8640 1 -- 172.21.15.1:0/1589845415 learned_addr learned my addr 172.21.15.1:0/1589845415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f98ca640 1 --2- 172.21.15.1:0/1589845415 >> v2:172.21.15.196:3300/0 conn(0x7f37f40c77c0 0x7f37f40c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f88c8640 1 -- 172.21.15.1:0/1589845415 >> v2:172.21.15.196:3300/0 conn(0x7f37f40c77c0 msgr2=0x7f37f40c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f88c8640 1 --2- 172.21.15.1:0/1589845415 >> v2:172.21.15.196:3300/0 conn(0x7f37f40c77c0 0x7f37f40c7bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f88c8640 1 -- 172.21.15.1:0/1589845415 >> v1:172.21.15.196:6789/0 conn(0x7f37f40b4ec0 legacy=0x7f37f40cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f88c8640 1 -- 172.21.15.1:0/1589845415 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f40ccd30 con 0x7f37f40ce010 2024-05-09T05:06:44.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.752+0000 7f37f98ca640 1 --2- 172.21.15.1:0/1589845415 >> v2:172.21.15.196:3300/0 conn(0x7f37f40c77c0 0x7f37f40c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.753+0000 7f37f88c8640 1 --2- 172.21.15.1:0/1589845415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 0x7f37f40c7280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f37f408fd40 tx=0x7f37e802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=258712e2f7b64869 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.753+0000 7f37f3fff640 1 -- 172.21.15.1:0/1589845415 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8037070 con 0x7f37f40ce010 2024-05-09T05:06:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.753+0000 7f37f3fff640 1 -- 172.21.15.1:0/1589845415 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37e8032440 con 0x7f37f40ce010 2024-05-09T05:06:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.753+0000 7f37f3fff640 1 -- 172.21.15.1:0/1589845415 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e8032810 con 0x7f37f40ce010 2024-05-09T05:06:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.754+0000 7f37fb32d640 1 -- 172.21.15.1:0/1589845415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 msgr2=0x7f37f40c7280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.754+0000 7f37fb32d640 1 --2- 172.21.15.1:0/1589845415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 0x7f37f40c7280 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f37f408fd40 tx=0x7f37e802a270 comp rx=0 tx=0).stop 2024-05-09T05:06:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.754+0000 7f37fb32d640 1 -- 172.21.15.1:0/1589845415 shutdown_connections 2024-05-09T05:06:44.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.754+0000 7f37fb32d640 1 --2- 172.21.15.1:0/1589845415 >> v2:172.21.15.196:3300/0 conn(0x7f37f40c77c0 0x7f37f40c7bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.754+0000 7f37fb32d640 1 --2- 172.21.15.1:0/1589845415 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 0x7f37f40c7280 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.754+0000 7f37fb32d640 1 -- 172.21.15.1:0/1589845415 >> 172.21.15.1:0/1589845415 conn(0x7f37f40d2360 msgr2=0x7f37f40ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.754+0000 7f37fb32d640 1 -- 172.21.15.1:0/1589845415 shutdown_connections 2024-05-09T05:06:44.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.754+0000 7f37fb32d640 1 -- 172.21.15.1:0/1589845415 wait complete. 2024-05-09T05:06:44.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.756+0000 7f37fb32d640 1 Processor -- start 2024-05-09T05:06:44.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.756+0000 7f37fb32d640 1 -- start start 2024-05-09T05:06:44.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37fb32d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37f40b4ec0 0x7f37f414e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37fb32d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f40c77c0 0x7f37f4150760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37fb32d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 0x7f37f4155bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37fb32d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f37f40cc750 con 0x7f37f40ce010 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37fb32d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f37f415cd50 con 0x7f37f40b4ec0 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37fb32d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f37f40c8110 con 0x7f37f40c77c0 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37f98ca640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 0x7f37f4155bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37f98ca640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 0x7f37f4155bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53562/0 (socket says 172.21.15.1:53562) 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37f98ca640 1 -- 172.21.15.1:0/3285995026 learned_addr learned my addr 172.21.15.1:0/3285995026 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37f88c8640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f40c77c0 0x7f37f4150760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.757+0000 7f37f90c9640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37f40b4ec0 0x7f37f414e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f98ca640 1 -- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f40c77c0 msgr2=0x7f37f4150760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f98ca640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f40c77c0 0x7f37f4150760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f98ca640 1 -- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37f40b4ec0 msgr2=0x7f37f414e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f98ca640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f37f40b4ec0 0x7f37f414e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f98ca640 1 -- 172.21.15.1:0/3285995026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37f414e6a0 con 0x7f37f40ce010 2024-05-09T05:06:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f98ca640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 0x7f37f4155bf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f37e4007e60 tx=0x7f37e400d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e401a070 con 0x7f37f40ce010 2024-05-09T05:06:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37e4015600 con 0x7f37f40ce010 2024-05-09T05:06:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37f4150ca0 con 0x7f37f40ce010 2024-05-09T05:06:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37e40159f0 con 0x7f37f40ce010 2024-05-09T05:06:44.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.758+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37f41f3500 con 0x7f37f40ce010 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.759+0000 7f37fb32d640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f414fe70 0x7f37f41f3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.759+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f37f414e910 con 0x7f37f414fe70 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.759+0000 7f37f90c9640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f414fe70 0x7f37f41f3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.760+0000 7f37f90c9640 1 -- 172.21.15.1:0/3285995026 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f37f4050870 con 0x7f37f414fe70 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.760+0000 7f37f90c9640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f414fe70 0x7f37f41f3740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37ec000f60 tx=0x7f37ec00e890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.760+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec0052e0 con 0x7f37f414fe70 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.760+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f37e401d070 con 0x7f37f40ce010 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.761+0000 7f37f1ffb640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f37e00761d0 0x7f37e00786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.761+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f37e4097850 con 0x7f37f40ce010 2024-05-09T05:06:44.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.761+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f37ec005480 con 0x7f37f414fe70 2024-05-09T05:06:44.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.761+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f414fe70 msgr2=0x7f37f41f3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.761+0000 7f37f1ffb640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f414fe70 0x7f37f41f3740 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f37ec000f60 tx=0x7f37ec00e890 comp rx=0 tx=0).stop 2024-05-09T05:06:44.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.761+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f414fe70 msgr2=0x7f37f41f3740 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:06:44.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.761+0000 7f37f1ffb640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f414fe70 0x7f37f41f3740 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.761+0000 7f37f88c8640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f37e00761d0 0x7f37e00786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.762+0000 7f37f88c8640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f37e00761d0 0x7f37e00786e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f37e8034000 tx=0x7f37e802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.768+0000 7f37fb32d640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f41f3c80 0x7f37f40b6580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.768+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f37f415a160 con 0x7f37f41f3c80 2024-05-09T05:06:44.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.768+0000 7f37f90c9640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f41f3c80 0x7f37f40b6580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.768+0000 7f37f90c9640 1 -- 172.21.15.1:0/3285995026 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f37f40cd2b0 con 0x7f37f41f3c80 2024-05-09T05:06:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f90c9640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f41f3c80 0x7f37f40b6580 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f37ec000f60 tx=0x7f37ec000fe0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f37ec006d30 con 0x7f37f41f3c80 2024-05-09T05:06:44.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5060 (secure 0 0 0) 0x7f37ec006d30 con 0x7f37f41f3c80 2024-05-09T05:06:44.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f41f3c80 msgr2=0x7f37f40b6580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f1ffb640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f41f3c80 0x7f37f40b6580 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f37ec000f60 tx=0x7f37ec000fe0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f1ffb640 1 -- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f41f3c80 msgr2=0x7f37f40b6580 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:06:44.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f1ffb640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f37f41f3c80 0x7f37f40b6580 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.770 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 8, 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 68857, 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 62, 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:06:44.771 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.772 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.773 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:06:44.774 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:06:44.775 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.776 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.777 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:06:44.778 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.779 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:06:44.780 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.781 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:06:44.782 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:06:44.783 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:06:44.784 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:06:44.784 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:06:44.784 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:06:44.784 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:06:44.784 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:06:44.784 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:06:44.784 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:06:44.784 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:06:44.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f37e00761d0 msgr2=0x7f37e00786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37fb32d640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f37e00761d0 0x7f37e00786e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f37e8034000 tx=0x7f37e802e000 comp rx=0 tx=0).stop 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 msgr2=0x7f37f4155bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37fb32d640 1 --2- 172.21.15.1:0/3285995026 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f37f40ce010 0x7f37f4155bf0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f37e4007e60 tx=0x7f37e400d4a0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f98ca640 1 -- 172.21.15.1:0/3285995026 reap_dead start 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37f98ca640 1 -- 172.21.15.1:0/3285995026 reap_dead start 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 shutdown_connections 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.769+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 >> 172.21.15.1:0/3285995026 conn(0x7f37f40d2360 msgr2=0x7f37f40cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.770+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 shutdown_connections 2024-05-09T05:06:44.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.770+0000 7f37fb32d640 1 -- 172.21.15.1:0/3285995026 wait complete. 2024-05-09T05:06:44.786 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:06:44.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-05-09T05:06:44.787 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:06:44.787 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-05-09T05:06:44.787 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:06:44.787 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:06:44.787 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:06:44.803 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:06:44.803 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:06:44.803 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:06:44.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.916+0000 7f3353957640 1 Processor -- start 2024-05-09T05:06:44.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.917+0000 7f3353957640 1 -- start start 2024-05-09T05:06:44.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3353957640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f334c0746a0 0x7f334c0cb690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3353957640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f334c074ab0 0x7f334c0cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3353957640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f334c15b370 con 0x7f334c15af60 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3353957640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f334c0c9b00 con 0x7f334c0746a0 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3353957640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f334c15dc70 con 0x7f334c074ab0 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3350ef2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f334c0746a0 0x7f334c0cb690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3350ef2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f334c0746a0 0x7f334c0cb690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53568/0 (socket says 172.21.15.1:53568) 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3350ef2640 1 -- 172.21.15.1:0/3774382659 learned_addr learned my addr 172.21.15.1:0/3774382659 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3351ef4640 1 --2- 172.21.15.1:0/3774382659 >> v2:172.21.15.196:3300/0 conn(0x7f334c074ab0 0x7f334c0cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3350ef2640 1 -- 172.21.15.1:0/3774382659 >> v2:172.21.15.196:3300/0 conn(0x7f334c074ab0 msgr2=0x7f334c0cbbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3350ef2640 1 --2- 172.21.15.1:0/3774382659 >> v2:172.21.15.196:3300/0 conn(0x7f334c074ab0 0x7f334c0cbbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3350ef2640 1 -- 172.21.15.1:0/3774382659 >> v1:172.21.15.196:6789/0 conn(0x7f334c15af60 legacy=0x7f334c150e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.918+0000 7f3350ef2640 1 -- 172.21.15.1:0/3774382659 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334c15df50 con 0x7f334c0746a0 2024-05-09T05:06:44.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.919+0000 7f3350ef2640 1 --2- 172.21.15.1:0/3774382659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cb690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f334c151510 tx=0x7f334002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e07be88acbd3cfef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.919+0000 7f334bfff640 1 -- 172.21.15.1:0/3774382659 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3340037070 con 0x7f334c0746a0 2024-05-09T05:06:44.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.919+0000 7f334bfff640 1 -- 172.21.15.1:0/3774382659 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3340032440 con 0x7f334c0746a0 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.919+0000 7f334bfff640 1 -- 172.21.15.1:0/3774382659 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3340032810 con 0x7f334c0746a0 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.920+0000 7f3353957640 1 -- 172.21.15.1:0/3774382659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 msgr2=0x7f334c0cb690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.920+0000 7f3353957640 1 --2- 172.21.15.1:0/3774382659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cb690 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f334c151510 tx=0x7f334002a1e0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.920+0000 7f3353957640 1 -- 172.21.15.1:0/3774382659 shutdown_connections 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.920+0000 7f3353957640 1 --2- 172.21.15.1:0/3774382659 >> v2:172.21.15.196:3300/0 conn(0x7f334c074ab0 0x7f334c0cbbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.920+0000 7f3353957640 1 --2- 172.21.15.1:0/3774382659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cb690 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.920+0000 7f3353957640 1 -- 172.21.15.1:0/3774382659 >> 172.21.15.1:0/3774382659 conn(0x7f334c071b40 msgr2=0x7f334c071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.920+0000 7f3353957640 1 -- 172.21.15.1:0/3774382659 shutdown_connections 2024-05-09T05:06:44.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.920+0000 7f3353957640 1 -- 172.21.15.1:0/3774382659 wait complete. 2024-05-09T05:06:44.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.922+0000 7f3353957640 1 Processor -- start 2024-05-09T05:06:44.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.922+0000 7f3353957640 1 -- start start 2024-05-09T05:06:44.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f3353957640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cd390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f3353957640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f334c15af60 0x7f334c158c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f3353957640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f334c0ce350 0x7f334c15d060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f3353957640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f334c15e7f0 con 0x7f334c0746a0 2024-05-09T05:06:44.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f3353957640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f334c15e510 con 0x7f334c15af60 2024-05-09T05:06:44.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f3353957640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f334c0c9f90 con 0x7f334c0ce350 2024-05-09T05:06:44.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f33516f3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f33516f3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cd390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53578/0 (socket says 172.21.15.1:53578) 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f33516f3640 1 -- 172.21.15.1:0/1304401540 learned_addr learned my addr 172.21.15.1:0/1304401540 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f3351ef4640 1 -- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f334c0ce350 msgr2=0x7f334c15d060 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.923+0000 7f3351ef4640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f334c0ce350 0x7f334c15d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f3350ef2640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f334c15af60 0x7f334c158c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f33516f3640 1 -- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f334c0ce350 msgr2=0x7f334c15d060 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f33516f3640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f334c0ce350 0x7f334c15d060 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f33516f3640 1 -- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f334c15af60 msgr2=0x7f334c158c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f33516f3640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f334c15af60 0x7f334c158c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f33516f3640 1 -- 172.21.15.1:0/1304401540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f334c159170 con 0x7f334c0746a0 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f3350ef2640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f334c15af60 0x7f334c158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f33516f3640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cd390 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f334400b3e0 tx=0x7f334400b8a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f3349ffb640 1 -- 172.21.15.1:0/1304401540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344015020 con 0x7f334c0746a0 2024-05-09T05:06:44.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f3349ffb640 1 -- 172.21.15.1:0/1304401540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3344011850 con 0x7f334c0746a0 2024-05-09T05:06:44.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f334c15bf60 con 0x7f334c0746a0 2024-05-09T05:06:44.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.924+0000 7f3349ffb640 1 -- 172.21.15.1:0/1304401540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3344011c40 con 0x7f334c0746a0 2024-05-09T05:06:44.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.925+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f334c0ccb90 con 0x7f334c0746a0 2024-05-09T05:06:44.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.925+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f334c0c7640 con 0x7f334c0746a0 2024-05-09T05:06:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.926+0000 7f3349ffb640 1 -- 172.21.15.1:0/1304401540 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3344018020 con 0x7f334c0746a0 2024-05-09T05:06:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.927+0000 7f3349ffb640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33380762b0 0x7f33380787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.927+0000 7f3349ffb640 1 -- 172.21.15.1:0/1304401540 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f3344096f10 con 0x7f334c0746a0 2024-05-09T05:06:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.927+0000 7f3350ef2640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33380762b0 0x7f33380787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:44.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.928+0000 7f3350ef2640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33380762b0 0x7f33380787c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3340034000 tx=0x7f334002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:44.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:44.932+0000 7f3349ffb640 1 -- 172.21.15.1:0/1304401540 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f334401f070 con 0x7f334c0746a0 2024-05-09T05:06:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.204+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f334c0c7360 con 0x7f334c0746a0 2024-05-09T05:06:45.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.204+0000 7f3349ffb640 1 -- 172.21.15.1:0/1304401540 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f3344060870 con 0x7f334c0746a0 2024-05-09T05:06:45.205 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":63,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33380762b0 msgr2=0x7f33380787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33380762b0 0x7f33380787c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f3340034000 tx=0x7f334002e000 comp rx=0 tx=0).stop 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 msgr2=0x7f334c0cd390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cd390 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f334400b3e0 tx=0x7f334400b8a0 comp rx=0 tx=0).stop 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 shutdown_connections 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33380762b0 0x7f33380787c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f334c15af60 0x7f334c158c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f334c0ce350 0x7f334c15d060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 --2- 172.21.15.1:0/1304401540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f334c0746a0 0x7f334c0cd390 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 >> 172.21.15.1:0/1304401540 conn(0x7f334c071b40 msgr2=0x7f334c070670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:45.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 shutdown_connections 2024-05-09T05:06:45.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:45.206+0000 7f3353957640 1 -- 172.21.15.1:0/1304401540 wait complete. 2024-05-09T05:06:48.219 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:06:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.346+0000 7f30034ee640 1 Processor -- start 2024-05-09T05:06:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.346+0000 7f30034ee640 1 -- start start 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.347+0000 7f30034ee640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f2ffc0d16b0 0x7f2ffc0d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.347+0000 7f30034ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 0x7f2ffc0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.347+0000 7f30034ee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2ffc06f6d0 con 0x7f2ffc15ca90 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.347+0000 7f30034ee640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f2ffc0cd0e0 con 0x7f2ffc0d16b0 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.347+0000 7f30034ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2ffc0cc4d0 con 0x7f2ffc06f1b0 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f3000a89640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 msgr2=0x7f2ffc0ce740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f3000a89640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 0x7f2ffc0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f3001a8b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2ffc15ca90 0x7f2ffc15cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:59730/0 (socket says 172.21.15.1:59730) 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f3001a8b640 1 -- 172.21.15.1:0/3900462835 learned_addr learned my addr 172.21.15.1:0/3900462835 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f300128a640 1 --2- 172.21.15.1:0/3900462835 >> v2:172.21.15.196:3300/0 conn(0x7f2ffc0d16b0 0x7f2ffc0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f2ff3fff640 1 -- 172.21.15.1:0/3900462835 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2076865218 0 0) 0x7f2ffc06f6d0 con 0x7f2ffc15ca90 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f2ff3fff640 1 -- 172.21.15.1:0/3900462835 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2fe8003520 con 0x7f2ffc15ca90 2024-05-09T05:06:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f300128a640 1 -- 172.21.15.1:0/3900462835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 msgr2=0x7f2ffc0ce740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:06:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f300128a640 1 --2- 172.21.15.1:0/3900462835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 0x7f2ffc0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f2ff3fff640 1 -- 172.21.15.1:0/3900462835 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3664122963 0 0) 0x7f2fe8003520 con 0x7f2ffc15ca90 2024-05-09T05:06:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f300128a640 1 -- 172.21.15.1:0/3900462835 >> v1:172.21.15.1:6789/0 conn(0x7f2ffc15ca90 legacy=0x7f2ffc15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.348+0000 7f300128a640 1 -- 172.21.15.1:0/3900462835 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc0cc7b0 con 0x7f2ffc0d16b0 2024-05-09T05:06:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.349+0000 7f300128a640 1 --2- 172.21.15.1:0/3900462835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 0x7f2ffc0d03d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80097a0 tx=0x7f2ff802a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=18f2d15af752708c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.349+0000 7f2ff3fff640 1 -- 172.21.15.1:0/3900462835 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff8037070 con 0x7f2ffc0d16b0 2024-05-09T05:06:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.349+0000 7f2ff3fff640 1 -- 172.21.15.1:0/3900462835 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ff802ac50 con 0x7f2ffc0d16b0 2024-05-09T05:06:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.349+0000 7f2ff3fff640 1 -- 172.21.15.1:0/3900462835 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff4001a70 con 0x7f2ffc0d16b0 2024-05-09T05:06:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.350+0000 7f30034ee640 1 -- 172.21.15.1:0/3900462835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 msgr2=0x7f2ffc0d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.350+0000 7f30034ee640 1 --2- 172.21.15.1:0/3900462835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 0x7f2ffc0d03d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2ff80097a0 tx=0x7f2ff802a1c0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.350+0000 7f30034ee640 1 -- 172.21.15.1:0/3900462835 shutdown_connections 2024-05-09T05:06:48.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.350+0000 7f30034ee640 1 --2- 172.21.15.1:0/3900462835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 0x7f2ffc0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.350+0000 7f30034ee640 1 --2- 172.21.15.1:0/3900462835 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 0x7f2ffc0d03d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.350+0000 7f30034ee640 1 -- 172.21.15.1:0/3900462835 >> 172.21.15.1:0/3900462835 conn(0x7f2ffc0b3660 msgr2=0x7f2ffc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.350+0000 7f30034ee640 1 -- 172.21.15.1:0/3900462835 shutdown_connections 2024-05-09T05:06:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.350+0000 7f30034ee640 1 -- 172.21.15.1:0/3900462835 wait complete. 2024-05-09T05:06:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.352+0000 7f30034ee640 1 Processor -- start 2024-05-09T05:06:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.352+0000 7f30034ee640 1 -- start start 2024-05-09T05:06:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f30034ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 0x7f2ffc1f2340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f30034ee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 0x7f2ffc0b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f30034ee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ffc15ca90 0x7f2ffc151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f30034ee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc0d0910 con 0x7f2ffc15ca90 2024-05-09T05:06:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f30034ee640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2ffc0ccd70 con 0x7f2ffc0d16b0 2024-05-09T05:06:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f30034ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2ffc0cec80 con 0x7f2ffc06f1b0 2024-05-09T05:06:48.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f3001a8b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ffc15ca90 0x7f2ffc151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f3001a8b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ffc15ca90 0x7f2ffc151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53582/0 (socket says 172.21.15.1:53582) 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f300128a640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 msgr2=0x7f2ffc1f2340 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f3001a8b640 1 -- 172.21.15.1:0/2421301647 learned_addr learned my addr 172.21.15.1:0/2421301647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.353+0000 7f300128a640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 0x7f2ffc1f2340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f3000a89640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 0x7f2ffc0b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f3001a8b640 1 -- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 msgr2=0x7f2ffc1f2340 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f3001a8b640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 0x7f2ffc1f2340 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f3001a8b640 1 -- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 msgr2=0x7f2ffc0b47a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f3001a8b640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 0x7f2ffc0b47a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f3001a8b640 1 -- 172.21.15.1:0/2421301647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ffc1f2880 con 0x7f2ffc15ca90 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f3000a89640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 0x7f2ffc0b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f3001a8b640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ffc15ca90 0x7f2ffc151e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400cb30 tx=0x7f2ff400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f2ff1ffb640 1 -- 172.21.15.1:0/2421301647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff400ad60 con 0x7f2ffc15ca90 2024-05-09T05:06:48.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f2ff1ffb640 1 -- 172.21.15.1:0/2421301647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2ff4012400 con 0x7f2ffc15ca90 2024-05-09T05:06:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ffc0b7280 con 0x7f2ffc15ca90 2024-05-09T05:06:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f2ff1ffb640 1 -- 172.21.15.1:0/2421301647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ff40127f0 con 0x7f2ffc15ca90 2024-05-09T05:06:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.354+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ffc0b5a00 con 0x7f2ffc15ca90 2024-05-09T05:06:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.355+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ffc050870 con 0x7f2ffc15ca90 2024-05-09T05:06:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.357+0000 7f2ff1ffb640 1 -- 172.21.15.1:0/2421301647 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2ff4012990 con 0x7f2ffc15ca90 2024-05-09T05:06:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.357+0000 7f2ff1ffb640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fe0076280 0x7f2fe0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.357+0000 7f2ff1ffb640 1 -- 172.21.15.1:0/2421301647 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f2ff40988b0 con 0x7f2ffc15ca90 2024-05-09T05:06:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.358+0000 7f300128a640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fe0076280 0x7f2fe0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.358+0000 7f300128a640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fe0076280 0x7f2fe0078790 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8034000 tx=0x7f2ff802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.361+0000 7f2ff1ffb640 1 -- 172.21.15.1:0/2421301647 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2ff40620e0 con 0x7f2ffc15ca90 2024-05-09T05:06:48.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.627+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2ffc0b6580 con 0x7f2ffc15ca90 2024-05-09T05:06:48.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.627+0000 7f2ff1ffb640 1 -- 172.21.15.1:0/2421301647 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f2ff40619e0 con 0x7f2ffc15ca90 2024-05-09T05:06:48.628 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":66,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:06:48.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fe0076280 msgr2=0x7f2fe0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:48.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fe0076280 0x7f2fe0078790 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2ff8034000 tx=0x7f2ff802e000 comp rx=0 tx=0).stop 2024-05-09T05:06:48.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ffc15ca90 msgr2=0x7f2ffc151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:48.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ffc15ca90 0x7f2ffc151e60 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2ff400cb30 tx=0x7f2ff400a060 comp rx=0 tx=0).stop 2024-05-09T05:06:48.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 shutdown_connections 2024-05-09T05:06:48.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fe0076280 0x7f2fe0078790 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2ffc0d16b0 0x7f2ffc0b47a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2ffc15ca90 0x7f2ffc151e60 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 --2- 172.21.15.1:0/2421301647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2ffc06f1b0 0x7f2ffc1f2340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 >> 172.21.15.1:0/2421301647 conn(0x7f2ffc0b3660 msgr2=0x7f2ffc15bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 shutdown_connections 2024-05-09T05:06:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:48.629+0000 7f30034ee640 1 -- 172.21.15.1:0/2421301647 wait complete. 2024-05-09T05:06:51.642 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:06:51.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.770+0000 7f49d7be3640 1 Processor -- start 2024-05-09T05:06:51.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.770+0000 7f49d7be3640 1 -- start start 2024-05-09T05:06:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d7be3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d01548b0 0x7f49d00c2be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d7be3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f49d00c81d0 0x7f49d015e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d7be3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f49d0154cc0 con 0x7f49d00c85e0 2024-05-09T05:06:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d7be3640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f49d00c9680 con 0x7f49d00c81d0 2024-05-09T05:06:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d7be3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f49d00c98d0 con 0x7f49d01548b0 2024-05-09T05:06:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d597f640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d01548b0 msgr2=0x7f49d00c2be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:51.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d597f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d01548b0 0x7f49d00c2be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:06:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d517e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f49d00c81d0 0x7f49d015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d517e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f49d00c81d0 0x7f49d015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44360/0 (socket says 172.21.15.1:44360) 2024-05-09T05:06:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d517e640 1 -- 172.21.15.1:0/1432555957 learned_addr learned my addr 172.21.15.1:0/1432555957 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.772+0000 7f49d6180640 1 --1- 172.21.15.1:0/1432555957 >> v1:172.21.15.196:6789/0 conn(0x7f49d00c85e0 0x7f49d00ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:51166/0 (socket says 172.21.15.1:51166) 2024-05-09T05:06:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.773+0000 7f49d517e640 1 -- 172.21.15.1:0/1432555957 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d01548b0 msgr2=0x7f49d00c2be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:06:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.773+0000 7f49d517e640 1 --2- 172.21.15.1:0/1432555957 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d01548b0 0x7f49d00c2be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.773+0000 7f49d517e640 1 -- 172.21.15.1:0/1432555957 >> v1:172.21.15.196:6789/0 conn(0x7f49d00c85e0 legacy=0x7f49d00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:51.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.773+0000 7f49d517e640 1 -- 172.21.15.1:0/1432555957 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d00c9bb0 con 0x7f49d00c81d0 2024-05-09T05:06:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.773+0000 7f49d517e640 1 --2- 172.21.15.1:0/1432555957 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c81d0 0x7f49d015e240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009870 tx=0x7f49c402a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c30f23e0a1bf98f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d497d640 1 -- 172.21.15.1:0/1432555957 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4037070 con 0x7f49d00c81d0 2024-05-09T05:06:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d497d640 1 -- 172.21.15.1:0/1432555957 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c4032440 con 0x7f49d00c81d0 2024-05-09T05:06:51.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d497d640 1 -- 172.21.15.1:0/1432555957 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c4032810 con 0x7f49d00c81d0 2024-05-09T05:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d7be3640 1 -- 172.21.15.1:0/1432555957 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c81d0 msgr2=0x7f49d015e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d7be3640 1 --2- 172.21.15.1:0/1432555957 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c81d0 0x7f49d015e240 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f49c4009870 tx=0x7f49c402a270 comp rx=0 tx=0).stop 2024-05-09T05:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d7be3640 1 -- 172.21.15.1:0/1432555957 shutdown_connections 2024-05-09T05:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d7be3640 1 --2- 172.21.15.1:0/1432555957 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c81d0 0x7f49d015e240 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d7be3640 1 --2- 172.21.15.1:0/1432555957 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d01548b0 0x7f49d00c2be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d7be3640 1 -- 172.21.15.1:0/1432555957 >> 172.21.15.1:0/1432555957 conn(0x7f49d0070dd0 msgr2=0x7f49d00711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d7be3640 1 -- 172.21.15.1:0/1432555957 shutdown_connections 2024-05-09T05:06:51.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.774+0000 7f49d7be3640 1 -- 172.21.15.1:0/1432555957 wait complete. 2024-05-09T05:06:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.776+0000 7f49d7be3640 1 Processor -- start 2024-05-09T05:06:51.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.776+0000 7f49d7be3640 1 -- start start 2024-05-09T05:06:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.777+0000 7f49d7be3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d00c81d0 0x7f49d01f2ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.777+0000 7f49d7be3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c85e0 0x7f49d00cd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.777+0000 7f49d7be3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49d01548b0 0x7f49d00c5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.777+0000 7f49d7be3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f49d00c3120 con 0x7f49d01548b0 2024-05-09T05:06:51.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.777+0000 7f49d7be3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f49d015dd20 con 0x7f49d00c85e0 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.777+0000 7f49d7be3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f49d00cae00 con 0x7f49d00c81d0 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d597f640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d00c81d0 msgr2=0x7f49d01f2ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d597f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d00c81d0 0x7f49d01f2ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49d01548b0 0x7f49d00c5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49d01548b0 0x7f49d00c5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:59010/0 (socket says 172.21.15.1:59010) 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 -- 172.21.15.1:0/2399424997 learned_addr learned my addr 172.21.15.1:0/2399424997 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d517e640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c85e0 0x7f49d00cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 -- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d00c81d0 msgr2=0x7f49d01f2ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d00c81d0 0x7f49d01f2ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 -- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c85e0 msgr2=0x7f49d00cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c85e0 0x7f49d00cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 -- 172.21.15.1:0/2399424997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f49d01f31e0 con 0x7f49d01548b0 2024-05-09T05:06:51.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d6180640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49d01548b0 0x7f49d00c5f00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f49c800c760 tx=0x7f49c800cc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49ce7fc640 1 -- 172.21.15.1:0/2399424997 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c8017070 con 0x7f49d01548b0 2024-05-09T05:06:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49ce7fc640 1 -- 172.21.15.1:0/2399424997 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f49c800ce10 con 0x7f49d01548b0 2024-05-09T05:06:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f49d01f6190 con 0x7f49d01548b0 2024-05-09T05:06:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.778+0000 7f49ce7fc640 1 -- 172.21.15.1:0/2399424997 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f49c800a820 con 0x7f49d01548b0 2024-05-09T05:06:51.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.779+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f49d01f6750 con 0x7f49d01548b0 2024-05-09T05:06:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.780+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f49d0050870 con 0x7f49d01548b0 2024-05-09T05:06:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.780+0000 7f49ce7fc640 1 -- 172.21.15.1:0/2399424997 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f49c801a070 con 0x7f49d01548b0 2024-05-09T05:06:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.781+0000 7f49ce7fc640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f49bc0762b0 0x7f49bc0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.781+0000 7f49ce7fc640 1 -- 172.21.15.1:0/2399424997 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 5709+0+0 (secure 0 0 0) 0x7f49c8096b60 con 0x7f49d01548b0 2024-05-09T05:06:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.781+0000 7f49d597f640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f49bc0762b0 0x7f49bc0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:51.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.781+0000 7f49d597f640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f49bc0762b0 0x7f49bc0787c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f49c0006fd0 tx=0x7f49c0006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:06:51.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:51.786+0000 7f49ce7fc640 1 -- 172.21.15.1:0/2399424997 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f49c805fde0 con 0x7f49d01548b0 2024-05-09T05:06:52.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.050+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f49d00c8bc0 con 0x7f49d01548b0 2024-05-09T05:06:52.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.051+0000 7f49ce7fc640 1 -- 172.21.15.1:0/2399424997 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f49c8068420 con 0x7f49d01548b0 2024-05-09T05:06:52.052 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":8,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":69,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:06:52.054 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f49bc0762b0 msgr2=0x7f49bc0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f49bc0762b0 0x7f49bc0787c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f49c0006fd0 tx=0x7f49c0006d80 comp rx=0 tx=0).stop 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49d01548b0 msgr2=0x7f49d00c5f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49d01548b0 0x7f49d00c5f00 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f49c800c760 tx=0x7f49c800cc20 comp rx=0 tx=0).stop 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 shutdown_connections 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f49bc0762b0 0x7f49bc0787c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f49d00c85e0 0x7f49d00cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f49d01548b0 0x7f49d00c5f00 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 --2- 172.21.15.1:0/2399424997 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f49d00c81d0 0x7f49d01f2ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 >> 172.21.15.1:0/2399424997 conn(0x7f49d0070dd0 msgr2=0x7f49d014e910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 shutdown_connections 2024-05-09T05:06:52.055 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:52.053+0000 7f49d7be3640 1 -- 172.21.15.1:0/2399424997 wait complete. 2024-05-09T05:06:55.067 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:06:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.194+0000 7f972c015640 1 Processor -- start 2024-05-09T05:06:55.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.195+0000 7f972c015640 1 -- start start 2024-05-09T05:06:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.196+0000 7f972c015640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 0x7f9724156900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.196+0000 7f972c015640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f972406de20 con 0x7f97240cc560 2024-05-09T05:06:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.196+0000 7f972c015640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9724160c10 con 0x7f97240cc970 2024-05-09T05:06:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.196+0000 7f972c015640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f972415c9a0 con 0x7f972406efa0 2024-05-09T05:06:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.196+0000 7f9729db1640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 msgr2=0x7f9724156900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.196+0000 7f9729db1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 0x7f9724156900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:06:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.196+0000 7f97295b0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f97240cc560 0x7f972415fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45258/0 (socket says 172.21.15.1:45258) 2024-05-09T05:06:55.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.196+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 learned_addr learned my addr 172.21.15.1:0/3547401990 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:06:55.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.397+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 msgr2=0x7f9724156900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:55.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.397+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 0x7f9724156900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:06:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.798+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 msgr2=0x7f9724156900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:55.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:55.798+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 0x7f9724156900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:06:56.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:56.599+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 msgr2=0x7f9724156900 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:06:56.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:56.599+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 0x7f9724156900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:06:58.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.196+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 msgr2=0x7f9724156900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:06:58.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.196+0000 7f971b7fe640 1 --2- 172.21.15.1:0/3547401990 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f972406efa0 0x7f9724156900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:06:58.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.196+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 >> v1:172.21.15.196:6789/0 conn(0x7f97240cc970 legacy=0x7f9724160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:58.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.196+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 >> v1:172.21.15.1:6789/0 conn(0x7f97240cc560 legacy=0x7f972415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:06:58.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.196+0000 7f971b7fe640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:58.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.196+0000 7f971b7fe640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:06:58.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.196+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f972415c9a0 con 0x7f9708003710 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.196+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f9708008e40 con 0x7f97080062c0 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:06:58.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.197+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:06:58.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:58.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:58.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:06:58.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:06:58.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:06:58.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:06:58.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:06:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:06:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:06:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.398+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:06:58.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:58.799+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:06:59.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:59.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:06:59.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f9729db1640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:06:59.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:06:59.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:06:59.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:06:59.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:06:59.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:06:59.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:06:59.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:06:59.600+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:07:01.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:01.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 msgr2=0x7f9708005be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f972a5b2640 1 -- 172.21.15.1:0/3547401990 reap_dead start 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97240b51a0 con 0x7f97240cc970 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f972415c9a0 con 0x7f9708003710 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f971b7fe640 1 -- 172.21.15.1:0/3547401990 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f9708008e40 con 0x7f97080062c0 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.1:3300/0 conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.197+0000 7f9729db1640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.198+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 msgr2=0x7f9708008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.198+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.198+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 >> v1:172.21.15.196:6789/0 conn(0x7f97240cc970 legacy=0x7f970800b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.198+0000 7f97295b0640 1 -- 172.21.15.1:0/3547401990 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f970800db10 con 0x7f9708003710 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.198+0000 7f97295b0640 1 --2- 172.21.15.1:0/3547401990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9708003710 0x7f9708005be0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f972406d670 tx=0x7f971402ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ac4ecf42a2d9aea8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.199+0000 7f9728daf640 1 -- 172.21.15.1:0/3547401990 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714031070 con 0x7f9708003710 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.199+0000 7f9728daf640 1 -- 172.21.15.1:0/3547401990 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9714004ac0 con 0x7f9708003710 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.199+0000 7f9728daf640 1 -- 172.21.15.1:0/3547401990 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714005030 con 0x7f9708003710 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.200+0000 7f972c015640 1 -- 172.21.15.1:0/3547401990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9708003710 msgr2=0x7f9708005be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.200+0000 7f972c015640 1 --2- 172.21.15.1:0/3547401990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9708003710 0x7f9708005be0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f972406d670 tx=0x7f971402ad60 comp rx=0 tx=0).stop 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.200+0000 7f972c015640 1 -- 172.21.15.1:0/3547401990 shutdown_connections 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.200+0000 7f972c015640 1 --2- 172.21.15.1:0/3547401990 >> v2:172.21.15.196:3300/0 conn(0x7f97080062c0 0x7f9708008790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.200+0000 7f972c015640 1 --2- 172.21.15.1:0/3547401990 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9708003710 0x7f9708005be0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.200+0000 7f972c015640 1 -- 172.21.15.1:0/3547401990 >> 172.21.15.1:0/3547401990 conn(0x7f9724068d90 msgr2=0x7f97240d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.200+0000 7f972c015640 1 -- 172.21.15.1:0/3547401990 shutdown_connections 2024-05-09T05:07:01.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.200+0000 7f972c015640 1 -- 172.21.15.1:0/3547401990 wait complete. 2024-05-09T05:07:01.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.202+0000 7f972c015640 1 Processor -- start 2024-05-09T05:07:01.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.202+0000 7f972c015640 1 -- start start 2024-05-09T05:07:01.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f972c015640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9708003710 0x7f97241f32f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f972c015640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97080062c0 0x7f9724154060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f972c015640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97240cc970 0x7f9724148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f972c015640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f972415cc40 con 0x7f97080062c0 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f972c015640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f972406db40 con 0x7f97240cc970 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f972c015640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f972415cf20 con 0x7f9708003710 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f9729db1640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9708003710 msgr2=0x7f97241f32f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f9729db1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9708003710 0x7f97241f32f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f97295b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97080062c0 0x7f9724154060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f97295b0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97080062c0 0x7f9724154060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40550/0 (socket says 172.21.15.1:40550) 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f97295b0640 1 -- 172.21.15.1:0/1163910746 learned_addr learned my addr 172.21.15.1:0/1163910746 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f972a5b2640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97240cc970 0x7f9724148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f97295b0640 1 -- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9708003710 msgr2=0x7f97241f32f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f97295b0640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9708003710 0x7f97241f32f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f97295b0640 1 -- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97240cc970 msgr2=0x7f9724148b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f97295b0640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97240cc970 0x7f9724148b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.204+0000 7f97295b0640 1 -- 172.21.15.1:0/1163910746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97241f3830 con 0x7f97080062c0 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.205+0000 7f97295b0640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97080062c0 0x7f9724154060 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f971402b350 tx=0x7f9714001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.205+0000 7f971a7fc640 1 -- 172.21.15.1:0/1163910746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9714041030 con 0x7f97080062c0 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.205+0000 7f971a7fc640 1 -- 172.21.15.1:0/1163910746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f971402b660 con 0x7f97080062c0 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.205+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f972414cfa0 con 0x7f97080062c0 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.205+0000 7f971a7fc640 1 -- 172.21.15.1:0/1163910746 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f971402b830 con 0x7f97080062c0 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.205+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97241e7e50 con 0x7f97080062c0 2024-05-09T05:07:01.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.206+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9724050870 con 0x7f97080062c0 2024-05-09T05:07:01.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.207+0000 7f971a7fc640 1 -- 172.21.15.1:0/1163910746 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9714044070 con 0x7f97080062c0 2024-05-09T05:07:01.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.208+0000 7f971a7fc640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9708077330 0x7f9708079840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.208+0000 7f971a7fc640 1 -- 172.21.15.1:0/1163910746 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7f97140b7820 con 0x7f97080062c0 2024-05-09T05:07:01.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.209+0000 7f9729db1640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9708077330 0x7f9708079840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.209+0000 7f9729db1640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9708077330 0x7f9708079840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9720003930 tx=0x7f972000d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.214+0000 7f971a7fc640 1 -- 172.21.15.1:0/1163910746 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f971404b070 con 0x7f97080062c0 2024-05-09T05:07:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.496+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f97240c8cd0 con 0x7f97080062c0 2024-05-09T05:07:01.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.496+0000 7f971a7fc640 1 -- 172.21.15.1:0/1163910746 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f9714004a90 con 0x7f97080062c0 2024-05-09T05:07:01.497 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":12,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:07:01.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9708077330 msgr2=0x7f9708079840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9708077330 0x7f9708079840 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f9720003930 tx=0x7f972000d040 comp rx=0 tx=0).stop 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97080062c0 msgr2=0x7f9724154060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97080062c0 0x7f9724154060 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f971402b350 tx=0x7f9714001120 comp rx=0 tx=0).stop 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 shutdown_connections 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9708077330 0x7f9708079840 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97240cc970 0x7f9724148b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97080062c0 0x7f9724154060 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 --2- 172.21.15.1:0/1163910746 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9708003710 0x7f97241f32f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 >> 172.21.15.1:0/1163910746 conn(0x7f9724068d90 msgr2=0x7f972414ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 shutdown_connections 2024-05-09T05:07:01.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.499+0000 7f972c015640 1 -- 172.21.15.1:0/1163910746 wait complete. 2024-05-09T05:07:01.511 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:07:01.511 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:07:01.511 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:07:01.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.626+0000 7fcf5927e640 1 Processor -- start 2024-05-09T05:07:01.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.627+0000 7fcf5927e640 1 -- start start 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf5927e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcf54156c30 0x7fcf540cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf5927e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 0x7fcf5415fe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf5927e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf5415c9a0 con 0x7fcf540c6990 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf5927e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fcf54157040 con 0x7fcf54156c30 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf5927e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcf5415d570 con 0x7fcf540ccd80 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf52ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcf540c6990 0x7fcf540cb010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53238/0 (socket says 172.21.15.1:53238) 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf52ffd640 1 -- 172.21.15.1:0/1281659511 learned_addr learned my addr 172.21.15.1:0/1281659511 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf537fe640 1 -- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 msgr2=0x7fcf5415fe80 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf537fe640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 0x7fcf5415fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf527fc640 1 --2- 172.21.15.1:0/1281659511 >> v2:172.21.15.196:3300/0 conn(0x7fcf54156c30 0x7fcf540cc840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1163643666 0 0) 0x7fcf5415c9a0 con 0x7fcf540c6990 2024-05-09T05:07:01.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.628+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcf40003520 con 0x7fcf540c6990 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1782546656 0 0) 0x7fcf40003520 con 0x7fcf540c6990 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 msgr2=0x7fcf5415fe80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 0x7fcf5415fe80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 >> v2:172.21.15.196:3300/0 conn(0x7fcf54156c30 msgr2=0x7fcf540cc840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 --2- 172.21.15.1:0/1281659511 >> v2:172.21.15.196:3300/0 conn(0x7fcf54156c30 0x7fcf540cc840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf5415cc40 con 0x7fcf540c6990 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fcf44001a70 con 0x7fcf540c6990 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 >> v1:172.21.15.1:6789/0 conn(0x7fcf540c6990 legacy=0x7fcf540cb010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf40003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf40004800 0x7fcf40004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 0x7fcf40005240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcf54157040 con 0x7fcf400039b0 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5415d570 con 0x7fcf40004800 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcf4000bc60 con 0x7fcf540ccd80 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf52ffd640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf40003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf537fe640 1 -- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 msgr2=0x7fcf40005240 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf537fe640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 0x7fcf40005240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf527fc640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf40004800 0x7fcf40004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf52ffd640 1 -- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 msgr2=0x7fcf40005240 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf52ffd640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf540ccd80 0x7fcf40005240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf52ffd640 1 -- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf40004800 msgr2=0x7fcf40004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf52ffd640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf40004800 0x7fcf40004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.629+0000 7fcf52ffd640 1 -- 172.21.15.1:0/1281659511 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf40003520 con 0x7fcf400039b0 2024-05-09T05:07:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.630+0000 7fcf52ffd640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf40003e20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400a8b0 tx=0x7fcf40005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6726bfa9254893e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.630+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4400a4d0 con 0x7fcf400039b0 2024-05-09T05:07:01.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.630+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf4402f070 con 0x7fcf400039b0 2024-05-09T05:07:01.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.630+0000 7fcf51ffb640 1 -- 172.21.15.1:0/1281659511 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4402bb60 con 0x7fcf400039b0 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.631+0000 7fcf5927e640 1 -- 172.21.15.1:0/1281659511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 msgr2=0x7fcf40003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.631+0000 7fcf5927e640 1 --2- 172.21.15.1:0/1281659511 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf40003e20 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400a8b0 tx=0x7fcf40005830 comp rx=0 tx=0).stop 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.631+0000 7fcf537fe640 1 -- 172.21.15.1:0/1281659511 reap_dead start 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.631+0000 7fcf5927e640 1 -- 172.21.15.1:0/1281659511 shutdown_connections 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.631+0000 7fcf5927e640 1 -- 172.21.15.1:0/1281659511 >> 172.21.15.1:0/1281659511 conn(0x7fcf54068830 msgr2=0x7fcf54068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.631+0000 7fcf5927e640 1 -- 172.21.15.1:0/1281659511 shutdown_connections 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.631+0000 7fcf5927e640 1 -- 172.21.15.1:0/1281659511 wait complete. 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.633+0000 7fcf5927e640 1 Processor -- start 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.633+0000 7fcf5927e640 1 -- start start 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf5927e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf541f2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf5927e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf5415da50 0x7fcf5415f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf5927e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf5406ea20 0x7fcf5406ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf5927e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcf5415d200 con 0x7fcf400039b0 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf5927e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcf540cb720 con 0x7fcf5415da50 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf5927e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcf540cb980 con 0x7fcf5406ea20 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf52ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf541f2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf52ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf541f2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40570/0 (socket says 172.21.15.1:40570) 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf52ffd640 1 -- 172.21.15.1:0/375829444 learned_addr learned my addr 172.21.15.1:0/375829444 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf537fe640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf5406ea20 msgr2=0x7fcf5406ee30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf537fe640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf5406ea20 0x7fcf5406ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf527fc640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf5415da50 0x7fcf5415f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.634+0000 7fcf52ffd640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf5406ea20 msgr2=0x7fcf5406ee30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf52ffd640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf5406ea20 0x7fcf5406ee30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf52ffd640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf5415da50 msgr2=0x7fcf5415f8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf52ffd640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf5415da50 0x7fcf5415f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf52ffd640 1 -- 172.21.15.1:0/375829444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf54160460 con 0x7fcf400039b0 2024-05-09T05:07:01.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf52ffd640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf541f2510 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400a910 tx=0x7fcf44001de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44002020 con 0x7fcf400039b0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf44004b60 con 0x7fcf400039b0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf44005020 con 0x7fcf400039b0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf541f6530 con 0x7fcf400039b0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.635+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf540ce6f0 con 0x7fcf400039b0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.636+0000 7fcf5927e640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf541f2ab0 0x7fcf54151240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.636+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcf54160210 con 0x7fcf541f2ab0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.636+0000 7fcf527fc640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf541f2ab0 0x7fcf54151240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.636+0000 7fcf527fc640 1 -- 172.21.15.1:0/375829444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf540cec10 con 0x7fcf541f2ab0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.637+0000 7fcf527fc640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf541f2ab0 0x7fcf54151240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cf790 tx=0x7fcf3c012d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.637+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf3c01c020 con 0x7fcf541f2ab0 2024-05-09T05:07:01.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.637+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcf3c004240 con 0x7fcf541f2ab0 2024-05-09T05:07:01.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.637+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf541f2ab0 msgr2=0x7fcf54151240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.637+0000 7fcf4b7fe640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf541f2ab0 0x7fcf54151240 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cf790 tx=0x7fcf3c012d00 comp rx=0 tx=0).stop 2024-05-09T05:07:01.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.637+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf541f2ab0 msgr2=0x7fcf54151240 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:01.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.637+0000 7fcf4b7fe640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf541f2ab0 0x7fcf54151240 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.637+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcf440051c0 con 0x7fcf400039b0 2024-05-09T05:07:01.641 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:07:01.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.638+0000 7fcf4b7fe640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf34075c80 0x7fcf34078190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.638+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(22..22 src has 1..22) ==== 5825+0+0 (secure 0 0 0) 0x7fcf440b7cd0 con 0x7fcf400039b0 2024-05-09T05:07:01.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.638+0000 7fcf527fc640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf34075c80 0x7fcf34078190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.638+0000 7fcf527fc640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf34075c80 0x7fcf34078190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cf790 tx=0x7fcf3c023040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.645 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:07:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.644+0000 7fcf5927e640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf54151780 0x7fcf54151b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.644+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcf541520d0 con 0x7fcf54151780 2024-05-09T05:07:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.644+0000 7fcf537fe640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf54151780 0x7fcf54151b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.644+0000 7fcf537fe640 1 -- 172.21.15.1:0/375829444 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf54050870 con 0x7fcf54151780 2024-05-09T05:07:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.644+0000 7fcf537fe640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf54151780 0x7fcf54151b90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcf54074eb0 tx=0x7fcf4c00b7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf4c003bd0 con 0x7fcf54151780 2024-05-09T05:07:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5051 (secure 0 0 0) 0x7fcf4c003bd0 con 0x7fcf54151780 2024-05-09T05:07:01.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf54151780 msgr2=0x7fcf54151b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf4b7fe640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf54151780 0x7fcf54151b90 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fcf54074eb0 tx=0x7fcf4c00b7a0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf4b7fe640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf54151780 msgr2=0x7fcf54151b90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf4b7fe640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf54151780 0x7fcf54151b90 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 12, 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 85769, 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:01.647 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:01.648 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:01.649 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:01.650 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:01.651 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:01.653 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:01.655 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:01.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:01.658 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.659 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.660 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:01.660 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf34075c80 msgr2=0x7fcf34078190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf34075c80 0x7fcf34078190 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcf540cf790 tx=0x7fcf3c023040 comp rx=0 tx=0).stop 2024-05-09T05:07:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 msgr2=0x7fcf541f2510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf537fe640 1 -- 172.21.15.1:0/375829444 reap_dead start 2024-05-09T05:07:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf541f2510 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fcf4400a910 tx=0x7fcf44001de0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 shutdown_connections 2024-05-09T05:07:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 --2- 172.21.15.1:0/375829444 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf400039b0 0x7fcf541f2510 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 >> 172.21.15.1:0/375829444 conn(0x7fcf54068830 msgr2=0x7fcf5415a450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:01.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 shutdown_connections 2024-05-09T05:07:01.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.645+0000 7fcf5927e640 1 -- 172.21.15.1:0/375829444 wait complete. 2024-05-09T05:07:01.662 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:07:01.662 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:07:01.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:07:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.853+0000 7fa712bdc640 1 Processor -- start 2024-05-09T05:07:01.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.853+0000 7fa712bdc640 1 -- start start 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa712bdc640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa70c0cf360 0x7fa70c0b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa712bdc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa70c15c9a0 0x7fa70c15cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa712bdc640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa70c0ce5c0 con 0x7fa70c0b4570 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa712bdc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa70c15d4c0 con 0x7fa70c15c9a0 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa712bdc640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fa70c15d7a0 con 0x7fa70c0cf360 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa711179640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa70c15c9a0 0x7fa70c15cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa711179640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa70c15c9a0 0x7fa70c15cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40598/0 (socket says 172.21.15.1:40598) 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa711179640 1 -- 172.21.15.1:0/2656289956 learned_addr learned my addr 172.21.15.1:0/2656289956 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa710978640 1 --2- 172.21.15.1:0/2656289956 >> v2:172.21.15.196:3300/0 conn(0x7fa70c0cf360 0x7fa70c0b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa711179640 1 -- 172.21.15.1:0/2656289956 >> v2:172.21.15.196:3300/0 conn(0x7fa70c0cf360 msgr2=0x7fa70c0b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa711179640 1 --2- 172.21.15.1:0/2656289956 >> v2:172.21.15.196:3300/0 conn(0x7fa70c0cf360 0x7fa70c0b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa711179640 1 -- 172.21.15.1:0/2656289956 >> v1:172.21.15.196:6789/0 conn(0x7fa70c0b4570 legacy=0x7fa70c156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.855+0000 7fa711179640 1 -- 172.21.15.1:0/2656289956 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70c0ce880 con 0x7fa70c15c9a0 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.856+0000 7fa711179640 1 --2- 172.21.15.1:0/2656289956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c15c9a0 0x7fa70c15cdb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa70c08f300 tx=0x7fa6f402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d02e01cdbbcac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.856+0000 7fa70b7fe640 1 -- 172.21.15.1:0/2656289956 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4037070 con 0x7fa70c15c9a0 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.856+0000 7fa70b7fe640 1 -- 172.21.15.1:0/2656289956 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa6f402ae50 con 0x7fa70c15c9a0 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.856+0000 7fa70b7fe640 1 -- 172.21.15.1:0/2656289956 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6f4032860 con 0x7fa70c15c9a0 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.857+0000 7fa712bdc640 1 -- 172.21.15.1:0/2656289956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c15c9a0 msgr2=0x7fa70c15cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.857+0000 7fa712bdc640 1 --2- 172.21.15.1:0/2656289956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c15c9a0 0x7fa70c15cdb0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fa70c08f300 tx=0x7fa6f402a300 comp rx=0 tx=0).stop 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.857+0000 7fa712bdc640 1 -- 172.21.15.1:0/2656289956 shutdown_connections 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.857+0000 7fa712bdc640 1 --2- 172.21.15.1:0/2656289956 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c15c9a0 0x7fa70c15cdb0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.857+0000 7fa712bdc640 1 --2- 172.21.15.1:0/2656289956 >> v2:172.21.15.196:3300/0 conn(0x7fa70c0cf360 0x7fa70c0b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.857+0000 7fa712bdc640 1 -- 172.21.15.1:0/2656289956 >> 172.21.15.1:0/2656289956 conn(0x7fa70c06de10 msgr2=0x7fa70c06e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.857+0000 7fa712bdc640 1 -- 172.21.15.1:0/2656289956 shutdown_connections 2024-05-09T05:07:01.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.857+0000 7fa712bdc640 1 -- 172.21.15.1:0/2656289956 wait complete. 2024-05-09T05:07:01.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.859+0000 7fa712bdc640 1 Processor -- start 2024-05-09T05:07:01.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.859+0000 7fa712bdc640 1 -- start start 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa712bdc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c0b4570 0x7fa70c0c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa712bdc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa70c15c9a0 0x7fa70c153240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa712bdc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c1f58b0 0x7fa70c0cb2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa712bdc640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c0b5090 con 0x7fa70c0b4570 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa712bdc640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c0cee40 con 0x7fa70c1f58b0 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa712bdc640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa70c0cd3f0 con 0x7fa70c15c9a0 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa710978640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c0b4570 0x7fa70c0c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa710978640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c0b4570 0x7fa70c0c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40600/0 (socket says 172.21.15.1:40600) 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa710978640 1 -- 172.21.15.1:0/1578727106 learned_addr learned my addr 172.21.15.1:0/1578727106 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa70bfff640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa70c15c9a0 msgr2=0x7fa70c153240 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:07:01.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa70bfff640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa70c15c9a0 0x7fa70c153240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa711179640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c1f58b0 0x7fa70c0cb2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa710978640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa70c15c9a0 msgr2=0x7fa70c153240 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa710978640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa70c15c9a0 0x7fa70c153240 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa710978640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c1f58b0 msgr2=0x7fa70c0cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa710978640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c1f58b0 0x7fa70c0cb2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.860+0000 7fa710978640 1 -- 172.21.15.1:0/1578727106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa70c0c8220 con 0x7fa70c0b4570 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.861+0000 7fa710978640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c0b4570 0x7fa70c0c7ce0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa70400e870 tx=0x7fa70400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.861+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa704017070 con 0x7fa70c0b4570 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.861+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa70400ee60 con 0x7fa70c0b4570 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.861+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa70c1f2340 con 0x7fa70c0b4570 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.861+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa70400cb00 con 0x7fa70c0b4570 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.861+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa70c1f2bd0 con 0x7fa70c0b4570 2024-05-09T05:07:01.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.862+0000 7fa712bdc640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0cb7f0 0x7fa70c1f2e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.862+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c0c86f0 con 0x7fa70c0cb7f0 2024-05-09T05:07:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.862+0000 7fa70bfff640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0cb7f0 0x7fa70c1f2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.862+0000 7fa70bfff640 1 -- 172.21.15.1:0/1578727106 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa70c050870 con 0x7fa70c0cb7f0 2024-05-09T05:07:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.862+0000 7fa70bfff640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0cb7f0 0x7fa70c1f2e10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa70c1e9b40 tx=0x7fa6fc00caa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc017070 con 0x7fa70c0cb7f0 2024-05-09T05:07:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa70401a070 con 0x7fa70c0b4570 2024-05-09T05:07:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa6ec076200 0x7fa6ec078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa704097820 con 0x7fa70c0b4570 2024-05-09T05:07:01.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa6fc007070 con 0x7fa70c0cb7f0 2024-05-09T05:07:01.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0cb7f0 msgr2=0x7fa70c1f2e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0cb7f0 0x7fa70c1f2e10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa70c1e9b40 tx=0x7fa6fc00caa0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0cb7f0 msgr2=0x7fa70c1f2e10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:01.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa7097fa640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0cb7f0 0x7fa70c1f2e10 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa70c1e9b40 tx=0x7fa6fc00caa0 comp rx=0 tx=0).stop 2024-05-09T05:07:01.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.863+0000 7fa711179640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa6ec076200 0x7fa6ec078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.864+0000 7fa711179640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa6ec076200 0x7fa6ec078710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4034000 tx=0x7fa6f402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.870+0000 7fa712bdc640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0c55b0 0x7fa70c0c59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:01.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.870+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa70c1f26a0 con 0x7fa70c0c55b0 2024-05-09T05:07:01.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.870+0000 7fa70bfff640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0c55b0 0x7fa70c0c59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.871+0000 7fa70bfff640 1 -- 172.21.15.1:0/1578727106 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa70c15d0d0 con 0x7fa70c0c55b0 2024-05-09T05:07:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.871+0000 7fa70bfff640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0c55b0 0x7fa70c0c59c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc014000 tx=0x7fa6fc001250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.871+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6fc01add0 con 0x7fa70c0c55b0 2024-05-09T05:07:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.871+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7fa6fc024430 con 0x7fa70c0c55b0 2024-05-09T05:07:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.871+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0c55b0 msgr2=0x7fa70c0c59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.871+0000 7fa7097fa640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0c55b0 0x7fa70c0c59c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc014000 tx=0x7fa6fc001250 comp rx=0 tx=0).stop 2024-05-09T05:07:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.871+0000 7fa7097fa640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0c55b0 msgr2=0x7fa70c0c59c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:01.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.871+0000 7fa7097fa640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa70c0c55b0 0x7fa70c0c59c0 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa6fc014000 tx=0x7fa6fc001250 comp rx=0 tx=0).stop 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 12, 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 85962, 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:01.873 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.874 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.875 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:01.876 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:01.877 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:01.880 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:01.881 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.882 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.883 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.884 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:01.885 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa6ec076200 msgr2=0x7fa6ec078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa712bdc640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa6ec076200 0x7fa6ec078710 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fa6f4034000 tx=0x7fa6f402e000 comp rx=0 tx=0).stop 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c0b4570 msgr2=0x7fa70c0c7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa712bdc640 1 --2- 172.21.15.1:0/1578727106 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa70c0b4570 0x7fa70c0c7ce0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa70400e870 tx=0x7fa70400ed30 comp rx=0 tx=0).stop 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa711179640 1 -- 172.21.15.1:0/1578727106 reap_dead start 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa711179640 1 -- 172.21.15.1:0/1578727106 reap_dead start 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 shutdown_connections 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 >> 172.21.15.1:0/1578727106 conn(0x7fa70c06de10 msgr2=0x7fa70c0cfec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 shutdown_connections 2024-05-09T05:07:01.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:01.872+0000 7fa712bdc640 1 -- 172.21.15.1:0/1578727106 wait complete. 2024-05-09T05:07:01.888 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:07:01.888 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:07:21.888 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:07:21.888 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-05-09T05:07:21.888 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T05:07:21.889 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T05:07:21.892 INFO:tasks.ceph.mon.c:Started 2024-05-09T05:07:21.892 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:07:21.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:07:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.020+0000 7ffa72813640 1 Processor -- start 2024-05-09T05:07:22.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.020+0000 7ffa72813640 1 -- start start 2024-05-09T05:07:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa72813640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 0x7ffa6c0c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa72813640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6c06de20 con 0x7ffa6c0c7870 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa72813640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa6c15c9a0 con 0x7ffa6c06efa0 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa72813640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6c156d10 con 0x7ffa6c156900 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa70db0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffa6c0c7870 0x7ffa6c0c7c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33018/0 (socket says 172.21.15.1:33018) 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa70db0640 1 -- 172.21.15.1:0/1542118274 learned_addr learned my addr 172.21.15.1:0/1542118274 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6b7fe640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 0x7ffa6c0c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6b7fe640 1 -- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 msgr2=0x7ffa6c0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6b7fe640 1 -- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 msgr2=0x7ffa6c0c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6b7fe640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 0x7ffa6c0c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6b7fe640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 0x7ffa6c0c7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 843720542 0 0) 0x7ffa6c06de20 con 0x7ffa6c0c7870 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa54003520 con 0x7ffa6c0c7870 2024-05-09T05:07:22.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2287466922 0 0) 0x7ffa6c15c9a0 con 0x7ffa6c06efa0 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa6c06de20 con 0x7ffa6c06efa0 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 986652123 0 0) 0x7ffa54003520 con 0x7ffa6c0c7870 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 msgr2=0x7ffa6c0c7330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 0x7ffa6c0c7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 >> v1:172.21.15.196:6789/0 conn(0x7ffa6c06efa0 legacy=0x7ffa6c0cc560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c0b51a0 con 0x7ffa6c0c7870 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7ffa64001a70 con 0x7ffa6c0c7870 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.022+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 >> v1:172.21.15.1:6789/0 conn(0x7ffa6c0c7870 legacy=0x7ffa6c0c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6affd640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa54004c50 0x7ffa54005080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6affd640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 0x7ffa540037a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6affd640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa54003e00 0x7ffa54004230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c156d10 con 0x7ffa54004c50 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5400bc20 con 0x7ffa54003e00 2024-05-09T05:07:22.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffa5400be20 con 0x7ffa6c156900 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6bfff640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa54004c50 0x7ffa54005080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6b7fe640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 0x7ffa540037a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa70db0640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa54003e00 0x7ffa54004230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6bfff640 1 -- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 msgr2=0x7ffa540037a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6bfff640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa6c156900 0x7ffa540037a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6bfff640 1 -- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa54003e00 msgr2=0x7ffa54004230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6bfff640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa54003e00 0x7ffa54004230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.023+0000 7ffa6bfff640 1 -- 172.21.15.1:0/1542118274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa54003520 con 0x7ffa54004c50 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.024+0000 7ffa6bfff640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa54004c50 0x7ffa54005080 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c007ef0 tx=0x7ffa54005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cefdfe7fcebc694 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.024+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c007b10 con 0x7ffa54004c50 2024-05-09T05:07:22.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.024+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa5c005270 con 0x7ffa54004c50 2024-05-09T05:07:22.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.024+0000 7ffa6affd640 1 -- 172.21.15.1:0/1542118274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa5c003830 con 0x7ffa54004c50 2024-05-09T05:07:22.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.025+0000 7ffa72813640 1 -- 172.21.15.1:0/1542118274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa54004c50 msgr2=0x7ffa54005080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.025+0000 7ffa72813640 1 --2- 172.21.15.1:0/1542118274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa54004c50 0x7ffa54005080 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c007ef0 tx=0x7ffa54005830 comp rx=0 tx=0).stop 2024-05-09T05:07:22.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.025+0000 7ffa70db0640 1 -- 172.21.15.1:0/1542118274 reap_dead start 2024-05-09T05:07:22.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.025+0000 7ffa72813640 1 -- 172.21.15.1:0/1542118274 shutdown_connections 2024-05-09T05:07:22.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.025+0000 7ffa72813640 1 -- 172.21.15.1:0/1542118274 >> 172.21.15.1:0/1542118274 conn(0x7ffa6c068d90 msgr2=0x7ffa6c0d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:22.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.025+0000 7ffa72813640 1 -- 172.21.15.1:0/1542118274 shutdown_connections 2024-05-09T05:07:22.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.025+0000 7ffa72813640 1 -- 172.21.15.1:0/1542118274 wait complete. 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.026+0000 7ffa72813640 1 Processor -- start 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.027+0000 7ffa72813640 1 -- start start 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa72813640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa54004c50 0x7ffa6c0b56f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa72813640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa6c1f3710 0x7ffa6c1f5fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa72813640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c159dd0 0x7ffa6c14b380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa72813640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c15cc40 con 0x7ffa6c159dd0 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa72813640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffa6c06db40 con 0x7ffa6c1f3710 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa72813640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffa6c15cf20 con 0x7ffa54004c50 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa70db0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c159dd0 0x7ffa6c14b380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa70db0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c159dd0 0x7ffa6c14b380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60816/0 (socket says 172.21.15.1:60816) 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa70db0640 1 -- 172.21.15.1:0/3768805036 learned_addr learned my addr 172.21.15.1:0/3768805036 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa6b7fe640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa6c1f3710 0x7ffa6c1f5fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa6bfff640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa54004c50 0x7ffa6c0b56f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa70db0640 1 -- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa54004c50 msgr2=0x7ffa6c0b56f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa70db0640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa54004c50 0x7ffa6c0b56f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa70db0640 1 -- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa6c1f3710 msgr2=0x7ffa6c1f5fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa70db0640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa6c1f3710 0x7ffa6c1f5fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.028+0000 7ffa70db0640 1 -- 172.21.15.1:0/3768805036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa6c06e950 con 0x7ffa6c159dd0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.029+0000 7ffa70db0640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c159dd0 0x7ffa6c14b380 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffa64001330 tx=0x7ffa64007bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.029+0000 7ffa68ff9640 1 -- 172.21.15.1:0/3768805036 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6400ac30 con 0x7ffa6c159dd0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.029+0000 7ffa68ff9640 1 -- 172.21.15.1:0/3768805036 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa64018070 con 0x7ffa6c159dd0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.029+0000 7ffa68ff9640 1 -- 172.21.15.1:0/3768805036 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa640133f0 con 0x7ffa6c159dd0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.029+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa6c1f6c60 con 0x7ffa6c159dd0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.029+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa6c1f2f10 con 0x7ffa6c159dd0 2024-05-09T05:07:22.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.030+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa6c1f69d0 con 0x7ffa6c159dd0 2024-05-09T05:07:22.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.031+0000 7ffa68ff9640 1 -- 172.21.15.1:0/3768805036 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa640135b0 con 0x7ffa6c159dd0 2024-05-09T05:07:22.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.031+0000 7ffa68ff9640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa580762d0 0x7ffa580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.031+0000 7ffa68ff9640 1 -- 172.21.15.1:0/3768805036 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ffa64097e00 con 0x7ffa6c159dd0 2024-05-09T05:07:22.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.031+0000 7ffa6bfff640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa580762d0 0x7ffa580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.032+0000 7ffa6bfff640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa580762d0 0x7ffa580787e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c007ec0 tx=0x7ffa5c009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.036+0000 7ffa68ff9640 1 -- 172.21.15.1:0/3768805036 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffa64061650 con 0x7ffa6c159dd0 2024-05-09T05:07:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.305+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffa6c1f2a40 con 0x7ffa6c159dd0 2024-05-09T05:07:22.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.306+0000 7ffa68ff9640 1 -- 172.21.15.1:0/3768805036 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7ffa64060f50 con 0x7ffa6c159dd0 2024-05-09T05:07:22.306 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa580762d0 msgr2=0x7ffa580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa580762d0 0x7ffa580787e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c007ec0 tx=0x7ffa5c009030 comp rx=0 tx=0).stop 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c159dd0 msgr2=0x7ffa6c14b380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c159dd0 0x7ffa6c14b380 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7ffa64001330 tx=0x7ffa64007bd0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 shutdown_connections 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa580762d0 0x7ffa580787e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa6c1f3710 0x7ffa6c1f5fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa6c159dd0 0x7ffa6c14b380 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 --2- 172.21.15.1:0/3768805036 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa54004c50 0x7ffa6c0b56f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 >> 172.21.15.1:0/3768805036 conn(0x7ffa6c068d90 msgr2=0x7ffa6c14e8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 shutdown_connections 2024-05-09T05:07:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.308+0000 7ffa72813640 1 -- 172.21.15.1:0/3768805036 wait complete. 2024-05-09T05:07:22.320 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:07:22.320 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:07:22.320 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:07:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.434+0000 7f5cc9013640 1 Processor -- start 2024-05-09T05:07:22.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.434+0000 7f5cc9013640 1 -- start start 2024-05-09T05:07:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc9013640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 0x7f5cc415ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc9013640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5cc40cdaf0 0x7f5cc415d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc9013640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5cc4160160 con 0x7f5cc4156c20 2024-05-09T05:07:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc9013640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc4160ce0 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc9013640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc4160440 con 0x7f5cc40cd610 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc2d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5cc4156c20 0x7f5cc40cea20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33024/0 (socket says 172.21.15.1:33024) 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc2d9d640 1 -- 172.21.15.1:0/3026176670 learned_addr learned my addr 172.21.15.1:0/3026176670 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc359e640 1 --2- 172.21.15.1:0/3026176670 >> v2:172.21.15.196:3300/0 conn(0x7f5cc40cdaf0 0x7f5cc415d350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc259c640 1 --2- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 0x7f5cc415ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc1d9b640 1 -- 172.21.15.1:0/3026176670 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1313684856 0 0) 0x7f5cc4160160 con 0x7f5cc4156c20 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc1d9b640 1 -- 172.21.15.1:0/3026176670 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5cb0003520 con 0x7f5cc4156c20 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc259c640 1 -- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 msgr2=0x7f5cc415ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc259c640 1 -- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 msgr2=0x7f5cc415ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc259c640 1 --2- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 0x7f5cc415ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc259c640 1 --2- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 0x7f5cc415ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc359e640 1 -- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 msgr2=0x7f5cc415ce10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc359e640 1 --2- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 0x7f5cc415ce10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc1d9b640 1 -- 172.21.15.1:0/3026176670 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1117763756 0 0) 0x7f5cb0003520 con 0x7f5cc4156c20 2024-05-09T05:07:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc359e640 1 -- 172.21.15.1:0/3026176670 >> v1:172.21.15.1:6789/0 conn(0x7f5cc4156c20 legacy=0x7f5cc40cea20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.436+0000 7f5cc359e640 1 -- 172.21.15.1:0/3026176670 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc4160720 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.437+0000 7f5cc359e640 1 --2- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc40cdaf0 0x7f5cc415d350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0097a0 tx=0x7f5cbc02a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd99a6de2ed18b9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.437+0000 7f5cc1d9b640 1 -- 172.21.15.1:0/3026176670 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc037070 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.437+0000 7f5cc1d9b640 1 -- 172.21.15.1:0/3026176670 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cbc02ac30 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.437+0000 7f5cc1d9b640 1 -- 172.21.15.1:0/3026176670 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb4001a70 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.438+0000 7f5cc9013640 1 -- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc40cdaf0 msgr2=0x7f5cc415d350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.438+0000 7f5cc9013640 1 --2- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc40cdaf0 0x7f5cc415d350 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f5cbc0097a0 tx=0x7f5cbc02a1a0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.438+0000 7f5cc9013640 1 -- 172.21.15.1:0/3026176670 shutdown_connections 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.438+0000 7f5cc9013640 1 --2- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc40cdaf0 0x7f5cc415d350 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.438+0000 7f5cc9013640 1 --2- 172.21.15.1:0/3026176670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc40cd610 0x7f5cc415ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.438+0000 7f5cc9013640 1 -- 172.21.15.1:0/3026176670 >> 172.21.15.1:0/3026176670 conn(0x7f5cc4068830 msgr2=0x7f5cc4068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:22.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.438+0000 7f5cc9013640 1 -- 172.21.15.1:0/3026176670 shutdown_connections 2024-05-09T05:07:22.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.438+0000 7f5cc9013640 1 -- 172.21.15.1:0/3026176670 wait complete. 2024-05-09T05:07:22.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.440+0000 7f5cc9013640 1 Processor -- start 2024-05-09T05:07:22.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.440+0000 7f5cc9013640 1 -- start start 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.441+0000 7f5cc9013640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc40cd610 0x7f5cc40d14a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.441+0000 7f5cc9013640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc40cdaf0 0x7f5cc40cfed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.441+0000 7f5cc9013640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc41f40e0 0x7f5cc414e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.441+0000 7f5cc9013640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc40ce2a0 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.441+0000 7f5cc9013640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc415c8f0 con 0x7f5cc40cd610 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.441+0000 7f5cc9013640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc40cf130 con 0x7f5cc41f40e0 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc40cdaf0 0x7f5cc40cfed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc40cdaf0 0x7f5cc40cfed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60824/0 (socket says 172.21.15.1:60824) 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 -- 172.21.15.1:0/2525372549 learned_addr learned my addr 172.21.15.1:0/2525372549 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc359e640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc41f40e0 0x7f5cc414e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc2d9d640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc40cd610 0x7f5cc40d14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc359e640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc41f40e0 msgr2=0x7f5cc414e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc359e640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc41f40e0 msgr2=0x7f5cc414e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc359e640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc41f40e0 0x7f5cc414e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc359e640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc41f40e0 0x7f5cc414e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc41f40e0 msgr2=0x7f5cc414e6a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc41f40e0 0x7f5cc414e6a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc40cd610 msgr2=0x7f5cc40d14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc40cd610 0x7f5cc40d14a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 -- 172.21.15.1:0/2525372549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc40d1e30 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc259c640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc40cdaf0 0x7f5cc40cfed0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00d830 tx=0x7f5cac00dcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac00b6c0 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc40d2110 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cac019070 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cac00bac0 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.442+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc40d26d0 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.443+0000 7f5cc9013640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc414ebe0 0x7f5cc41f5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.443+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc41f4af0 con 0x7f5cc414ebe0 2024-05-09T05:07:22.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.443+0000 7f5cc2d9d640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc414ebe0 0x7f5cc41f5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.444+0000 7f5cc2d9d640 1 -- 172.21.15.1:0/2525372549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cc40ce420 con 0x7f5cc414ebe0 2024-05-09T05:07:22.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.444+0000 7f5cc2d9d640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc414ebe0 0x7f5cc41f5f30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4001b90 tx=0x7f5cb400d520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.444+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb400de50 con 0x7f5cc414ebe0 2024-05-09T05:07:22.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.444+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5cb4018070 con 0x7f5cc414ebe0 2024-05-09T05:07:22.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.444+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc414ebe0 msgr2=0x7f5cc41f5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.444+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc414ebe0 0x7f5cc41f5f30 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4001b90 tx=0x7f5cb400d520 comp rx=0 tx=0).stop 2024-05-09T05:07:22.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.444+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc414ebe0 msgr2=0x7f5cc41f5f30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:22.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.445+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc414ebe0 0x7f5cc41f5f30 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.445+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5cac00bc60 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.445+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca4075cb0 0x7f5ca40781c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.445+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5cac097af0 con 0x7f5cc40cdaf0 2024-05-09T05:07:22.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.445+0000 7f5cc2d9d640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca4075cb0 0x7f5ca40781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.446+0000 7f5cc2d9d640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca4075cb0 0x7f5ca40781c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400e010 tx=0x7f5cb4007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.451+0000 7f5cc9013640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc41f6470 0x7f5cc41f27e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.451+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc4050870 con 0x7f5cc41f6470 2024-05-09T05:07:22.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.451+0000 7f5cc359e640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc41f6470 0x7f5cc41f27e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.451+0000 7f5cc359e640 1 -- 172.21.15.1:0/2525372549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5cc40cc040 con 0x7f5cc41f6470 2024-05-09T05:07:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc359e640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc41f6470 0x7f5cc41f27e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40cc7d0 tx=0x7f5cbc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cbc037040 con 0x7f5cc41f6470 2024-05-09T05:07:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7f5cbc02ac30 con 0x7f5cc41f6470 2024-05-09T05:07:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc41f6470 msgr2=0x7f5cc41f27e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc41f6470 0x7f5cc41f27e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f5cc40cc7d0 tx=0x7f5cbc0046c0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cbb7fe640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc41f6470 msgr2=0x7f5cc41f27e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:22.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cbb7fe640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc41f6470 0x7f5cc41f27e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.453 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 14, 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 106576, 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:22.454 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:22.455 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:22.457 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:22.458 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:22.459 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:22.460 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:22.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:22.463 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:22.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:07:22.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.466 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.466 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.466 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:22.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.466 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:22.466 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca4075cb0 msgr2=0x7f5ca40781c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc9013640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca4075cb0 0x7f5ca40781c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5cb400e010 tx=0x7f5cb4007000 comp rx=0 tx=0).stop 2024-05-09T05:07:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc40cdaf0 msgr2=0x7f5cc40cfed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc9013640 1 --2- 172.21.15.1:0/2525372549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc40cdaf0 0x7f5cc40cfed0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f5cac00d830 tx=0x7f5cac00dcf0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc359e640 1 -- 172.21.15.1:0/2525372549 reap_dead start 2024-05-09T05:07:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc359e640 1 -- 172.21.15.1:0/2525372549 reap_dead start 2024-05-09T05:07:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 shutdown_connections 2024-05-09T05:07:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 >> 172.21.15.1:0/2525372549 conn(0x7f5cc4068830 msgr2=0x7f5cc415ce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 shutdown_connections 2024-05-09T05:07:22.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.452+0000 7f5cc9013640 1 -- 172.21.15.1:0/2525372549 wait complete. 2024-05-09T05:07:22.468 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:07:22.468 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:07:22.468 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:07:22.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.584+0000 7f463e508640 1 Processor -- start 2024-05-09T05:07:22.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.584+0000 7f463e508640 1 -- start start 2024-05-09T05:07:22.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f463e508640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46380cc5c0 0x7f46380cc9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f463e508640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f463806e840 0x7f463806ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f463e508640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f46380b5290 con 0x7f4638156bf0 2024-05-09T05:07:22.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f463e508640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f463815ff50 con 0x7f46380cc5c0 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f463e508640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f46380c8020 con 0x7f463806e840 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f46377fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46380cc5c0 0x7f46380cc9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f46377fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f46380cc5c0 0x7f46380cc9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60864/0 (socket says 172.21.15.1:60864) 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f46377fe640 1 -- 172.21.15.1:0/1865287419 learned_addr learned my addr 172.21.15.1:0/1865287419 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f463caa5640 1 --2- 172.21.15.1:0/1865287419 >> v2:172.21.15.196:3300/0 conn(0x7f463806e840 0x7f463806ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f46377fe640 1 -- 172.21.15.1:0/1865287419 >> v2:172.21.15.196:3300/0 conn(0x7f463806e840 msgr2=0x7f463806ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f46377fe640 1 --2- 172.21.15.1:0/1865287419 >> v2:172.21.15.196:3300/0 conn(0x7f463806e840 0x7f463806ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f46377fe640 1 -- 172.21.15.1:0/1865287419 >> v1:172.21.15.196:6789/0 conn(0x7f4638156bf0 legacy=0x7f463806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.585+0000 7f46377fe640 1 -- 172.21.15.1:0/1865287419 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46380c82e0 con 0x7f46380cc5c0 2024-05-09T05:07:22.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.586+0000 7f46377fe640 1 --2- 172.21.15.1:0/1865287419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46380cc5c0 0x7f46380cc9d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f46380cc240 tx=0x7f462c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5725778f5d2fb49 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.586+0000 7f4636ffd640 1 -- 172.21.15.1:0/1865287419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c037070 con 0x7f46380cc5c0 2024-05-09T05:07:22.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.586+0000 7f4636ffd640 1 -- 172.21.15.1:0/1865287419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f462c032440 con 0x7f46380cc5c0 2024-05-09T05:07:22.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.586+0000 7f4636ffd640 1 -- 172.21.15.1:0/1865287419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c032810 con 0x7f46380cc5c0 2024-05-09T05:07:22.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.587+0000 7f463e508640 1 -- 172.21.15.1:0/1865287419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46380cc5c0 msgr2=0x7f46380cc9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.587+0000 7f463e508640 1 --2- 172.21.15.1:0/1865287419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46380cc5c0 0x7f46380cc9d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f46380cc240 tx=0x7f462c02a270 comp rx=0 tx=0).stop 2024-05-09T05:07:22.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.587+0000 7f463e508640 1 -- 172.21.15.1:0/1865287419 shutdown_connections 2024-05-09T05:07:22.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.587+0000 7f463e508640 1 --2- 172.21.15.1:0/1865287419 >> v2:172.21.15.196:3300/0 conn(0x7f463806e840 0x7f463806ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.587+0000 7f463e508640 1 --2- 172.21.15.1:0/1865287419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f46380cc5c0 0x7f46380cc9d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.587+0000 7f463e508640 1 -- 172.21.15.1:0/1865287419 >> 172.21.15.1:0/1865287419 conn(0x7f4638068db0 msgr2=0x7f46380d1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:22.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.587+0000 7f463e508640 1 -- 172.21.15.1:0/1865287419 shutdown_connections 2024-05-09T05:07:22.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.587+0000 7f463e508640 1 -- 172.21.15.1:0/1865287419 wait complete. 2024-05-09T05:07:22.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.589+0000 7f463e508640 1 Processor -- start 2024-05-09T05:07:22.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.589+0000 7f463e508640 1 -- start start 2024-05-09T05:07:22.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f463e508640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f463806e840 0x7f46381f67d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f463e508640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380cc5c0 0x7f463814fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f463e508640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4638156bf0 0x7f46381e5bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f463e508640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f46381604d0 con 0x7f463806e840 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f463e508640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f46381601f0 con 0x7f46380cc5c0 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f463e508640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f46381607b0 con 0x7f4638156bf0 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f4637fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f463806e840 0x7f46381f67d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f4637fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f463806e840 0x7f46381f67d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60866/0 (socket says 172.21.15.1:60866) 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f4637fff640 1 -- 172.21.15.1:0/2046596973 learned_addr learned my addr 172.21.15.1:0/2046596973 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.590+0000 7f46377fe640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380cc5c0 0x7f463814fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f463caa5640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4638156bf0 0x7f46381e5bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f463caa5640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4638156bf0 msgr2=0x7f46381e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f463caa5640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4638156bf0 msgr2=0x7f46381e5bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f463caa5640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4638156bf0 0x7f46381e5bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:22.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f463caa5640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4638156bf0 0x7f46381e5bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4637fff640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4638156bf0 msgr2=0x7f46381e5bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4637fff640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4638156bf0 0x7f46381e5bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4637fff640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380cc5c0 msgr2=0x7f463814fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4637fff640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380cc5c0 0x7f463814fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4637fff640 1 -- 172.21.15.1:0/2046596973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46381f6d10 con 0x7f463806e840 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4637fff640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f463806e840 0x7f46381f67d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f462800c370 tx=0x7f462800c830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46280052d0 con 0x7f463806e840 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4628007020 con 0x7f463806e840 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4638150930 con 0x7f463806e840 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46280054b0 con 0x7f463806e840 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.591+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4638151230 con 0x7f463806e840 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.592+0000 7f463e508640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4638151940 0x7f46381e6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.592+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4638150ed0 con 0x7f4638151940 2024-05-09T05:07:22.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.592+0000 7f46377fe640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4638151940 0x7f46381e6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.592+0000 7f46377fe640 1 -- 172.21.15.1:0/2046596973 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4638050870 con 0x7f4638151940 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.593+0000 7f46377fe640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4638151940 0x7f46381e6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f46380ce260 tx=0x7f462c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.593+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c042070 con 0x7f4638151940 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.593+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4628007410 con 0x7f463806e840 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f4634ff9640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f46240761d0 0x7f46240786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4628096af0 con 0x7f463806e840 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f462c03e660 con 0x7f4638151940 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4638151940 msgr2=0x7f46381e6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f4634ff9640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4638151940 0x7f46381e6130 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f46380ce260 tx=0x7f462c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4638151940 msgr2=0x7f46381e6130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:22.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f4634ff9640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4638151940 0x7f46381e6130 secure :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f46380ce260 tx=0x7f462c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f463caa5640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f46240761d0 0x7f46240786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.594+0000 7f463caa5640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f46240761d0 0x7f46240786e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f463815a1c0 tx=0x7f463000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.600+0000 7f463e508640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380d0e40 0x7f463815d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.600+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4638151090 con 0x7f46380d0e40 2024-05-09T05:07:22.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.601+0000 7f46377fe640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380d0e40 0x7f463815d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.601+0000 7f46377fe640 1 -- 172.21.15.1:0/2046596973 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f46380b4ed0 con 0x7f46380d0e40 2024-05-09T05:07:22.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.601+0000 7f46377fe640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380d0e40 0x7f463815d980 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f462c000c00 tx=0x7f462c009240 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f462c037040 con 0x7f46380d0e40 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7f462c03ee10 con 0x7f46380d0e40 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380d0e40 msgr2=0x7f463815d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f4634ff9640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380d0e40 0x7f463815d980 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f462c000c00 tx=0x7f462c009240 comp rx=0 tx=0).stop 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f4634ff9640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380d0e40 msgr2=0x7f463815d980 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f4634ff9640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f46380d0e40 0x7f463815d980 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:07:22.603 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 14, 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 106692, 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:22.604 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:22.605 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:22.606 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:22.607 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:22.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:22.611 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:22.612 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.613 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.614 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:22.615 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:22.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f46240761d0 msgr2=0x7f46240786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463e508640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f46240761d0 0x7f46240786e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f463815a1c0 tx=0x7f463000b040 comp rx=0 tx=0).stop 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f463806e840 msgr2=0x7f46381f67d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463e508640 1 --2- 172.21.15.1:0/2046596973 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f463806e840 0x7f46381f67d0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f462800c370 tx=0x7f462800c830 comp rx=0 tx=0).stop 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463caa5640 1 -- 172.21.15.1:0/2046596973 reap_dead start 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463caa5640 1 -- 172.21.15.1:0/2046596973 reap_dead start 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 shutdown_connections 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 >> 172.21.15.1:0/2046596973 conn(0x7f4638068db0 msgr2=0x7f463806ec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 shutdown_connections 2024-05-09T05:07:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.602+0000 7f463e508640 1 -- 172.21.15.1:0/2046596973 wait complete. 2024-05-09T05:07:22.617 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:07:22.617 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:07:22.618 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:07:22.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.732+0000 7f7b47e47640 1 Processor -- start 2024-05-09T05:07:22.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.732+0000 7f7b47e47640 1 -- start start 2024-05-09T05:07:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.733+0000 7f7b47e47640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 0x7f7b400c77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.733+0000 7f7b47e47640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b40151130 0x7f7b400c7ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.733+0000 7f7b47e47640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7b400cd5d0 con 0x7f7b4006c6f0 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b47e47640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7b400cc2f0 con 0x7f7b40151130 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b47e47640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7b400cc9c0 con 0x7f7b40068a50 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b453e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b40151130 0x7f7b400c7ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b453e2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7b40151130 0x7f7b400c7ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60880/0 (socket says 172.21.15.1:60880) 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b453e2640 1 -- 172.21.15.1:0/3465605404 learned_addr learned my addr 172.21.15.1:0/3465605404 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b45be3640 1 --2- 172.21.15.1:0/3465605404 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 0x7f7b400c77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b45be3640 1 -- 172.21.15.1:0/3465605404 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 msgr2=0x7f7b400c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b45be3640 1 -- 172.21.15.1:0/3465605404 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 msgr2=0x7f7b400c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b45be3640 1 --2- 172.21.15.1:0/3465605404 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 0x7f7b400c77a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b453e2640 1 -- 172.21.15.1:0/3465605404 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 msgr2=0x7f7b400c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b45be3640 1 --2- 172.21.15.1:0/3465605404 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 0x7f7b400c77a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b453e2640 1 --2- 172.21.15.1:0/3465605404 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 0x7f7b400c77a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b453e2640 1 -- 172.21.15.1:0/3465605404 >> v1:172.21.15.196:6789/0 conn(0x7f7b4006c6f0 legacy=0x7f7b4015a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.734+0000 7f7b453e2640 1 -- 172.21.15.1:0/3465605404 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b400ccca0 con 0x7f7b40151130 2024-05-09T05:07:22.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.735+0000 7f7b453e2640 1 --2- 172.21.15.1:0/3465605404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40151130 0x7f7b400c7ce0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b3402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14d3c588e34c8b9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.735+0000 7f7b44be1640 1 -- 172.21.15.1:0/3465605404 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34037070 con 0x7f7b40151130 2024-05-09T05:07:22.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.735+0000 7f7b44be1640 1 -- 172.21.15.1:0/3465605404 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b3402ae50 con 0x7f7b40151130 2024-05-09T05:07:22.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.735+0000 7f7b44be1640 1 -- 172.21.15.1:0/3465605404 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34032860 con 0x7f7b40151130 2024-05-09T05:07:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.736+0000 7f7b47e47640 1 -- 172.21.15.1:0/3465605404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40151130 msgr2=0x7f7b400c7ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.736+0000 7f7b47e47640 1 --2- 172.21.15.1:0/3465605404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40151130 0x7f7b400c7ce0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b3402a300 comp rx=0 tx=0).stop 2024-05-09T05:07:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.736+0000 7f7b47e47640 1 -- 172.21.15.1:0/3465605404 shutdown_connections 2024-05-09T05:07:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.736+0000 7f7b47e47640 1 --2- 172.21.15.1:0/3465605404 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40151130 0x7f7b400c7ce0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.736+0000 7f7b47e47640 1 --2- 172.21.15.1:0/3465605404 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40068a50 0x7f7b400c77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.736+0000 7f7b47e47640 1 -- 172.21.15.1:0/3465605404 >> 172.21.15.1:0/3465605404 conn(0x7f7b400cab90 msgr2=0x7f7b400cafa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.736+0000 7f7b47e47640 1 -- 172.21.15.1:0/3465605404 shutdown_connections 2024-05-09T05:07:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.736+0000 7f7b47e47640 1 -- 172.21.15.1:0/3465605404 wait complete. 2024-05-09T05:07:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.738+0000 7f7b47e47640 1 Processor -- start 2024-05-09T05:07:22.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.738+0000 7f7b47e47640 1 -- start start 2024-05-09T05:07:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b47e47640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40068a50 0x7f7b400ca210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b47e47640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b4006c6f0 0x7f7b401ee410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b47e47640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40151130 0x7f7b4015b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b47e47640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7b400cbeb0 con 0x7f7b40068a50 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b47e47640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7b400cd260 con 0x7f7b4006c6f0 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b47e47640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7b400cc030 con 0x7f7b40151130 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40068a50 0x7f7b400ca210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40068a50 0x7f7b400ca210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60896/0 (socket says 172.21.15.1:60896) 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 -- 172.21.15.1:0/2865205064 learned_addr learned my addr 172.21.15.1:0/2865205064 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b4006c6f0 0x7f7b401ee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b463e4640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40151130 0x7f7b4015b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40151130 msgr2=0x7f7b4015b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b40151130 0x7f7b4015b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b4006c6f0 msgr2=0x7f7b401ee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7b4006c6f0 0x7f7b401ee410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:22.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 -- 172.21.15.1:0/2865205064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b401ee950 con 0x7f7b40068a50 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.739+0000 7f7b45be3640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40068a50 0x7f7b400ca210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800bea0 tx=0x7f7b3800bf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.740+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38018620 con 0x7f7b40068a50 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.740+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7b3801d070 con 0x7f7b40068a50 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.740+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b4015c590 con 0x7f7b40068a50 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.740+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b38018a20 con 0x7f7b40068a50 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.740+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b4015cb10 con 0x7f7b40068a50 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.740+0000 7f7b47e47640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.740+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7b400ca9a0 con 0x7f7b401eeb20 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.741+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.741+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.741+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.741+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:22.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.741+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.741+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7b38018bc0 con 0x7f7b40068a50 2024-05-09T05:07:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.742+0000 7f7b3e7fc640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7b2c076300 0x7f7b2c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.742+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7b3809c0c0 con 0x7f7b40068a50 2024-05-09T05:07:22.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.742+0000 7f7b463e4640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7b2c076300 0x7f7b2c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.743+0000 7f7b463e4640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7b2c076300 0x7f7b2c078810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7b30007fd0 tx=0x7f7b30009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.942+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.942+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:07:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.942+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.942+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:22.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:22.942+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:07:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:23.343+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:23.343+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:07:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:23.343+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:23.343+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:23.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:23.343+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:07:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:24.145+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:24.145+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:07:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:24.145+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:24.145+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:24.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:24.145+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:07:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:25.747+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:25.747+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:07:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:25.747+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:25.747+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:25.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:25.747+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T05:07:26.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.739+0000 7f7b3d7fa640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:26.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.739+0000 7f7b3d7fa640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:26.740 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.739+0000 7f7b3d7fa640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.739+0000 7f7b3d7fa640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 msgr2=0x7f7b40157120 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.739+0000 7f7b3d7fa640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b401eeb20 0x7f7b40157120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.739+0000 7f7b3d7fa640 1 -- 172.21.15.1:0/2865205064 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7b240060d0 con 0x7f7b24003540 2024-05-09T05:07:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.740+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.740+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 msgr2=0x7f7b24005a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:07:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.740+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 msgr2=0x7f7b24005a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.740+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:26.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.740+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.941+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.941+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 msgr2=0x7f7b24005a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:07:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.941+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 msgr2=0x7f7b24005a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:07:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.941+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:26.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:26.941+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:07:27.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.342+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.342+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7b24006280 con 0x7f7b24003540 2024-05-09T05:07:27.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.343+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7b400cc830 tx=0x7f7b340046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.343+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34037040 con 0x7f7b24003540 2024-05-09T05:07:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.343+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7b3402ae50 con 0x7f7b24003540 2024-05-09T05:07:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.343+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 msgr2=0x7f7b24005a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.343+0000 7f7b3e7fc640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7b400cc830 tx=0x7f7b340046c0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.343+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 msgr2=0x7f7b24005a20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:27.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.343+0000 7f7b3e7fc640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b24003540 0x7f7b24005a20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.350+0000 7f7b47e47640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b4015de30 0x7f7b40157660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.350+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7b40154b80 con 0x7f7b4015de30 2024-05-09T05:07:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.350+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b4015de30 0x7f7b40157660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.351+0000 7f7b453e2640 1 -- 172.21.15.1:0/2865205064 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7b4006cdf0 con 0x7f7b4015de30 2024-05-09T05:07:27.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.351+0000 7f7b453e2640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b4015de30 0x7f7b40157660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b34008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.351+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b34037040 con 0x7f7b4015de30 2024-05-09T05:07:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f7b34050230 con 0x7f7b4015de30 2024-05-09T05:07:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b4015de30 msgr2=0x7f7b40157660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b3e7fc640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b4015de30 0x7f7b40157660 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b34008030 comp rx=0 tx=0).stop 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b3e7fc640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b4015de30 msgr2=0x7f7b40157660 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b3e7fc640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7b4015de30 0x7f7b40157660 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7b34009870 tx=0x7f7b34008030 comp rx=0 tx=0).stop 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b463e4640 1 -- 172.21.15.1:0/2865205064 reap_dead start 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 14, 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 5348, 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:07:27.353 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:27.354 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:27.355 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:27.356 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:27.357 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:27.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:27.360 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:27.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:27.362 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:27.363 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:27.364 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:27.364 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:27.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:27.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:27.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:27.364 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:27.364 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7b2c076300 msgr2=0x7f7b2c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7b2c076300 0x7f7b2c078810 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7b30007fd0 tx=0x7f7b30009040 comp rx=0 tx=0).stop 2024-05-09T05:07:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40068a50 msgr2=0x7f7b400ca210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40068a50 0x7f7b400ca210 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f7b3800bea0 tx=0x7f7b3800bf90 comp rx=0 tx=0).stop 2024-05-09T05:07:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 shutdown_connections 2024-05-09T05:07:27.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7b2c076300 0x7f7b2c078810 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 --2- 172.21.15.1:0/2865205064 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7b40068a50 0x7f7b400ca210 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 >> 172.21.15.1:0/2865205064 conn(0x7f7b400cab90 msgr2=0x7f7b4014ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 shutdown_connections 2024-05-09T05:07:27.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.352+0000 7f7b47e47640 1 -- 172.21.15.1:0/2865205064 wait complete. 2024-05-09T05:07:27.366 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:07:27.366 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:07:27.366 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:07:27.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.480+0000 7f0ec80d1640 1 Processor -- start 2024-05-09T05:07:27.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.481+0000 7f0ec80d1640 1 -- start start 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec80d1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0ec00ca340 0x7f0ec00c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec80d1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 0x7f0ec0070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec80d1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0ec015b110 con 0x7f0ec0072120 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec80d1640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0ec00c53f0 con 0x7f0ec00ca340 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec80d1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec00c55b0 con 0x7f0ec00c62e0 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec566c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0ec0072120 0x7f0ec01547b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:33036/0 (socket says 172.21.15.1:33036) 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec566c640 1 -- 172.21.15.1:0/3700996566 learned_addr learned my addr 172.21.15.1:0/3700996566 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec666e640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 0x7f0ec0070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec5e6d640 1 --2- 172.21.15.1:0/3700996566 >> v2:172.21.15.196:3300/0 conn(0x7f0ec00ca340 0x7f0ec00c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3262902107 0 0) 0x7f0ec015b110 con 0x7f0ec0072120 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.482+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0eac003520 con 0x7f0ec0072120 2024-05-09T05:07:27.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3026704868 0 0) 0x7f0eac003520 con 0x7f0ec0072120 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 msgr2=0x7f0ec0070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 0x7f0ec0070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 >> v2:172.21.15.196:3300/0 conn(0x7f0ec00ca340 msgr2=0x7f0ec00c4eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 --2- 172.21.15.1:0/3700996566 >> v2:172.21.15.196:3300/0 conn(0x7f0ec00ca340 0x7f0ec00c4eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec015b3b0 con 0x7f0ec0072120 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec666e640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 0x7f0ec0070e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f0eb00016f0 con 0x7f0ec0072120 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 >> v1:172.21.15.1:6789/0 conn(0x7f0ec0072120 legacy=0x7f0ec01547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eac003a00 0x7f0eac003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eac004800 0x7f0eac004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 0x7f0eac005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec00c53f0 con 0x7f0eac004800 2024-05-09T05:07:27.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec00c55b0 con 0x7f0eac003a00 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0eac00baa0 con 0x7f0ec00c62e0 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec566c640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eac004800 0x7f0eac004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec5e6d640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eac003a00 0x7f0eac003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec666e640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 0x7f0eac005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec566c640 1 -- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 msgr2=0x7f0eac005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec566c640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0ec00c62e0 0x7f0eac005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec566c640 1 -- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eac003a00 msgr2=0x7f0eac003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec566c640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0eac003a00 0x7f0eac003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.483+0000 7f0ec566c640 1 -- 172.21.15.1:0/3700996566 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0eac003520 con 0x7f0eac004800 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.484+0000 7f0ec566c640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eac004800 0x7f0eac004c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000c290 tx=0x7f0eac005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6976cd76477c7a88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.484+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0007d50 con 0x7f0eac004800 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.484+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb0005af0 con 0x7f0eac004800 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.484+0000 7f0ec4e6b640 1 -- 172.21.15.1:0/3700996566 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb0007070 con 0x7f0eac004800 2024-05-09T05:07:27.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.485+0000 7f0ec80d1640 1 -- 172.21.15.1:0/3700996566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eac004800 msgr2=0x7f0eac004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.485+0000 7f0ec80d1640 1 --2- 172.21.15.1:0/3700996566 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0eac004800 0x7f0eac004c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f0eb000c290 tx=0x7f0eac005830 comp rx=0 tx=0).stop 2024-05-09T05:07:27.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.485+0000 7f0ec666e640 1 -- 172.21.15.1:0/3700996566 reap_dead start 2024-05-09T05:07:27.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.485+0000 7f0ec80d1640 1 -- 172.21.15.1:0/3700996566 shutdown_connections 2024-05-09T05:07:27.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.485+0000 7f0ec80d1640 1 -- 172.21.15.1:0/3700996566 >> 172.21.15.1:0/3700996566 conn(0x7f0ec00731d0 msgr2=0x7f0ec00cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:27.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.485+0000 7f0ec80d1640 1 -- 172.21.15.1:0/3700996566 shutdown_connections 2024-05-09T05:07:27.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.485+0000 7f0ec80d1640 1 -- 172.21.15.1:0/3700996566 wait complete. 2024-05-09T05:07:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.487+0000 7f0ec80d1640 1 Processor -- start 2024-05-09T05:07:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.487+0000 7f0ec80d1640 1 -- start start 2024-05-09T05:07:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec80d1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0eac004800 0x7f0ec00cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec80d1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ec01515d0 0x7f0ec00ce790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec80d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ec0159730 0x7f0ec014b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec80d1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec00c5dc0 con 0x7f0ec0159730 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec80d1640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0ec00c5ae0 con 0x7f0ec01515d0 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec80d1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0ec00c66f0 con 0x7f0eac004800 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ec0159730 0x7f0ec014b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ec0159730 0x7f0ec014b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60922/0 (socket says 172.21.15.1:60922) 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 -- 172.21.15.1:0/871672850 learned_addr learned my addr 172.21.15.1:0/871672850 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec566c640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ec01515d0 0x7f0ec00ce790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec5e6d640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0eac004800 0x7f0ec00cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 -- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0eac004800 msgr2=0x7f0ec00cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0eac004800 0x7f0ec00cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 -- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ec01515d0 msgr2=0x7f0ec00ce790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ec01515d0 0x7f0ec00ce790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 -- 172.21.15.1:0/871672850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ec00cf4c0 con 0x7f0ec0159730 2024-05-09T05:07:27.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0ec666e640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ec0159730 0x7f0ec014b660 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00cbe40 tx=0x7f0eb8009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:27.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0eb67fc640 1 -- 172.21.15.1:0/871672850 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb800fd70 con 0x7f0ec0159730 2024-05-09T05:07:27.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.489+0000 7f0eb67fc640 1 -- 172.21.15.1:0/871672850 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0eb8003990 con 0x7f0ec0159730 2024-05-09T05:07:27.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.490+0000 7f0eb67fc640 1 -- 172.21.15.1:0/871672850 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0eb8003d80 con 0x7f0ec0159730 2024-05-09T05:07:27.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.490+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ec006f430 con 0x7f0ec0159730 2024-05-09T05:07:27.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.490+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ec00cf780 con 0x7f0ec0159730 2024-05-09T05:07:27.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.490+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0ec00cc0a0 con 0x7f0ec0159730 2024-05-09T05:07:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.492+0000 7f0eb67fc640 1 -- 172.21.15.1:0/871672850 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0eb8004020 con 0x7f0ec0159730 2024-05-09T05:07:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.492+0000 7f0eb67fc640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ea40762d0 0x7f0ea40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.492+0000 7f0eb67fc640 1 -- 172.21.15.1:0/871672850 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0eb80984b0 con 0x7f0ec0159730 2024-05-09T05:07:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.492+0000 7f0ec5e6d640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ea40762d0 0x7f0ea40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:27.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.493+0000 7f0ec5e6d640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ea40762d0 0x7f0ea40787e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009b00 tx=0x7f0ebc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:27.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.496+0000 7f0eb67fc640 1 -- 172.21.15.1:0/871672850 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0ec00cc0a0 con 0x7f0ec0159730 2024-05-09T05:07:27.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.786+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f0ec0050870 con 0x7f0ec0159730 2024-05-09T05:07:27.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.787+0000 7f0eb67fc640 1 -- 172.21.15.1:0/871672850 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f0eb8019030 con 0x7f0ec0159730 2024-05-09T05:07:27.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ea40762d0 msgr2=0x7f0ea40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ea40762d0 0x7f0ea40787e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0ebc009b00 tx=0x7f0ebc008040 comp rx=0 tx=0).stop 2024-05-09T05:07:27.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ec0159730 msgr2=0x7f0ec014b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ec0159730 0x7f0ec014b660 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f0ec00cbe40 tx=0x7f0eb8009f90 comp rx=0 tx=0).stop 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 shutdown_connections 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0ea40762d0 0x7f0ea40787e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0ec01515d0 0x7f0ec00ce790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0ec0159730 0x7f0ec014b660 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 --2- 172.21.15.1:0/871672850 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0eac004800 0x7f0ec00cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.789+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 >> 172.21.15.1:0/871672850 conn(0x7f0ec00731d0 msgr2=0x7f0ec00b44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.790+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 shutdown_connections 2024-05-09T05:07:27.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:27.790+0000 7f0ec80d1640 1 -- 172.21.15.1:0/871672850 wait complete. 2024-05-09T05:07:27.802 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:07:31.727 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:07:31.730 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:07:37.804 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:07:37.804 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:07:37.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.922+0000 7ff5fbe13640 1 Processor -- start 2024-05-09T05:07:37.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.922+0000 7ff5fbe13640 1 -- start start 2024-05-09T05:07:37.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5fbe13640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff5f40c5150 0x7ff5f40c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:37.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5fbe13640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5f4160320 con 0x7ff5f4156b30 2024-05-09T05:07:37.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5fbe13640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5f406f8d0 con 0x7ff5f415ff10 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5fbe13640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff5f40caf50 con 0x7ff5f40c5150 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5fa3b0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff5f4156b30 0x7ff5f41672a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40940/0 (socket says 172.21.15.1:40940) 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5fa3b0640 1 -- 172.21.15.1:0/838647130 learned_addr learned my addr 172.21.15.1:0/838647130 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5f9baf640 1 --2- 172.21.15.1:0/838647130 >> v2:172.21.15.196:3300/0 conn(0x7ff5f40c5150 0x7ff5f40c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1524183752 0 0) 0x7ff5ec001490 con 0x7ff5f4156b30 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e0003520 con 0x7ff5f4156b30 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2238155711 0 0) 0x7ff5e0003520 con 0x7ff5f4156b30 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 >> v2:172.21.15.196:3300/0 conn(0x7ff5f40c5150 msgr2=0x7ff5f40c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5f8bad640 1 --2- 172.21.15.1:0/838647130 >> v2:172.21.15.196:3300/0 conn(0x7ff5f40c5150 0x7ff5f40c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 >> v1:172.21.15.196:6789/0 conn(0x7ff5f415ff10 legacy=0x7ff5f40cfcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.924+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f40cb230 con 0x7ff5f4156b30 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7ff5ec001c40 con 0x7ff5f4156b30 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 >> v1:172.21.15.1:6789/0 conn(0x7ff5f4156b30 legacy=0x7ff5f41672a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f8bad640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5e0003a50 0x7ff5e0003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f8bad640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5e0004800 0x7ff5e0004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f8bad640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5e000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f40caf50 con 0x7ff5e0005250 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff5e000c1d0 con 0x7ff5e0004800 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff5e000c3d0 con 0x7ff5e0003a50 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f9baf640 1 --2- 172.21.15.1:0/838647130 >> v2:172.21.15.196:3300/0 conn(0x7ff5f40c5150 0x7ff5f40c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5fa3b0640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5e000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f93ae640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5e0004800 0x7ff5e0004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5f9baf640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5e0003a50 0x7ff5e0003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5fa3b0640 1 -- 172.21.15.1:0/838647130 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5e0003a50 msgr2=0x7ff5e0003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5fa3b0640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5e0003a50 0x7ff5e0003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5fa3b0640 1 -- 172.21.15.1:0/838647130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5e0004800 msgr2=0x7ff5e0004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5fa3b0640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5e0004800 0x7ff5e0004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.925+0000 7ff5fa3b0640 1 -- 172.21.15.1:0/838647130 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5e0003520 con 0x7ff5e0005250 2024-05-09T05:07:37.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.926+0000 7ff5fa3b0640 1 -- 172.21.15.1:0/838647130 reap_dead start 2024-05-09T05:07:37.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.926+0000 7ff5fa3b0640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5e000b9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00aa20 tx=0x7ff5e0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a71c2bf238a4428e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:37.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.926+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec00a670 con 0x7ff5e0005250 2024-05-09T05:07:37.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.926+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5ec02f070 con 0x7ff5e0005250 2024-05-09T05:07:37.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.926+0000 7ff5f8bad640 1 -- 172.21.15.1:0/838647130 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec02bdb0 con 0x7ff5e0005250 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.927+0000 7ff5fbe13640 1 -- 172.21.15.1:0/838647130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 msgr2=0x7ff5e000b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.927+0000 7ff5fbe13640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5e000b9c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00aa20 tx=0x7ff5e0005830 comp rx=0 tx=0).stop 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.927+0000 7ff5fbe13640 1 -- 172.21.15.1:0/838647130 shutdown_connections 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.927+0000 7ff5fbe13640 1 --2- 172.21.15.1:0/838647130 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5e000b9c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.927+0000 7ff5fbe13640 1 -- 172.21.15.1:0/838647130 >> 172.21.15.1:0/838647130 conn(0x7ff5f4068830 msgr2=0x7ff5f4068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.927+0000 7ff5fbe13640 1 -- 172.21.15.1:0/838647130 shutdown_connections 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.927+0000 7ff5fbe13640 1 -- 172.21.15.1:0/838647130 wait complete. 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.929+0000 7ff5fbe13640 1 Processor -- start 2024-05-09T05:07:37.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.929+0000 7ff5fbe13640 1 -- start start 2024-05-09T05:07:37.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5fbe13640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5f414e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:37.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5fbe13640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40c90b0 0x7ff5f41f67c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:37.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5fbe13640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40c8ad0 0x7ff5f41524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:37.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5fbe13640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f40cb750 con 0x7ff5e0005250 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5fbe13640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f40c5cb0 con 0x7ff5f40c8ad0 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5fbe13640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f406f590 con 0x7ff5f40c90b0 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5f414e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5f414e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48616/0 (socket says 172.21.15.1:48616) 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 -- 172.21.15.1:0/249075033 learned_addr learned my addr 172.21.15.1:0/249075033 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f93ae640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40c90b0 0x7ff5f41f67c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5fa3b0640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40c8ad0 0x7ff5f41524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 -- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40c90b0 msgr2=0x7ff5f41f67c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40c90b0 0x7ff5f41f67c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 -- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40c8ad0 msgr2=0x7ff5f41524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40c8ad0 0x7ff5f41524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 -- 172.21.15.1:0/249075033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f41f6d00 con 0x7ff5e0005250 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.931+0000 7ff5f9baf640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5f414e0b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0011ac0 tx=0x7ff5f0011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:37.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.932+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/249075033 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f000fc10 con 0x7ff5e0005250 2024-05-09T05:07:37.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.932+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/249075033 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff5f000fdb0 con 0x7ff5e0005250 2024-05-09T05:07:37.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.932+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/249075033 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0004260 con 0x7ff5e0005250 2024-05-09T05:07:37.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.932+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f40c9db0 con 0x7ff5e0005250 2024-05-09T05:07:37.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.932+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f40b58b0 con 0x7ff5e0005250 2024-05-09T05:07:37.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.933+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f4050870 con 0x7ff5e0005250 2024-05-09T05:07:37.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.933+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/249075033 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff5f0012040 con 0x7ff5e0005250 2024-05-09T05:07:37.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.934+0000 7ff5ea7fc640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d80762d0 0x7ff5d80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:37.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.934+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/249075033 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff5f0097b90 con 0x7ff5e0005250 2024-05-09T05:07:37.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.934+0000 7ff5f93ae640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d80762d0 0x7ff5d80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:37.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.935+0000 7ff5f93ae640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d80762d0 0x7ff5d80787e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40c8a80 tx=0x7ff5e4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:37.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:37.939+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/249075033 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff5f0019030 con 0x7ff5e0005250 2024-05-09T05:07:38.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.219+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff5f40cb9f0 con 0x7ff5e0005250 2024-05-09T05:07:38.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.220+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/249075033 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff5f0060d90 con 0x7ff5e0005250 2024-05-09T05:07:38.221 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:07:38.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d80762d0 msgr2=0x7ff5d80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d80762d0 0x7ff5d80787e0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff5f40c8a80 tx=0x7ff5e4009040 comp rx=0 tx=0).stop 2024-05-09T05:07:38.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 msgr2=0x7ff5f414e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5f414e0b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0011ac0 tx=0x7ff5f0011f80 comp rx=0 tx=0).stop 2024-05-09T05:07:38.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 shutdown_connections 2024-05-09T05:07:38.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d80762d0 0x7ff5d80787e0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40c8ad0 0x7ff5f41524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40c90b0 0x7ff5f41f67c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 --2- 172.21.15.1:0/249075033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5e0005250 0x7ff5f414e0b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 >> 172.21.15.1:0/249075033 conn(0x7ff5f4068830 msgr2=0x7ff5f4151690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 shutdown_connections 2024-05-09T05:07:38.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.222+0000 7ff5fbe13640 1 -- 172.21.15.1:0/249075033 wait complete. 2024-05-09T05:07:38.235 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:07:38.235 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:07:38.235 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:07:38.235 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:07:38.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.351+0000 7fec81108640 1 Processor -- start 2024-05-09T05:07:38.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.352+0000 7fec81108640 1 -- start start 2024-05-09T05:07:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec81108640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec7c070230 0x7fec7c070640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec81108640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec7c0c3f50 con 0x7fec7c15d2d0 2024-05-09T05:07:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec81108640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fec7c06c800 con 0x7fec7c070f70 2024-05-09T05:07:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec81108640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fec7c15ce40 con 0x7fec7c070230 2024-05-09T05:07:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec7b59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fec7c070230 0x7fec7c070640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec7a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fec7c15d2d0 0x7fec7c0c9520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:40952/0 (socket says 172.21.15.1:40952) 2024-05-09T05:07:38.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec7a59c640 1 -- 172.21.15.1:0/3409969266 learned_addr learned my addr 172.21.15.1:0/3409969266 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:07:38.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec7b59e640 1 -- 172.21.15.1:0/3409969266 >> v1:172.21.15.196:6789/0 conn(0x7fec7c070f70 legacy=0x7fec7c1568e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec7b59e640 1 -- 172.21.15.1:0/3409969266 >> v1:172.21.15.1:6789/0 conn(0x7fec7c15d2d0 legacy=0x7fec7c0c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec79d9b640 1 -- 172.21.15.1:0/3409969266 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2014450547 0 0) 0x7fec7c0c3f50 con 0x7fec7c15d2d0 2024-05-09T05:07:38.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.353+0000 7fec7b59e640 1 -- 172.21.15.1:0/3409969266 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7c06cae0 con 0x7fec7c070230 2024-05-09T05:07:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.354+0000 7fec7b59e640 1 --2- 172.21.15.1:0/3409969266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 0x7fec7c070640 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fec7c15cfc0 tx=0x7fec7402a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de362336efaec752 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.354+0000 7fec79d9b640 1 -- 172.21.15.1:0/3409969266 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74037070 con 0x7fec7c070230 2024-05-09T05:07:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.354+0000 7fec79d9b640 1 -- 172.21.15.1:0/3409969266 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec74007740 con 0x7fec7c070230 2024-05-09T05:07:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.354+0000 7fec79d9b640 1 -- 172.21.15.1:0/3409969266 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74007b10 con 0x7fec7c070230 2024-05-09T05:07:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.355+0000 7fec81108640 1 -- 172.21.15.1:0/3409969266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 msgr2=0x7fec7c070640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.355+0000 7fec81108640 1 --2- 172.21.15.1:0/3409969266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 0x7fec7c070640 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fec7c15cfc0 tx=0x7fec7402a780 comp rx=0 tx=0).stop 2024-05-09T05:07:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.355+0000 7fec81108640 1 -- 172.21.15.1:0/3409969266 shutdown_connections 2024-05-09T05:07:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.355+0000 7fec81108640 1 --2- 172.21.15.1:0/3409969266 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 0x7fec7c070640 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.355+0000 7fec81108640 1 -- 172.21.15.1:0/3409969266 >> 172.21.15.1:0/3409969266 conn(0x7fec7c06d800 msgr2=0x7fec7c06dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.355+0000 7fec81108640 1 -- 172.21.15.1:0/3409969266 shutdown_connections 2024-05-09T05:07:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.355+0000 7fec81108640 1 -- 172.21.15.1:0/3409969266 wait complete. 2024-05-09T05:07:38.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.358+0000 7fec81108640 1 Processor -- start 2024-05-09T05:07:38.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.358+0000 7fec81108640 1 -- start start 2024-05-09T05:07:38.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec81108640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 0x7fec7c0cdd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec81108640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fec7c070f70 0x7fec7c151490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec81108640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec7c15d2d0 0x7fec7c0c61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec81108640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c15cad0 con 0x7fec7c070230 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec81108640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c06d0a0 con 0x7fec7c15d2d0 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec81108640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fec7c0c9c30 con 0x7fec7c070f70 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 0x7fec7c0cdd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 0x7fec7c0cdd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48636/0 (socket says 172.21.15.1:48636) 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 -- 172.21.15.1:0/3006220102 learned_addr learned my addr 172.21.15.1:0/3006220102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7b59e640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec7c15d2d0 0x7fec7c0c61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7a59c640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fec7c070f70 0x7fec7c151490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 -- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fec7c070f70 msgr2=0x7fec7c151490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fec7c070f70 0x7fec7c151490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 -- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec7c15d2d0 msgr2=0x7fec7c0c61e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fec7c15d2d0 0x7fec7c0c61e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 -- 172.21.15.1:0/3006220102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fec7c0ce290 con 0x7fec7c070230 2024-05-09T05:07:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.361+0000 7fec7ad9d640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 0x7fec7c0cdd50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00c400 tx=0x7fec6c00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.362+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c017070 con 0x7fec7c070230 2024-05-09T05:07:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.362+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fec6c0052d0 con 0x7fec7c070230 2024-05-09T05:07:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.362+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fec7c14f4a0 con 0x7fec7c070230 2024-05-09T05:07:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.362+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec6c003830 con 0x7fec7c070230 2024-05-09T05:07:38.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.362+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fec7c14e440 con 0x7fec7c070230 2024-05-09T05:07:38.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.363+0000 7fec81108640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c0c6720 0x7fec7c14e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.363+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c0696d0 con 0x7fec7c0c6720 2024-05-09T05:07:38.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.363+0000 7fec7a59c640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c0c6720 0x7fec7c14e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.363+0000 7fec7a59c640 1 -- 172.21.15.1:0/3006220102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fec7c0cda50 con 0x7fec7c0c6720 2024-05-09T05:07:38.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.363+0000 7fec7a59c640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c0c6720 0x7fec7c14e780 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fec7c151e00 tx=0x7fec700017e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.364+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec70019070 con 0x7fec7c0c6720 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.364+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fec70011580 con 0x7fec7c0c6720 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.364+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c0c6720 msgr2=0x7fec7c14e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.364+0000 7fec637fe640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c0c6720 0x7fec7c14e780 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fec7c151e00 tx=0x7fec700017e0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.364+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c0c6720 msgr2=0x7fec7c14e780 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.364+0000 7fec637fe640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c0c6720 0x7fec7c14e780 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.364+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fec6c01a070 con 0x7fec7c070230 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.365+0000 7fec637fe640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fec68075d40 0x7fec68078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.365+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fec6c096910 con 0x7fec7c070230 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.365+0000 7fec7a59c640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fec68075d40 0x7fec68078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.366+0000 7fec7a59c640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fec68075d40 0x7fec68078250 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fec7c151e00 tx=0x7fec70009070 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.371+0000 7fec81108640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c14c230 0x7fec7c14c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.371+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fec7c070d50 con 0x7fec7c14c230 2024-05-09T05:07:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.371+0000 7fec7b59e640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c14c230 0x7fec7c14c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.371+0000 7fec7b59e640 1 -- 172.21.15.1:0/3006220102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fec7c06dc10 con 0x7fec7c14c230 2024-05-09T05:07:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.371+0000 7fec7b59e640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c14c230 0x7fec7c14c640 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0729e0 tx=0x7fec74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.371+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fec74042030 con 0x7fec7c14c230 2024-05-09T05:07:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7fec74009a30 con 0x7fec7c14c230 2024-05-09T05:07:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c14c230 msgr2=0x7fec7c14c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec637fe640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c14c230 0x7fec7c14c640 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fec7c0729e0 tx=0x7fec74004a40 comp rx=0 tx=0).stop 2024-05-09T05:07:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec637fe640 1 -- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c14c230 msgr2=0x7fec7c14c640 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec637fe640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c14c230 0x7fec7c14c640 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.373 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 14, 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 122496, 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.374 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.375 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.376 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:38.377 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:38.378 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.379 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:38.380 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.381 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:38.382 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:38.383 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.384 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.385 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.386 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.387 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.388 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.389 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:38.389 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fec68075d40 msgr2=0x7fec68078250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec81108640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fec68075d40 0x7fec68078250 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fec7c151e00 tx=0x7fec70009070 comp rx=0 tx=0).stop 2024-05-09T05:07:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 msgr2=0x7fec7c0cdd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec81108640 1 --2- 172.21.15.1:0/3006220102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fec7c070230 0x7fec7c0cdd50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fec6c00c400 tx=0x7fec6c00c8c0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec7b59e640 1 -- 172.21.15.1:0/3006220102 reap_dead start 2024-05-09T05:07:38.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec7b59e640 1 -- 172.21.15.1:0/3006220102 reap_dead start 2024-05-09T05:07:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 shutdown_connections 2024-05-09T05:07:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 >> 172.21.15.1:0/3006220102 conn(0x7fec7c06d800 msgr2=0x7fec7c15c220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 shutdown_connections 2024-05-09T05:07:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.372+0000 7fec81108640 1 -- 172.21.15.1:0/3006220102 wait complete. 2024-05-09T05:07:38.391 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:07:38.391 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:07:38.391 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:07:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.513+0000 7fbff635e640 1 Processor -- start 2024-05-09T05:07:38.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.513+0000 7fbff635e640 1 -- start start 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbff635e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbff00b4ec0 0x7fbff00cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbff635e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fbff00ce010 0x7fbff015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbff635e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff0160290 0x7fbff01606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbff635e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbff015d0c0 con 0x7fbff00b4ec0 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbff635e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fbff0160be0 con 0x7fbff00ce010 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbff635e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbff015c7b0 con 0x7fbff0160290 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbfeffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbff00b4ec0 0x7fbff00cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbfeffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbff00b4ec0 0x7fbff00cf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48670/0 (socket says 172.21.15.1:48670) 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.514+0000 7fbfeffff640 1 -- 172.21.15.1:0/3526676873 learned_addr learned my addr 172.21.15.1:0/3526676873 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.515+0000 7fbfef7fe640 1 --2- 172.21.15.1:0/3526676873 >> v2:172.21.15.196:3300/0 conn(0x7fbff00ce010 0x7fbff015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.515+0000 7fbff48fb640 1 --2- 172.21.15.1:0/3526676873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff0160290 0x7fbff01606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.515+0000 7fbfeffff640 1 -- 172.21.15.1:0/3526676873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff0160290 msgr2=0x7fbff01606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:38.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.515+0000 7fbfeffff640 1 --2- 172.21.15.1:0/3526676873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff0160290 0x7fbff01606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.515+0000 7fbfeffff640 1 -- 172.21.15.1:0/3526676873 >> v2:172.21.15.196:3300/0 conn(0x7fbff00ce010 msgr2=0x7fbff015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:38.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.515+0000 7fbfeffff640 1 --2- 172.21.15.1:0/3526676873 >> v2:172.21.15.196:3300/0 conn(0x7fbff00ce010 0x7fbff015fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.515+0000 7fbfeffff640 1 -- 172.21.15.1:0/3526676873 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff00cca50 con 0x7fbff00b4ec0 2024-05-09T05:07:38.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.515+0000 7fbfef7fe640 1 --2- 172.21.15.1:0/3526676873 >> v2:172.21.15.196:3300/0 conn(0x7fbff00ce010 0x7fbff015fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.516+0000 7fbfeffff640 1 --2- 172.21.15.1:0/3526676873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 0x7fbff00cf9d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbff00c8b40 tx=0x7fbfe002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=903c207d7603fc45 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.516+0000 7fbfeeffd640 1 -- 172.21.15.1:0/3526676873 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0037070 con 0x7fbff00b4ec0 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.516+0000 7fbfeeffd640 1 -- 172.21.15.1:0/3526676873 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe0032440 con 0x7fbff00b4ec0 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.516+0000 7fbfeeffd640 1 -- 172.21.15.1:0/3526676873 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0032810 con 0x7fbff00b4ec0 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 -- 172.21.15.1:0/3526676873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 msgr2=0x7fbff00cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 --2- 172.21.15.1:0/3526676873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 0x7fbff00cf9d0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fbff00c8b40 tx=0x7fbfe002a270 comp rx=0 tx=0).stop 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 -- 172.21.15.1:0/3526676873 shutdown_connections 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 --2- 172.21.15.1:0/3526676873 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff0160290 0x7fbff01606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 --2- 172.21.15.1:0/3526676873 >> v2:172.21.15.196:3300/0 conn(0x7fbff00ce010 0x7fbff015fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 --2- 172.21.15.1:0/3526676873 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 0x7fbff00cf9d0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 -- 172.21.15.1:0/3526676873 >> 172.21.15.1:0/3526676873 conn(0x7fbff00d2360 msgr2=0x7fbff00ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 -- 172.21.15.1:0/3526676873 shutdown_connections 2024-05-09T05:07:38.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.517+0000 7fbff635e640 1 -- 172.21.15.1:0/3526676873 wait complete. 2024-05-09T05:07:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.520+0000 7fbff635e640 1 Processor -- start 2024-05-09T05:07:38.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.520+0000 7fbff635e640 1 -- start start 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbff635e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 0x7fbff01f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbff635e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff00ce010 0x7fbff015e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbff635e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0160290 0x7fbff0151c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbff635e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbff00cc750 con 0x7fbff00b4ec0 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbff635e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbff015cd50 con 0x7fbff0160290 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbff635e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbff00cff10 con 0x7fbff00ce010 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 0x7fbff01f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbfeffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 0x7fbff01f38c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48680/0 (socket says 172.21.15.1:48680) 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbfeffff640 1 -- 172.21.15.1:0/1886598722 learned_addr learned my addr 172.21.15.1:0/1886598722 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:38.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbfef7fe640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff00ce010 0x7fbff015e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.522+0000 7fbff48fb640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0160290 0x7fbff0151c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfeffff640 1 -- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff00ce010 msgr2=0x7fbff015e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfeffff640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbff00ce010 0x7fbff015e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfeffff640 1 -- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0160290 msgr2=0x7fbff0151c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfeffff640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0160290 0x7fbff0151c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfeffff640 1 -- 172.21.15.1:0/1886598722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbff015e900 con 0x7fbff00b4ec0 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfeffff640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 0x7fbff01f38c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbfe002a2b0 tx=0x7fbfe00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0041020 con 0x7fbff00b4ec0 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbfe003d620 con 0x7fbff00b4ec0 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbff015fc10 con 0x7fbff00b4ec0 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe0037040 con 0x7fbff00b4ec0 2024-05-09T05:07:38.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.523+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbff015dbc0 con 0x7fbff00b4ec0 2024-05-09T05:07:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.524+0000 7fbff635e640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff015ead0 0x7fbff0155280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.524+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbff015f9c0 con 0x7fbff015ead0 2024-05-09T05:07:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.525+0000 7fbfef7fe640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff015ead0 0x7fbff0155280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.526+0000 7fbfef7fe640 1 -- 172.21.15.1:0/1886598722 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbff00b63c0 con 0x7fbff015ead0 2024-05-09T05:07:38.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.526+0000 7fbfef7fe640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff015ead0 0x7fbff0155280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbff015b390 tx=0x7fbfe4011cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.526+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe400fa10 con 0x7fbff015ead0 2024-05-09T05:07:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.526+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fbfe400fbb0 con 0x7fbff015ead0 2024-05-09T05:07:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.526+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff015ead0 msgr2=0x7fbff0155280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.526+0000 7fbfecff9640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff015ead0 0x7fbff0155280 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fbff015b390 tx=0x7fbfe4011cf0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.526+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff015ead0 msgr2=0x7fbff0155280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:38.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.526+0000 7fbfecff9640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff015ead0 0x7fbff0155280 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.528+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbfe0044020 con 0x7fbff00b4ec0 2024-05-09T05:07:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.528+0000 7fbfecff9640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbfdc075d40 0x7fbfdc078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.528+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbfe00b7ae0 con 0x7fbff00b4ec0 2024-05-09T05:07:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.528+0000 7fbfef7fe640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbfdc075d40 0x7fbfdc078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.529+0000 7fbfef7fe640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbfdc075d40 0x7fbfdc078250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400f680 tx=0x7fbfe4008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.534+0000 7fbff635e640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0156180 0x7fbff01557c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.534+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbff00b44a0 con 0x7fbff0156180 2024-05-09T05:07:38.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.534+0000 7fbff48fb640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0156180 0x7fbff01557c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.535+0000 7fbff48fb640 1 -- 172.21.15.1:0/1886598722 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fbff01538b0 con 0x7fbff0156180 2024-05-09T05:07:38.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.535+0000 7fbff48fb640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0156180 0x7fbff01557c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbff01ed780 tx=0x7fbfe800f580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.535+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfe800fea0 con 0x7fbff0156180 2024-05-09T05:07:38.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.535+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4892 (secure 0 0 0) 0x7fbfe8003b90 con 0x7fbff0156180 2024-05-09T05:07:38.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.535+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0156180 msgr2=0x7fbff01557c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.535+0000 7fbfecff9640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0156180 0x7fbff01557c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fbff01ed780 tx=0x7fbfe800f580 comp rx=0 tx=0).stop 2024-05-09T05:07:38.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.535+0000 7fbfecff9640 1 -- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0156180 msgr2=0x7fbff01557c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:38.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.535+0000 7fbfecff9640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbff0156180 0x7fbff01557c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.536 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 14, 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 122625, 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:38.537 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.538 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.539 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:38.540 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:38.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.542 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.544 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:38.545 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:38.546 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:07:38.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.548 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:38.549 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:38.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbfdc075d40 msgr2=0x7fbfdc078250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff635e640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbfdc075d40 0x7fbfdc078250 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbfe400f680 tx=0x7fbfe4008030 comp rx=0 tx=0).stop 2024-05-09T05:07:38.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 msgr2=0x7fbff01f38c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff635e640 1 --2- 172.21.15.1:0/1886598722 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbff00b4ec0 0x7fbff01f38c0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fbfe002a2b0 tx=0x7fbfe00046c0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff48fb640 1 -- 172.21.15.1:0/1886598722 reap_dead start 2024-05-09T05:07:38.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 shutdown_connections 2024-05-09T05:07:38.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 >> 172.21.15.1:0/1886598722 conn(0x7fbff00d2360 msgr2=0x7fbff00b56a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:38.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 shutdown_connections 2024-05-09T05:07:38.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.536+0000 7fbff635e640 1 -- 172.21.15.1:0/1886598722 wait complete. 2024-05-09T05:07:38.551 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:07:38.551 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:07:38.551 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:07:38.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.688+0000 7f715eab1640 1 Processor -- start 2024-05-09T05:07:38.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.688+0000 7f715eab1640 1 -- start start 2024-05-09T05:07:38.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.689+0000 7f715eab1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c8b70 0x7f7158156b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.689+0000 7f715eab1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71580c9520 0x7f71580717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.689+0000 7f715eab1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7158071d30 0x7f7158159700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.689+0000 7f715eab1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f71580cb450 con 0x7f71580c9520 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.689+0000 7f715eab1640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f71580c9e50 con 0x7f7158071d30 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.689+0000 7f715eab1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71580caed0 con 0x7f71580c8b70 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.689+0000 7f7157fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71580c9520 0x7f71580717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f7157fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f71580c9520 0x7f71580717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48692/0 (socket says 172.21.15.1:48692) 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f7157fff640 1 -- 172.21.15.1:0/3015296577 learned_addr learned my addr 172.21.15.1:0/3015296577 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f715c84d640 1 --2- 172.21.15.1:0/3015296577 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c8b70 0x7f7158156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f715d04e640 1 --2- 172.21.15.1:0/3015296577 >> v2:172.21.15.196:3300/0 conn(0x7f7158071d30 0x7f7158159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f7157fff640 1 -- 172.21.15.1:0/3015296577 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c8b70 msgr2=0x7f7158156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f7157fff640 1 --2- 172.21.15.1:0/3015296577 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c8b70 0x7f7158156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f7157fff640 1 -- 172.21.15.1:0/3015296577 >> v2:172.21.15.196:3300/0 conn(0x7f7158071d30 msgr2=0x7f7158159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f7157fff640 1 --2- 172.21.15.1:0/3015296577 >> v2:172.21.15.196:3300/0 conn(0x7f7158071d30 0x7f7158159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f7157fff640 1 -- 172.21.15.1:0/3015296577 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71580cb170 con 0x7f71580c9520 2024-05-09T05:07:38.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f715d04e640 1 --2- 172.21.15.1:0/3015296577 >> v2:172.21.15.196:3300/0 conn(0x7f7158071d30 0x7f7158159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:07:38.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.690+0000 7f715c84d640 1 --2- 172.21.15.1:0/3015296577 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c8b70 0x7f7158156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.691+0000 7f7157fff640 1 --2- 172.21.15.1:0/3015296577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580c9520 0x7f71580717f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7158073f00 tx=0x7f714402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86a1d231b9586c73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.691+0000 7f71577fe640 1 -- 172.21.15.1:0/3015296577 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144037070 con 0x7f71580c9520 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.691+0000 7f71577fe640 1 -- 172.21.15.1:0/3015296577 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7144032440 con 0x7f71580c9520 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.691+0000 7f71577fe640 1 -- 172.21.15.1:0/3015296577 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144032810 con 0x7f71580c9520 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 -- 172.21.15.1:0/3015296577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580c9520 msgr2=0x7f71580717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 --2- 172.21.15.1:0/3015296577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580c9520 0x7f71580717f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f7158073f00 tx=0x7f714402a1e0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 -- 172.21.15.1:0/3015296577 shutdown_connections 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 --2- 172.21.15.1:0/3015296577 >> v2:172.21.15.196:3300/0 conn(0x7f7158071d30 0x7f7158159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 --2- 172.21.15.1:0/3015296577 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71580c9520 0x7f71580717f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 --2- 172.21.15.1:0/3015296577 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c8b70 0x7f7158156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 -- 172.21.15.1:0/3015296577 >> 172.21.15.1:0/3015296577 conn(0x7f715806d800 msgr2=0x7f715806dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:38.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 -- 172.21.15.1:0/3015296577 shutdown_connections 2024-05-09T05:07:38.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.692+0000 7f715eab1640 1 -- 172.21.15.1:0/3015296577 wait complete. 2024-05-09T05:07:38.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.694+0000 7f715eab1640 1 Processor -- start 2024-05-09T05:07:38.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.694+0000 7f715eab1640 1 -- start start 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715eab1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158071d30 0x7f71580c8360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715eab1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71580c8b70 0x7f7158159dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715eab1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c9520 0x7f71580c6b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715eab1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7158071210 con 0x7f7158071d30 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715eab1640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f71580c3f30 con 0x7f71580c8b70 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715eab1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71581570d0 con 0x7f71580c9520 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715c84d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158071d30 0x7f71580c8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715c84d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158071d30 0x7f71580c8360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48704/0 (socket says 172.21.15.1:48704) 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715c84d640 1 -- 172.21.15.1:0/667544982 learned_addr learned my addr 172.21.15.1:0/667544982 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:38.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f7157fff640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71580c8b70 0x7f7158159dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.696+0000 7f715d04e640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c9520 0x7f71580c6b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f7157fff640 1 -- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c9520 msgr2=0x7f71580c6b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f7157fff640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c9520 0x7f71580c6b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f7157fff640 1 -- 172.21.15.1:0/667544982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158071d30 msgr2=0x7f71580c8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f7157fff640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158071d30 0x7f71580c8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f7157fff640 1 -- 172.21.15.1:0/667544982 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71580c77c0 con 0x7f71580c8b70 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f715c84d640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7158071d30 0x7f71580c8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f7157fff640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71580c8b70 0x7f7158159dc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7144031000 tx=0x7f71440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144041020 con 0x7f71580c8b70 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7144030700 con 0x7f71580c8b70 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71580c7040 con 0x7f71580c8b70 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7144037040 con 0x7f71580c8b70 2024-05-09T05:07:38.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.697+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71580c7a30 con 0x7f71580c8b70 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.698+0000 7f715eab1640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c7c50 0x7f7158153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.698+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71581f1bb0 con 0x7f71580c7c50 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.699+0000 7f715c84d640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c7c50 0x7f7158153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.699+0000 7f715c84d640 1 -- 172.21.15.1:0/667544982 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71580696d0 con 0x7f71580c7c50 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.699+0000 7f715c84d640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c7c50 0x7f7158153470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f714c010c80 tx=0x7f714c00e5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.700+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f714c010e90 con 0x7f71580c7c50 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.700+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f714c004030 con 0x7f71580c7c50 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.700+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c7c50 msgr2=0x7f7158153470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.700+0000 7f71557fa640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c7c50 0x7f7158153470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f714c010c80 tx=0x7f714c00e5a0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.700+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c7c50 msgr2=0x7f7158153470 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:38.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.700+0000 7f71557fa640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71580c7c50 0x7f7158153470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.700+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7144044020 con 0x7f71580c8b70 2024-05-09T05:07:38.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.701+0000 7f71557fa640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f713c075f10 0x7f713c078420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.701+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f71440b7e80 con 0x7f71580c8b70 2024-05-09T05:07:38.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.701+0000 7f715c84d640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f713c075f10 0x7f713c078420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.702+0000 7f715c84d640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f713c075f10 0x7f713c078420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f714c00eb40 tx=0x7f714c015000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.712+0000 7f715eab1640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f715814a4d0 0x7f715814a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.712+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7158156450 con 0x7f715814a4d0 2024-05-09T05:07:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.712+0000 7f715d04e640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f715814a4d0 0x7f715814a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.713+0000 7f715d04e640 1 -- 172.21.15.1:0/667544982 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7158050870 con 0x7f715814a4d0 2024-05-09T05:07:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.713+0000 7f715d04e640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f715814a4d0 0x7f715814a8e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71581e8f80 tx=0x7f7150009f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.713+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7150007c80 con 0x7f715814a4d0 2024-05-09T05:07:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.713+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f7150007c80 con 0x7f715814a4d0 2024-05-09T05:07:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.713+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f715814a4d0 msgr2=0x7f715814a8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.713+0000 7f71557fa640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f715814a4d0 0x7f715814a8e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f71581e8f80 tx=0x7f7150009f80 comp rx=0 tx=0).stop 2024-05-09T05:07:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f71557fa640 1 -- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f715814a4d0 msgr2=0x7f715814a8e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:38.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f71557fa640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f715814a4d0 0x7f715814a8e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 14, 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 16710, 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:38.715 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.716 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:38.717 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:38.718 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.719 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.720 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.721 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.722 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:38.723 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.724 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:38.725 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f713c075f10 msgr2=0x7f713c078420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f713c075f10 0x7f713c078420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f714c00eb40 tx=0x7f714c015000 comp rx=0 tx=0).stop 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715d04e640 1 -- 172.21.15.1:0/667544982 reap_dead start 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71580c8b70 msgr2=0x7f7158159dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71580c8b70 0x7f7158159dc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f7144031000 tx=0x7f71440046c0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 shutdown_connections 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 --2- 172.21.15.1:0/667544982 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71580c8b70 0x7f7158159dc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 >> 172.21.15.1:0/667544982 conn(0x7f715806d800 msgr2=0x7f7158150be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 shutdown_connections 2024-05-09T05:07:38.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.714+0000 7f715eab1640 1 -- 172.21.15.1:0/667544982 wait complete. 2024-05-09T05:07:38.728 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:07:38.728 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-05-09T05:07:38.728 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:07:38.728 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-05-09T05:07:38.728 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-05-09T05:07:38.728 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T05:07:38.729 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:07:38.744 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T05:07:38.744 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:07:38.744 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:07:38.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.937+0000 7f34984f7640 1 Processor -- start 2024-05-09T05:07:38.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.938+0000 7f34984f7640 1 -- start start 2024-05-09T05:07:38.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.939+0000 7f34984f7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f3490070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.939+0000 7f34984f7640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3490070530 0x7f349015d3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.939+0000 7f34984f7640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34900c4430 con 0x7f3490153830 2024-05-09T05:07:38.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.939+0000 7f34984f7640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f34900c3b90 con 0x7f3490070530 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.939+0000 7f34984f7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f34900c3e70 con 0x7f3490159440 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3496293640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f3490070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3496293640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f3490070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:44306/0 (socket says 172.21.15.1:44306) 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3495a92640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3490070530 0x7f349015d3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3496293640 1 -- 172.21.15.1:0/904616017 learned_addr learned my addr 172.21.15.1:0/904616017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3496293640 1 -- 172.21.15.1:0/904616017 >> v2:172.21.15.196:3300/0 conn(0x7f3490070530 msgr2=0x7f349015d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3496293640 1 --2- 172.21.15.1:0/904616017 >> v2:172.21.15.196:3300/0 conn(0x7f3490070530 0x7f349015d3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3496293640 1 -- 172.21.15.1:0/904616017 >> v1:172.21.15.196:6789/0 conn(0x7f3490153830 legacy=0x7f34900cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3496293640 1 -- 172.21.15.1:0/904616017 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34900c4150 con 0x7f3490159440 2024-05-09T05:07:38.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.940+0000 7f3495a92640 1 --2- 172.21.15.1:0/904616017 >> v2:172.21.15.196:3300/0 conn(0x7f3490070530 0x7f349015d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:07:38.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.941+0000 7f3496293640 1 --2- 172.21.15.1:0/904616017 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f3490070ae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f348c009e50 tx=0x7f348c02c9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ddd9121c3511c96 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.941+0000 7f3495291640 1 -- 172.21.15.1:0/904616017 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c035020 con 0x7f3490159440 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.941+0000 7f3495291640 1 -- 172.21.15.1:0/904616017 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348c031400 con 0x7f3490159440 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.941+0000 7f3495291640 1 -- 172.21.15.1:0/904616017 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c0317d0 con 0x7f3490159440 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.942+0000 7f34984f7640 1 -- 172.21.15.1:0/904616017 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 msgr2=0x7f3490070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.942+0000 7f34984f7640 1 --2- 172.21.15.1:0/904616017 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f3490070ae0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f348c009e50 tx=0x7f348c02c9b0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.942+0000 7f34984f7640 1 -- 172.21.15.1:0/904616017 shutdown_connections 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.942+0000 7f34984f7640 1 --2- 172.21.15.1:0/904616017 >> v2:172.21.15.196:3300/0 conn(0x7f3490070530 0x7f349015d3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.942+0000 7f34984f7640 1 --2- 172.21.15.1:0/904616017 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f3490070ae0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.942+0000 7f34984f7640 1 -- 172.21.15.1:0/904616017 >> 172.21.15.1:0/904616017 conn(0x7f3490071b40 msgr2=0x7f3490071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.942+0000 7f34984f7640 1 -- 172.21.15.1:0/904616017 shutdown_connections 2024-05-09T05:07:38.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.942+0000 7f34984f7640 1 -- 172.21.15.1:0/904616017 wait complete. 2024-05-09T05:07:38.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.944+0000 7f34984f7640 1 Processor -- start 2024-05-09T05:07:38.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.944+0000 7f34984f7640 1 -- start start 2024-05-09T05:07:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f34984f7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3490070530 0x7f34901ee490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f34984f7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3490153830 0x7f349013e290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f34984f7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f349013c020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f34984f7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f349015d910 con 0x7f3490153830 2024-05-09T05:07:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f34984f7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f349015ceb0 con 0x7f3490070530 2024-05-09T05:07:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f3495a92640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3490153830 msgr2=0x7f349013e290 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:38.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f34984f7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f34900c3870 con 0x7f3490159440 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f3495a92640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3490153830 0x7f349013e290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f3496293640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3490070530 0x7f34901ee490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f3496a94640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f349013c020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f3496293640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3490070530 0x7f34901ee490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:54300/0 (socket says 172.21.15.1:54300) 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.945+0000 7f3496293640 1 -- 172.21.15.1:0/879605328 learned_addr learned my addr 172.21.15.1:0/879605328 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3496293640 1 -- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 msgr2=0x7f349013c020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3496293640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f349013c020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3496293640 1 -- 172.21.15.1:0/879605328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3490153830 msgr2=0x7f349013e290 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3496293640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3490153830 0x7f349013e290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3496293640 1 -- 172.21.15.1:0/879605328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34901f1aa0 con 0x7f3490070530 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3496a94640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f349013c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3496293640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3490070530 0x7f34901ee490 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f348c009f70 tx=0x7f348c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3486ffd640 1 -- 172.21.15.1:0/879605328 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c040070 con 0x7f3490070530 2024-05-09T05:07:38.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3486ffd640 1 -- 172.21.15.1:0/879605328 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f348c0079b0 con 0x7f3490070530 2024-05-09T05:07:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34901f1d40 con 0x7f3490070530 2024-05-09T05:07:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34901f2300 con 0x7f3490070530 2024-05-09T05:07:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.946+0000 7f3486ffd640 1 -- 172.21.15.1:0/879605328 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f348c035040 con 0x7f3490070530 2024-05-09T05:07:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.947+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3490050870 con 0x7f3490070530 2024-05-09T05:07:38.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.949+0000 7f3486ffd640 1 -- 172.21.15.1:0/879605328 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f348c038020 con 0x7f3490070530 2024-05-09T05:07:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.950+0000 7f3486ffd640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34740762b0 0x7f34740787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.950+0000 7f3486ffd640 1 -- 172.21.15.1:0/879605328 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f348c0b9810 con 0x7f3490070530 2024-05-09T05:07:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.950+0000 7f3495a92640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34740762b0 0x7f34740787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:38.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.950+0000 7f3495a92640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34740762b0 0x7f34740787c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f349013ec00 tx=0x7f3480009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:38.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:38.955+0000 7f3486ffd640 1 -- 172.21.15.1:0/879605328 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f348c083180 con 0x7f3490070530 2024-05-09T05:07:39.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.248+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f34900c94c0 con 0x7f3490070530 2024-05-09T05:07:39.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.248+0000 7f3486ffd640 1 -- 172.21.15.1:0/879605328 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f348c082a80 con 0x7f3490070530 2024-05-09T05:07:39.249 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:07:39.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34740762b0 msgr2=0x7f34740787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34740762b0 0x7f34740787c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f349013ec00 tx=0x7f3480009190 comp rx=0 tx=0).stop 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3490070530 msgr2=0x7f34901ee490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3490070530 0x7f34901ee490 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f348c009f70 tx=0x7f348c004a40 comp rx=0 tx=0).stop 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 shutdown_connections 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34740762b0 0x7f34740787c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3490159440 0x7f349013c020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3490153830 0x7f349013e290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 --2- 172.21.15.1:0/879605328 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3490070530 0x7f34901ee490 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 >> 172.21.15.1:0/879605328 conn(0x7f3490071b40 msgr2=0x7f34900ce8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 shutdown_connections 2024-05-09T05:07:39.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:39.251+0000 7f34984f7640 1 -- 172.21.15.1:0/879605328 wait complete. 2024-05-09T05:07:42.268 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:07:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.407+0000 7f71ed31f640 1 Processor -- start 2024-05-09T05:07:42.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.407+0000 7f71ed31f640 1 -- start start 2024-05-09T05:07:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.409+0000 7f71ed31f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80719b0 0x7f71e815b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.409+0000 7f71ed31f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e80c5f40 con 0x7f71e80cb700 2024-05-09T05:07:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.409+0000 7f71ed31f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71e80c56a0 con 0x7f71e80cbb10 2024-05-09T05:07:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.409+0000 7f71ed31f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71e80c5980 con 0x7f71e80719b0 2024-05-09T05:07:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.409+0000 7f71e67fc640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f71e80cb700 legacy=0x7f71e80ca3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:07:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.409+0000 7f71e6ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80719b0 0x7f71e815b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.409+0000 7f71e77fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f71e80cbb10 0x7f71e8165000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:48534/0 (socket says 172.21.15.1:48534) 2024-05-09T05:07:42.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.409+0000 7f71e77fe640 1 -- 172.21.15.1:0/1510356528 learned_addr learned my addr 172.21.15.1:0/1510356528 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:07:42.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.410+0000 7f71e5ffb640 1 -- 172.21.15.1:0/1510356528 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3076293103 0 0) 0x7f71e80c56a0 con 0x7f71e80cbb10 2024-05-09T05:07:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.410+0000 7f71e5ffb640 1 -- 172.21.15.1:0/1510356528 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71d0003520 con 0x7f71e80cbb10 2024-05-09T05:07:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.410+0000 7f71e6ffd640 1 -- 172.21.15.1:0/1510356528 >> v1:172.21.15.196:6789/0 conn(0x7f71e80cbb10 legacy=0x7f71e8165000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.410+0000 7f71e6ffd640 1 -- 172.21.15.1:0/1510356528 >> v1:172.21.15.1:6789/0 conn(0x7f71e80cb700 legacy=0x7f71e80ca3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.410+0000 7f71e6ffd640 1 -- 172.21.15.1:0/1510356528 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e80c5c60 con 0x7f71e80719b0 2024-05-09T05:07:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.411+0000 7f71e6ffd640 1 --2- 172.21.15.1:0/1510356528 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80719b0 0x7f71e815b070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f71e80721b0 tx=0x7f71d802a0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=849c7d626967b7e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.411+0000 7f71e5ffb640 1 -- 172.21.15.1:0/1510356528 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8037070 con 0x7f71e80719b0 2024-05-09T05:07:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.411+0000 7f71e5ffb640 1 -- 172.21.15.1:0/1510356528 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71d802ae20 con 0x7f71e80719b0 2024-05-09T05:07:42.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.411+0000 7f71e5ffb640 1 -- 172.21.15.1:0/1510356528 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71d8032640 con 0x7f71e80719b0 2024-05-09T05:07:42.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.412+0000 7f71ed31f640 1 -- 172.21.15.1:0/1510356528 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80719b0 msgr2=0x7f71e815b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:42.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.412+0000 7f71ed31f640 1 --2- 172.21.15.1:0/1510356528 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80719b0 0x7f71e815b070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f71e80721b0 tx=0x7f71d802a0d0 comp rx=0 tx=0).stop 2024-05-09T05:07:42.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.412+0000 7f71ed31f640 1 -- 172.21.15.1:0/1510356528 shutdown_connections 2024-05-09T05:07:42.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.412+0000 7f71ed31f640 1 --2- 172.21.15.1:0/1510356528 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80719b0 0x7f71e815b070 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:42.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.412+0000 7f71ed31f640 1 -- 172.21.15.1:0/1510356528 >> 172.21.15.1:0/1510356528 conn(0x7f71e80cc6a0 msgr2=0x7f71e80ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:42.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.412+0000 7f71ed31f640 1 -- 172.21.15.1:0/1510356528 shutdown_connections 2024-05-09T05:07:42.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.412+0000 7f71ed31f640 1 -- 172.21.15.1:0/1510356528 wait complete. 2024-05-09T05:07:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.415+0000 7f71ed31f640 1 Processor -- start 2024-05-09T05:07:42.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.415+0000 7f71ed31f640 1 -- start start 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71ed31f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71e80719b0 0x7f71e80cdd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71ed31f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80cb700 0x7f71e81eefd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71ed31f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80cbb10 0x7f71e80cf520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71e67fc640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80cb700 msgr2=0x7f71e81eefd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71ed31f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f71e80cb3c0 con 0x7f71e80cb700 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71e67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80cb700 0x7f71e81eefd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71ed31f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f71e815ab50 con 0x7f71e80719b0 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71ed31f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71e80cbf20 con 0x7f71e80cbb10 2024-05-09T05:07:42.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71e77fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80cbb10 0x7f71e80cf520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71e77fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80cbb10 0x7f71e80cf520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:37724/0 (socket says 172.21.15.1:37724) 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71e6ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71e80719b0 0x7f71e80cdd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.417+0000 7f71e77fe640 1 -- 172.21.15.1:0/3655008068 learned_addr learned my addr 172.21.15.1:0/3655008068 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71e77fe640 1 -- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71e80719b0 msgr2=0x7f71e80cdd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71e77fe640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71e80719b0 0x7f71e80cdd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71e77fe640 1 -- 172.21.15.1:0/3655008068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80cb700 msgr2=0x7f71e81eefd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71e77fe640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80cb700 0x7f71e81eefd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71e77fe640 1 -- 172.21.15.1:0/3655008068 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71e80ce240 con 0x7f71e80cbb10 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71e77fe640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80cbb10 0x7f71e80cf520 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f71e000ca70 tx=0x7f71e000cf30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71cf7fe640 1 -- 172.21.15.1:0/3655008068 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e000abd0 con 0x7f71e80cbb10 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71cf7fe640 1 -- 172.21.15.1:0/3655008068 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e000ad70 con 0x7f71e80cbb10 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71e81550a0 con 0x7f71e80cbb10 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71cf7fe640 1 -- 172.21.15.1:0/3655008068 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e0012620 con 0x7f71e80cbb10 2024-05-09T05:07:42.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.418+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f71e8155660 con 0x7f71e80cbb10 2024-05-09T05:07:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.420+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f71e806f8a0 con 0x7f71e80cbb10 2024-05-09T05:07:42.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.421+0000 7f71cf7fe640 1 -- 172.21.15.1:0/3655008068 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f71e00127c0 con 0x7f71e80cbb10 2024-05-09T05:07:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.422+0000 7f71cf7fe640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4076280 0x7f71d4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.422+0000 7f71cf7fe640 1 -- 172.21.15.1:0/3655008068 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f71e0097b00 con 0x7f71e80cbb10 2024-05-09T05:07:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.422+0000 7f71e6ffd640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4076280 0x7f71d4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:42.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.422+0000 7f71e6ffd640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4076280 0x7f71d4078790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f71d8034000 tx=0x7f71d802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:42.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.427+0000 7f71cf7fe640 1 -- 172.21.15.1:0/3655008068 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f71e806f8a0 con 0x7f71e80cbb10 2024-05-09T05:07:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.693+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f71e8050870 con 0x7f71e80cbb10 2024-05-09T05:07:42.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.693+0000 7f71cf7fe640 1 -- 172.21.15.1:0/3655008068 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f71e0061470 con 0x7f71e80cbb10 2024-05-09T05:07:42.694 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:07:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4076280 msgr2=0x7f71d4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4076280 0x7f71d4078790 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f71d8034000 tx=0x7f71d802e000 comp rx=0 tx=0).stop 2024-05-09T05:07:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80cbb10 msgr2=0x7f71e80cf520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80cbb10 0x7f71e80cf520 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f71e000ca70 tx=0x7f71e000cf30 comp rx=0 tx=0).stop 2024-05-09T05:07:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 shutdown_connections 2024-05-09T05:07:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4076280 0x7f71d4078790 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71e80cbb10 0x7f71e80cf520 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:42.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71e80cb700 0x7f71e81eefd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 --2- 172.21.15.1:0/3655008068 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71e80719b0 0x7f71e80cdd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 >> 172.21.15.1:0/3655008068 conn(0x7f71e80cc6a0 msgr2=0x7f71e814e4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 shutdown_connections 2024-05-09T05:07:42.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:42.695+0000 7f71ed31f640 1 -- 172.21.15.1:0/3655008068 wait complete. 2024-05-09T05:07:45.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:07:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.825+0000 7ff8a8f34640 1 Processor -- start 2024-05-09T05:07:45.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.825+0000 7ff8a8f34640 1 -- start start 2024-05-09T05:07:45.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a8f34640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8a40d0fd0 0x7ff8a415df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:45.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a8f34640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:45.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a259c640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7ff8a4161790 legacy=0x7ff8a40c8f50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:07:45.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a1d9b640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7ff8a40d0fd0 msgr2=0x7ff8a415df20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:45.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a8f34640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff8a40caec0 con 0x7ff8a4161790 2024-05-09T05:07:45.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a8f34640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff8a40cb1a0 con 0x7ff8a40d0fd0 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a8f34640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a40cb480 con 0x7ff8a40d13e0 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a1d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff8a40d0fd0 0x7ff8a415df20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a2d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a2d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:37738/0 (socket says 172.21.15.1:37738) 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.827+0000 7ff8a2d9d640 1 -- 172.21.15.1:0/2122960045 learned_addr learned my addr 172.21.15.1:0/2122960045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.828+0000 7ff8a2d9d640 1 -- 172.21.15.1:0/2122960045 >> v2:172.21.15.1:3300/0 conn(0x7ff8a40d0fd0 msgr2=0x7ff8a415df20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.828+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/2122960045 >> v2:172.21.15.1:3300/0 conn(0x7ff8a40d0fd0 0x7ff8a415df20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.828+0000 7ff8a2d9d640 1 -- 172.21.15.1:0/2122960045 >> v1:172.21.15.1:6789/0 conn(0x7ff8a4161790 legacy=0x7ff8a40c8f50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:45.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.828+0000 7ff8a2d9d640 1 -- 172.21.15.1:0/2122960045 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a40cb760 con 0x7ff8a40d13e0 2024-05-09T05:07:45.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.828+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/2122960045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415e460 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009e50 tx=0x7ff89c02a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3650b6ad80d051aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.829+0000 7ff8a159a640 1 -- 172.21.15.1:0/2122960045 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c036070 con 0x7ff8a40d13e0 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.829+0000 7ff8a159a640 1 -- 172.21.15.1:0/2122960045 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff89c031600 con 0x7ff8a40d13e0 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.829+0000 7ff8a159a640 1 -- 172.21.15.1:0/2122960045 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff89c0319d0 con 0x7ff8a40d13e0 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.829+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2122960045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 msgr2=0x7ff8a415e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.829+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2122960045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415e460 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff89c009e50 tx=0x7ff89c02a6a0 comp rx=0 tx=0).stop 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.830+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2122960045 shutdown_connections 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.830+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2122960045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415e460 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.830+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2122960045 >> v2:172.21.15.1:3300/0 conn(0x7ff8a40d0fd0 0x7ff8a415df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.830+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2122960045 >> 172.21.15.1:0/2122960045 conn(0x7ff8a40b3660 msgr2=0x7ff8a40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.830+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2122960045 shutdown_connections 2024-05-09T05:07:45.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.830+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2122960045 wait complete. 2024-05-09T05:07:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.832+0000 7ff8a8f34640 1 Processor -- start 2024-05-09T05:07:45.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.832+0000 7ff8a8f34640 1 -- start start 2024-05-09T05:07:45.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.833+0000 7ff8a8f34640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40d0fd0 0x7ff8a40b58b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:45.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.833+0000 7ff8a8f34640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:45.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.833+0000 7ff8a259c640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40d0fd0 msgr2=0x7ff8a40b58b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:45.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.833+0000 7ff8a8f34640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8a4161790 0x7ff8a415a5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:45.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.833+0000 7ff8a259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40d0fd0 0x7ff8a40b58b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.833+0000 7ff8a8f34640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40c97c0 con 0x7ff8a40d0fd0 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.833+0000 7ff8a8f34640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff8a40ca900 con 0x7ff8a4161790 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.833+0000 7ff8a8f34640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff8a4161ba0 con 0x7ff8a40d13e0 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:37750/0 (socket says 172.21.15.1:37750) 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 -- 172.21.15.1:0/2229565666 learned_addr learned my addr 172.21.15.1:0/2229565666 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a2d9d640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8a4161790 0x7ff8a415a5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 -- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8a4161790 msgr2=0x7ff8a415a5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8a4161790 0x7ff8a415a5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 -- 172.21.15.1:0/2229565666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40d0fd0 msgr2=0x7ff8a40b58b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40d0fd0 0x7ff8a40b58b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 -- 172.21.15.1:0/2229565666 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8a40b7450 con 0x7ff8a40d13e0 2024-05-09T05:07:45.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff8a1d9b640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415a080 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff89800b450 tx=0x7ff89800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.834+0000 7ff892ffd640 1 -- 172.21.15.1:0/2229565666 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898003dc0 con 0x7ff8a40d13e0 2024-05-09T05:07:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.835+0000 7ff892ffd640 1 -- 172.21.15.1:0/2229565666 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff898015070 con 0x7ff8a40d13e0 2024-05-09T05:07:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.835+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8a40b63b0 con 0x7ff8a40d13e0 2024-05-09T05:07:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.835+0000 7ff892ffd640 1 -- 172.21.15.1:0/2229565666 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff898005290 con 0x7ff8a40d13e0 2024-05-09T05:07:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.835+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8a40b6970 con 0x7ff8a40d13e0 2024-05-09T05:07:45.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.835+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8a4050870 con 0x7ff8a40d13e0 2024-05-09T05:07:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.837+0000 7ff892ffd640 1 -- 172.21.15.1:0/2229565666 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff89800f070 con 0x7ff8a40d13e0 2024-05-09T05:07:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.837+0000 7ff892ffd640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff884076280 0x7ff884078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.837+0000 7ff892ffd640 1 -- 172.21.15.1:0/2229565666 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff8980978d0 con 0x7ff8a40d13e0 2024-05-09T05:07:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.837+0000 7ff8a259c640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff884076280 0x7ff884078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:45.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.838+0000 7ff8a259c640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff884076280 0x7ff884078790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff894006fd0 tx=0x7ff894006d80 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:45.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:45.841+0000 7ff892ffd640 1 -- 172.21.15.1:0/2229565666 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff89800d040 con 0x7ff8a40d13e0 2024-05-09T05:07:46.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.208+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff8a40b7ad0 con 0x7ff8a40d13e0 2024-05-09T05:07:46.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.208+0000 7ff892ffd640 1 -- 172.21.15.1:0/2229565666 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff898061290 con 0x7ff8a40d13e0 2024-05-09T05:07:46.209 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":14,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:07:46.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff884076280 msgr2=0x7ff884078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:46.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff884076280 0x7ff884078790 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7ff894006fd0 tx=0x7ff894006d80 comp rx=0 tx=0).stop 2024-05-09T05:07:46.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 msgr2=0x7ff8a415a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415a080 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff89800b450 tx=0x7ff89800b910 comp rx=0 tx=0).stop 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 shutdown_connections 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff884076280 0x7ff884078790 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8a4161790 0x7ff8a415a5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8a40d13e0 0x7ff8a415a080 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 --2- 172.21.15.1:0/2229565666 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8a40d0fd0 0x7ff8a40b58b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.210+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 >> 172.21.15.1:0/2229565666 conn(0x7ff8a40b3660 msgr2=0x7ff8a40b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.211+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 shutdown_connections 2024-05-09T05:07:46.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:46.211+0000 7ff8a8f34640 1 -- 172.21.15.1:0/2229565666 wait complete. 2024-05-09T05:07:49.223 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:07:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.340+0000 7f41a2d0d640 1 Processor -- start 2024-05-09T05:07:49.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.341+0000 7f41a2d0d640 1 -- start start 2024-05-09T05:07:49.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f41a2d0d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:49.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f41a2d0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:49.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f41a2d0d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419c06cee0 con 0x7f419c0ce8a0 2024-05-09T05:07:49.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f41a2d0d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f419c0cb200 con 0x7f419c0c9310 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f41a2d0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f419c06d1c0 con 0x7f419c15ada0 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f41a12aa640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f419c0ce8a0 legacy=0x7f419c160be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f419bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f41a0aa9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f419bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:37760/0 (socket says 172.21.15.1:37760) 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.342+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 learned_addr learned my addr 172.21.15.1:0/819136759 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.343+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.343+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.343+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.343+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.343+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.343+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.343+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.343+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:49.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.543+0000 7f41a12aa640 1 -- 172.21.15.1:0/819136759 >> v1:172.21.15.1:6789/0 conn(0x7f419c0ce8a0 legacy=0x7f419c160be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.544+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:07:49.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.944+0000 7f41a12aa640 1 -- 172.21.15.1:0/819136759 >> v1:172.21.15.1:6789/0 conn(0x7f419c0ce8a0 legacy=0x7f419c160be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:07:49.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:49.945+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:07:50.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.745+0000 7f41a12aa640 1 -- 172.21.15.1:0/819136759 >> v1:172.21.15.1:6789/0 conn(0x7f419c0ce8a0 legacy=0x7f419c160be0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:07:50.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:50.746+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:07:52.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f419a7fc640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 msgr2=0x7f419c0cbc40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:52.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f419a7fc640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f419c15ada0 0x7f419c0cbc40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f419a7fc640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 msgr2=0x7f419c164ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f419a7fc640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.196:3300/0 conn(0x7f419c0c9310 0x7f419c164ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f419a7fc640 1 -- 172.21.15.1:0/819136759 >> v1:172.21.15.1:6789/0 conn(0x7f419c0ce8a0 legacy=0x7f419c160be0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f419a7fc640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 0x7f4180005b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f419a7fc640 1 -- 172.21.15.1:0/819136759 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f419c06d4a0 con 0x7f4180006230 2024-05-09T05:07:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f419a7fc640 1 -- 172.21.15.1:0/819136759 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f419c0cb200 con 0x7f4180003680 2024-05-09T05:07:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 msgr2=0x7f4180005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:52.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.342+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 0x7f4180005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.543+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 msgr2=0x7f4180005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:52.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.543+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 0x7f4180005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:07:52.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.944+0000 7f41a0aa9640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 msgr2=0x7f4180005b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:52.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:52.944+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 0x7f4180005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:07:53.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.039+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3207462544 0 0) 0x7f419c06d4a0 con 0x7f4180006230 2024-05-09T05:07:53.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.040+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f418c003520 con 0x7f4180006230 2024-05-09T05:07:53.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.043+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 678648101 0 0) 0x7f418c003520 con 0x7f4180006230 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.043+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 msgr2=0x7f4180005b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.043+0000 7f419b7fe640 1 --2- 172.21.15.1:0/819136759 >> v2:172.21.15.1:3300/0 conn(0x7f4180003680 0x7f4180005b50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.043+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c06cee0 con 0x7f4180006230 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.043+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f418800f040 con 0x7f4180006230 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.043+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 >> v1:172.21.15.196:6789/0 conn(0x7f4180006230 legacy=0x7f4180008740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.043+0000 7f41a12aa640 1 -- 172.21.15.1:0/819136759 reap_dead start 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419b7fe640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c0039b0 0x7f418c003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419b7fe640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c004800 0x7f418c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419b7fe640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 0x7f418c00bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f41a12aa640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 msgr2=0x7f418c00bdd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f41a12aa640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 0x7f418c00bdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:53.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f419c0cb200 con 0x7f418c0099b0 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f418c00c3c0 con 0x7f418c004800 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f418c00c5c0 con 0x7f418c0039b0 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c004800 0x7f418c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c0039b0 0x7f418c003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c0039b0 msgr2=0x7f418c003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c0039b0 0x7f418c003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 msgr2=0x7f418c00bdd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 0x7f418c00bdd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.044+0000 7f419bfff640 1 -- 172.21.15.1:0/819136759 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f418c003520 con 0x7f418c004800 2024-05-09T05:07:53.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.045+0000 7f419bfff640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c004800 0x7f418c004c10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f419c071500 tx=0x7f418c005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=289de54189f57665 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.046+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188031070 con 0x7f418c004800 2024-05-09T05:07:53.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.046+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f418802bc50 con 0x7f418c004800 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.046+0000 7f419b7fe640 1 -- 172.21.15.1:0/819136759 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418800c9f0 con 0x7f418c004800 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 -- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c004800 msgr2=0x7f418c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c004800 0x7f418c004c10 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f419c071500 tx=0x7f418c005830 comp rx=0 tx=0).stop 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 -- 172.21.15.1:0/819136759 shutdown_connections 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c004800 0x7f418c004c10 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 0x7f418c00bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/819136759 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c0039b0 0x7f418c003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 -- 172.21.15.1:0/819136759 >> 172.21.15.1:0/819136759 conn(0x7f419c06e400 msgr2=0x7f419c071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 -- 172.21.15.1:0/819136759 shutdown_connections 2024-05-09T05:07:53.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.048+0000 7f41a2d0d640 1 -- 172.21.15.1:0/819136759 wait complete. 2024-05-09T05:07:53.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.051+0000 7f41a2d0d640 1 Processor -- start 2024-05-09T05:07:53.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.051+0000 7f41a2d0d640 1 -- start start 2024-05-09T05:07:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a2d0d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c0039b0 0x7f419c1f6870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a2d0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c004800 0x7f419c157aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a2d0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 0x7f419c1554d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a2d0d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f419c074190 con 0x7f418c0099b0 2024-05-09T05:07:53.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a2d0d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f419c0cb720 con 0x7f418c0039b0 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a2d0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f419c074310 con 0x7f418c004800 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a12aa640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 msgr2=0x7f419c1554d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a12aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 0x7f419c1554d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f419bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c004800 0x7f419c157aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a0aa9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c0039b0 0x7f419c1f6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f419bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c004800 0x7f419c157aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:35846/0 (socket says 172.21.15.1:35846) 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f419bfff640 1 -- 172.21.15.1:0/370136752 learned_addr learned my addr 172.21.15.1:0/370136752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.053+0000 7f41a0aa9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c0039b0 0x7f419c1f6870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:60282/0 (socket says 172.21.15.1:60282) 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f419bfff640 1 -- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c0039b0 msgr2=0x7f419c1f6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f419bfff640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c0039b0 0x7f419c1f6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f419bfff640 1 -- 172.21.15.1:0/370136752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 msgr2=0x7f419c1554d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f419bfff640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 0x7f419c1554d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f419bfff640 1 -- 172.21.15.1:0/370136752 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c157fe0 con 0x7f418c004800 2024-05-09T05:07:53.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c0039b0 0x7f419c1f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:07:53.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f419bfff640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c004800 0x7f419c157aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f418800ce90 tx=0x7f41880046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f41997fa640 1 -- 172.21.15.1:0/370136752 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4188041020 con 0x7f418c004800 2024-05-09T05:07:53.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f41997fa640 1 -- 172.21.15.1:0/370136752 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f418802ba50 con 0x7f418c004800 2024-05-09T05:07:53.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.054+0000 7f41997fa640 1 -- 172.21.15.1:0/370136752 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418800f040 con 0x7f418c004800 2024-05-09T05:07:53.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.055+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419c0cac70 con 0x7f418c004800 2024-05-09T05:07:53.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.055+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f419c156ce0 con 0x7f418c004800 2024-05-09T05:07:53.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.056+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f419c050870 con 0x7f418c004800 2024-05-09T05:07:53.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.057+0000 7f41997fa640 1 -- 172.21.15.1:0/370136752 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4188044020 con 0x7f418c004800 2024-05-09T05:07:53.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.058+0000 7f41997fa640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41800769d0 0x7f4180078ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.058+0000 7f41997fa640 1 -- 172.21.15.1:0/370136752 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f41880b80f0 con 0x7f418c004800 2024-05-09T05:07:53.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.058+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41800769d0 0x7f4180078ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.059+0000 7f41a0aa9640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41800769d0 0x7f4180078ee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4190006fd0 tx=0x7f4190008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.063+0000 7f41997fa640 1 -- 172.21.15.1:0/370136752 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f418804b030 con 0x7f418c004800 2024-05-09T05:07:53.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.474+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f419c0c54d0 con 0x7f418c004800 2024-05-09T05:07:53.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.475+0000 7f41997fa640 1 -- 172.21.15.1:0/370136752 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f41880819c0 con 0x7f418c004800 2024-05-09T05:07:53.476 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":16,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:07:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41800769d0 msgr2=0x7f4180078ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41800769d0 0x7f4180078ee0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f4190006fd0 tx=0x7f4190008040 comp rx=0 tx=0).stop 2024-05-09T05:07:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c004800 msgr2=0x7f419c157aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c004800 0x7f419c157aa0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f418800ce90 tx=0x7f41880046c0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 shutdown_connections 2024-05-09T05:07:53.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f41800769d0 0x7f4180078ee0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f418c0099b0 0x7f419c1554d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f418c004800 0x7f419c157aa0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 --2- 172.21.15.1:0/370136752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f418c0039b0 0x7f419c1f6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 >> 172.21.15.1:0/370136752 conn(0x7f419c06e400 msgr2=0x7f419c158930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:53.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 shutdown_connections 2024-05-09T05:07:53.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.477+0000 7f41a2d0d640 1 -- 172.21.15.1:0/370136752 wait complete. 2024-05-09T05:07:53.491 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:07:53.491 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:07:53.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:07:53.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.609+0000 7fc19ee61640 1 Processor -- start 2024-05-09T05:07:53.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.609+0000 7fc19ee61640 1 -- start start 2024-05-09T05:07:53.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19ee61640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc1980c5670 0x7fc198153830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19ee61640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1981594d0 con 0x7fc1980cc260 2024-05-09T05:07:53.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19ee61640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc1980c85a0 con 0x7fc19806c6f0 2024-05-09T05:07:53.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19ee61640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc1980cc980 con 0x7fc1980c5670 2024-05-09T05:07:53.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19cbfd640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fc1980cc260 legacy=0x7fc1980c5f90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:07:53.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19d3fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc1980c5670 0x7fc198153830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19d3fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc1980c5670 0x7fc198153830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:60288/0 (socket says 172.21.15.1:60288) 2024-05-09T05:07:53.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc197fff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc19806c6f0 0x7fc1980c66a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:44320/0 (socket says 172.21.15.1:44320) 2024-05-09T05:07:53.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19d3fe640 1 -- 172.21.15.1:0/1400088415 learned_addr learned my addr 172.21.15.1:0/1400088415 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.612+0000 7fc19d3fe640 1 -- 172.21.15.1:0/1400088415 >> v1:172.21.15.196:6789/0 conn(0x7fc19806c6f0 legacy=0x7fc1980c66a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.613+0000 7fc19d3fe640 1 -- 172.21.15.1:0/1400088415 >> v1:172.21.15.1:6789/0 conn(0x7fc1980cc260 legacy=0x7fc1980c5f90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.613+0000 7fc19d3fe640 1 -- 172.21.15.1:0/1400088415 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1980ccc60 con 0x7fc1980c5670 2024-05-09T05:07:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.613+0000 7fc19d3fe640 1 --2- 172.21.15.1:0/1400088415 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1980c5670 0x7fc198153830 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc190009e50 tx=0x7fc19002a7a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4f968fcf0d59d8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.613+0000 7fc1977fe640 1 -- 172.21.15.1:0/1400088415 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc190036070 con 0x7fc1980c5670 2024-05-09T05:07:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.614+0000 7fc1977fe640 1 -- 172.21.15.1:0/1400088415 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc190031600 con 0x7fc1980c5670 2024-05-09T05:07:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.614+0000 7fc1977fe640 1 -- 172.21.15.1:0/1400088415 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1900319d0 con 0x7fc1980c5670 2024-05-09T05:07:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.614+0000 7fc19ee61640 1 -- 172.21.15.1:0/1400088415 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1980c5670 msgr2=0x7fc198153830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.614+0000 7fc19ee61640 1 --2- 172.21.15.1:0/1400088415 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1980c5670 0x7fc198153830 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fc190009e50 tx=0x7fc19002a7a0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.614+0000 7fc19ee61640 1 -- 172.21.15.1:0/1400088415 shutdown_connections 2024-05-09T05:07:53.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.614+0000 7fc19ee61640 1 --2- 172.21.15.1:0/1400088415 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1980c5670 0x7fc198153830 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.614+0000 7fc19ee61640 1 -- 172.21.15.1:0/1400088415 >> 172.21.15.1:0/1400088415 conn(0x7fc1980cac70 msgr2=0x7fc1980cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:53.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.615+0000 7fc19ee61640 1 -- 172.21.15.1:0/1400088415 shutdown_connections 2024-05-09T05:07:53.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.615+0000 7fc19ee61640 1 -- 172.21.15.1:0/1400088415 wait complete. 2024-05-09T05:07:53.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.616+0000 7fc19ee61640 1 Processor -- start 2024-05-09T05:07:53.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.616+0000 7fc19ee61640 1 -- start start 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19ee61640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19806c6f0 0x7fc1981f5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19ee61640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1980c5670 0x7fc1981e6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19cbfd640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19806c6f0 msgr2=0x7fc1981f5f60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19ee61640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1980cc260 0x7fc19814a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19cbfd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19806c6f0 0x7fc1981f5f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19ee61640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc1980c8240 con 0x7fc19806c6f0 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19ee61640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc198159a70 con 0x7fc1980cc260 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19ee61640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc198159f90 con 0x7fc1980c5670 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc197fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1980c5670 0x7fc1981e6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc197fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1980c5670 0x7fc1981e6ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:35850/0 (socket says 172.21.15.1:35850) 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc197fff640 1 -- 172.21.15.1:0/2972066819 learned_addr learned my addr 172.21.15.1:0/2972066819 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.618+0000 7fc19d3fe640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1980cc260 0x7fc19814a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc197fff640 1 -- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1980cc260 msgr2=0x7fc19814a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc197fff640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc1980cc260 0x7fc19814a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc197fff640 1 -- 172.21.15.1:0/2972066819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19806c6f0 msgr2=0x7fc1981f5f60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc197fff640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc19806c6f0 0x7fc1981f5f60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc197fff640 1 -- 172.21.15.1:0/2972066819 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1981f64a0 con 0x7fc1980c5670 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc197fff640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1980c5670 0x7fc1981e6ef0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc184001eb0 tx=0x7fc18400c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1840074e0 con 0x7fc1980c5670 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc184007680 con 0x7fc1980c5670 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc1840049e0 con 0x7fc1980c5670 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc19ee61640 1 -- 172.21.15.1:0/2972066819 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1981f27f0 con 0x7fc1980c5670 2024-05-09T05:07:53.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.619+0000 7fc19ee61640 1 -- 172.21.15.1:0/2972066819 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1981f2d90 con 0x7fc1980c5670 2024-05-09T05:07:53.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.620+0000 7fc19ee61640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815a690 0x7fc1981f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.620+0000 7fc19ee61640 1 -- 172.21.15.1:0/2972066819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc1981f6970 con 0x7fc19815a690 2024-05-09T05:07:53.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.620+0000 7fc19cbfd640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815a690 0x7fc1981f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.621+0000 7fc19cbfd640 1 -- 172.21.15.1:0/2972066819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc1981f3730 con 0x7fc19815a690 2024-05-09T05:07:53.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.621+0000 7fc19cbfd640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815a690 0x7fc1981f2fd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc18c003980 tx=0x7fc18c00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.621+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc18401a020 con 0x7fc1980c5670 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc1957fa640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc17c076280 0x7fc17c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc184096de0 con 0x7fc1980c5670 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc19d3fe640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc17c076280 0x7fc17c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c003e10 con 0x7fc19815a690 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc18c015070 con 0x7fc19815a690 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815a690 msgr2=0x7fc1981f2fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc1957fa640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815a690 0x7fc1981f2fd0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fc18c003980 tx=0x7fc18c00b8c0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815a690 msgr2=0x7fc1981f2fd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc1957fa640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815a690 0x7fc1981f2fd0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.622+0000 7fc19d3fe640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc17c076280 0x7fc17c078790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc190033000 tx=0x7fc190009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.628+0000 7fc19ee61640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815cff0 0x7fc19815d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.628+0000 7fc19ee61640 1 -- 172.21.15.1:0/2972066819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc19815d940 con 0x7fc19815cff0 2024-05-09T05:07:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.629+0000 7fc19cbfd640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815cff0 0x7fc19815d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.629+0000 7fc19cbfd640 1 -- 172.21.15.1:0/2972066819 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc1981f5d60 con 0x7fc19815cff0 2024-05-09T05:07:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.629+0000 7fc19cbfd640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815cff0 0x7fc19815d400 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc18c003980 tx=0x7fc18c005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc18c005800 con 0x7fc19815cff0 2024-05-09T05:07:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5052 (secure 0 0 0) 0x7fc18c005800 con 0x7fc19815cff0 2024-05-09T05:07:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815cff0 msgr2=0x7fc19815d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.632 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:07:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 16, 2024-05-09T05:07:53.632 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 137722, 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:53.633 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:53.634 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:53.635 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:53.636 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:53.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:53.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:53.640 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:53.641 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.643 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.644 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.645 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.646 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:53.646 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc1957fa640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815cff0 0x7fc19815d400 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fc18c003980 tx=0x7fc18c005a20 comp rx=0 tx=0).stop 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc1957fa640 1 -- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815cff0 msgr2=0x7fc19815d400 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc1957fa640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc19815cff0 0x7fc19815d400 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc182ffd640 1 -- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc17c076280 msgr2=0x7fc17c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc182ffd640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc17c076280 0x7fc17c078790 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fc190033000 tx=0x7fc190009000 comp rx=0 tx=0).stop 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc182ffd640 1 -- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1980c5670 msgr2=0x7fc1981e6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc182ffd640 1 --2- 172.21.15.1:0/2972066819 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc1980c5670 0x7fc1981e6ef0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc184001eb0 tx=0x7fc18400c890 comp rx=0 tx=0).stop 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc19d3fe640 1 -- 172.21.15.1:0/2972066819 reap_dead start 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.630+0000 7fc19d3fe640 1 -- 172.21.15.1:0/2972066819 reap_dead start 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.631+0000 7fc182ffd640 1 -- 172.21.15.1:0/2972066819 shutdown_connections 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.631+0000 7fc182ffd640 1 -- 172.21.15.1:0/2972066819 >> 172.21.15.1:0/2972066819 conn(0x7fc1980cac70 msgr2=0x7fc19806d9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.631+0000 7fc182ffd640 1 -- 172.21.15.1:0/2972066819 shutdown_connections 2024-05-09T05:07:53.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.631+0000 7fc182ffd640 1 -- 172.21.15.1:0/2972066819 wait complete. 2024-05-09T05:07:53.649 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:07:53.649 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:07:53.649 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:07:53.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.764+0000 7f85522cd640 1 Processor -- start 2024-05-09T05:07:53.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.764+0000 7f85522cd640 1 -- start start 2024-05-09T05:07:53.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f85522cd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f854c153730 0x7f854c164d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f854bfff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f854c0ca970 legacy=0x7f854c15c970 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:07:53.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f85522cd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c0c8a50 con 0x7f854c0ca970 2024-05-09T05:07:53.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f85522cd640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f854c0c8d10 con 0x7f854c0cae10 2024-05-09T05:07:53.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f85522cd640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f854c0c9d00 con 0x7f854c153730 2024-05-09T05:07:53.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f855086a640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f854c153730 0x7f854c164d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f854b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f854c0cae10 0x7f854c0c67b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:44328/0 (socket says 172.21.15.1:44328) 2024-05-09T05:07:53.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f854b7fe640 1 -- 172.21.15.1:0/3249582026 learned_addr learned my addr 172.21.15.1:0/3249582026 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:07:53.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f855086a640 1 -- 172.21.15.1:0/3249582026 >> v1:172.21.15.196:6789/0 conn(0x7f854c0cae10 legacy=0x7f854c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f855086a640 1 -- 172.21.15.1:0/3249582026 >> v1:172.21.15.1:6789/0 conn(0x7f854c0ca970 legacy=0x7f854c15c970 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:07:53.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.766+0000 7f855086a640 1 -- 172.21.15.1:0/3249582026 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c0c5e10 con 0x7f854c153730 2024-05-09T05:07:53.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.767+0000 7f855086a640 1 --2- 172.21.15.1:0/3249582026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f854c153730 0x7f854c164d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f854400bfa0 tx=0x7f854402c860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e67afee59025478 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.768+0000 7f854affd640 1 -- 172.21.15.1:0/3249582026 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544039070 con 0x7f854c153730 2024-05-09T05:07:53.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.768+0000 7f854affd640 1 -- 172.21.15.1:0/3249582026 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8544009540 con 0x7f854c153730 2024-05-09T05:07:53.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.768+0000 7f854affd640 1 -- 172.21.15.1:0/3249582026 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8544009910 con 0x7f854c153730 2024-05-09T05:07:53.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.768+0000 7f85522cd640 1 -- 172.21.15.1:0/3249582026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f854c153730 msgr2=0x7f854c164d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.768+0000 7f85522cd640 1 --2- 172.21.15.1:0/3249582026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f854c153730 0x7f854c164d10 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f854400bfa0 tx=0x7f854402c860 comp rx=0 tx=0).stop 2024-05-09T05:07:53.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.768+0000 7f85522cd640 1 -- 172.21.15.1:0/3249582026 shutdown_connections 2024-05-09T05:07:53.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.768+0000 7f85522cd640 1 --2- 172.21.15.1:0/3249582026 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f854c153730 0x7f854c164d10 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.769+0000 7f85522cd640 1 -- 172.21.15.1:0/3249582026 >> 172.21.15.1:0/3249582026 conn(0x7f854c0ce910 msgr2=0x7f854c0ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:53.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.769+0000 7f85522cd640 1 -- 172.21.15.1:0/3249582026 shutdown_connections 2024-05-09T05:07:53.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.769+0000 7f85522cd640 1 -- 172.21.15.1:0/3249582026 wait complete. 2024-05-09T05:07:53.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.770+0000 7f85522cd640 1 Processor -- start 2024-05-09T05:07:53.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.771+0000 7f85522cd640 1 -- start start 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f85522cd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c0ca970 0x7f854c146cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f85522cd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f854c0cae10 0x7f854c141530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f85522cd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c153730 0x7f854c1e29e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f85522cd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f854c0c63d0 con 0x7f854c153730 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f85522cd640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f854c0c9870 con 0x7f854c0cae10 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f85522cd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f854c0c9500 con 0x7f854c0ca970 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f855086a640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c153730 msgr2=0x7f854c1e29e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f855086a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c153730 0x7f854c1e29e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c0ca970 0x7f854c146cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c0ca970 0x7f854c146cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:35858/0 (socket says 172.21.15.1:35858) 2024-05-09T05:07:53.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854bfff640 1 -- 172.21.15.1:0/2574544317 learned_addr learned my addr 172.21.15.1:0/2574544317 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854b7fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f854c0cae10 0x7f854c141530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854bfff640 1 -- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f854c0cae10 msgr2=0x7f854c141530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854bfff640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f854c0cae10 0x7f854c141530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854bfff640 1 -- 172.21.15.1:0/2574544317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c153730 msgr2=0x7f854c1e29e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854bfff640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f854c153730 0x7f854c1e29e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.773+0000 7f854bfff640 1 -- 172.21.15.1:0/2574544317 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f854c1ef6e0 con 0x7f854c0ca970 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.774+0000 7f854bfff640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c0ca970 0x7f854c146cc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f853c00b550 tx=0x7f853c00ba10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.774+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c013020 con 0x7f854c0ca970 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.774+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f853c003de0 con 0x7f854c0ca970 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.774+0000 7f85522cd640 1 -- 172.21.15.1:0/2574544317 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f854c1458d0 con 0x7f854c0ca970 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.774+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f853c0052b0 con 0x7f854c0ca970 2024-05-09T05:07:53.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.774+0000 7f85522cd640 1 -- 172.21.15.1:0/2574544317 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f854c145e90 con 0x7f854c0ca970 2024-05-09T05:07:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.775+0000 7f85327fc640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c146090 0x7f854c1f1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.775+0000 7f85327fc640 1 -- 172.21.15.1:0/2574544317 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f854c15d930 con 0x7f854c146090 2024-05-09T05:07:53.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.775+0000 7f854b7fe640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c146090 0x7f854c1f1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.776+0000 7f854b7fe640 1 -- 172.21.15.1:0/2574544317 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f854c151910 con 0x7f854c146090 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.776+0000 7f854b7fe640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c146090 0x7f854c1f1cb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f854c0ccfe0 tx=0x7f854000fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.776+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85400058b0 con 0x7f854c146090 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.776+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f853c016020 con 0x7f854c0ca970 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.777+0000 7f8548ff9640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85380761d0 0x7f85380786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.777+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8540018070 con 0x7f854c146090 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.777+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c146090 msgr2=0x7f854c1f1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.777+0000 7f8548ff9640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c146090 0x7f854c1f1cb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f854c0ccfe0 tx=0x7f854000fc00 comp rx=0 tx=0).stop 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.777+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c146090 msgr2=0x7f854c1f1cb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.777+0000 7f8548ff9640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c146090 0x7f854c1f1cb0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f854c0ccfe0 tx=0x7f854000fc00 comp rx=0 tx=0).stop 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.777+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f853c097400 con 0x7f854c0ca970 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.777+0000 7f855086a640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85380761d0 0x7f85380786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.778+0000 7f855086a640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85380761d0 0x7f85380786e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8544036000 tx=0x7f854400b000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.790+0000 7f85327fc640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c1ef1a0 0x7f854c0cd010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:07:53.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.790+0000 7f85327fc640 1 -- 172.21.15.1:0/2574544317 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f854c156da0 con 0x7f854c1ef1a0 2024-05-09T05:07:53.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.790+0000 7f854b7fe640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c1ef1a0 0x7f854c0cd010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:07:53.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.791+0000 7f854b7fe640 1 -- 172.21.15.1:0/2574544317 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f854c0c9eb0 con 0x7f854c1ef1a0 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.791+0000 7f854b7fe640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c1ef1a0 0x7f854c0cd010 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8540017000 tx=0x7f85400057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.791+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8540007720 con 0x7f854c1ef1a0 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.791+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4879 (secure 0 0 0) 0x7f8540007720 con 0x7f854c1ef1a0 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.791+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c1ef1a0 msgr2=0x7f854c0cd010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.791+0000 7f8548ff9640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c1ef1a0 0x7f854c0cd010 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8540017000 tx=0x7f85400057f0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.791+0000 7f8548ff9640 1 -- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c1ef1a0 msgr2=0x7f854c0cd010 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.791+0000 7f8548ff9640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c1ef1a0 0x7f854c0cd010 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f8540017000 tx=0x7f85400057f0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:53.793 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 16, 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 31790, 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:53.794 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:53.795 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:07:53.796 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:07:53.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:07:53.798 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.799 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:53.800 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:53.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:07:53.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:07:53.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.804 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:07:53.805 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:07:53.806 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:07:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 -- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85380761d0 msgr2=0x7f85380786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85380761d0 0x7f85380786e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f8544036000 tx=0x7f854400b000 comp rx=0 tx=0).stop 2024-05-09T05:07:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 -- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c0ca970 msgr2=0x7f854c146cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:07:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c0ca970 0x7f854c146cc0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f853c00b550 tx=0x7f853c00ba10 comp rx=0 tx=0).stop 2024-05-09T05:07:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f855086a640 1 -- 172.21.15.1:0/2574544317 reap_dead start 2024-05-09T05:07:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 -- 172.21.15.1:0/2574544317 shutdown_connections 2024-05-09T05:07:53.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 --2- 172.21.15.1:0/2574544317 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f854c0ca970 0x7f854c146cc0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:07:53.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 -- 172.21.15.1:0/2574544317 >> 172.21.15.1:0/2574544317 conn(0x7f854c0ce910 msgr2=0x7f854c14e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:07:53.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 -- 172.21.15.1:0/2574544317 shutdown_connections 2024-05-09T05:07:53.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:07:53.792+0000 7f85327fc640 1 -- 172.21.15.1:0/2574544317 wait complete. 2024-05-09T05:07:53.808 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:07:53.808 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:08:01.857 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:08:01.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:08:10.152 INFO:tasks.workunit.client.0.smithi001.stderr:Updating files: 94% (11553/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-05-09T05:08:10.571 INFO:tasks.workunit.client.0.smithi001.stderr:Note: switching to 'c9da4049f638c0f84bdd9b4e33c42e86046ac554'. 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr:state without impacting any branches by switching back to a branch. 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr: git switch -c 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr:Or undo this operation with: 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr: git switch - 2024-05-09T05:08:10.572 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-05-09T05:08:10.573 INFO:tasks.workunit.client.0.smithi001.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-09T05:08:10.573 INFO:tasks.workunit.client.0.smithi001.stderr: 2024-05-09T05:08:10.573 INFO:tasks.workunit.client.0.smithi001.stderr:HEAD is now at c9da4049f63 Merge PR #57081 into wip-pdonnell-testing-20240508.183908-debug 2024-05-09T05:08:10.578 DEBUG:teuthology.orchestra.run.smithi001:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-09T05:08:10.618 INFO:tasks.workunit.client.0.smithi001.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-09T05:08:10.620 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T05:08:10.620 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-09T05:08:11.215 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-09T05:08:11.271 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-09T05:08:11.312 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T05:08:11.314 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T05:08:11.314 INFO:tasks.workunit.client.0.smithi001.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-09T05:08:11.354 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T05:08:11.358 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:08:11.358 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-09T05:08:11.415 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-05-09T05:08:11.416 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-05-09T05:08:11.416 DEBUG:teuthology.orchestra.run.smithi001:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=c9da4049f638c0f84bdd9b4e33c42e86046ac554 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-05-09T05:08:11.484 INFO:tasks.workunit.client.0.smithi001.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-05-09T05:08:11.485 INFO:tasks.workunit.client.0.smithi001.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-05-09T05:08:11.485 INFO:tasks.workunit.client.0.smithi001.stderr:+ trap -- retry EXIT 2024-05-09T05:08:11.485 INFO:tasks.workunit.client.0.smithi001.stderr:+ rm -rf ffsb 2024-05-09T05:08:11.486 INFO:tasks.workunit.client.0.smithi001.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-05-09T05:08:11.491 INFO:tasks.workunit.client.0.smithi001.stderr:Cloning into 'ffsb'... 2024-05-09T05:08:13.165 INFO:tasks.workunit.client.0.smithi001.stderr:+ trap - EXIT 2024-05-09T05:08:13.166 INFO:tasks.workunit.client.0.smithi001.stderr:+ cd ffsb 2024-05-09T05:08:13.166 INFO:tasks.workunit.client.0.smithi001.stderr:+ ./configure 2024-05-09T05:08:13.302 INFO:tasks.workunit.client.0.smithi001.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-05-09T05:08:13.809 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-05-09T05:08:13.809 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-05-09T05:08:13.809 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T05:08:13.809 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T05:08:13.813 INFO:tasks.ceph.mon.a:Started 2024-05-09T05:08:13.813 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:08:13.813 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:13.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.934+0000 7f2668e3b640 1 Processor -- start 2024-05-09T05:08:13.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.934+0000 7f2668e3b640 1 -- start start 2024-05-09T05:08:13.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f2668e3b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f266406f1b0 0x7f26640ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:13.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f2668e3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f266415ca90 0x7f266415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:13.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f2668e3b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f266406f6d0 con 0x7f26640d16b0 2024-05-09T05:08:13.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f2668e3b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f26640cd0e0 con 0x7f266406f1b0 2024-05-09T05:08:13.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f2668e3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f26640cc4d0 con 0x7f266415ca90 2024-05-09T05:08:13.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f266259c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f26640d16b0 0x7f26640d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45820/0 (socket says 172.21.15.1:45820) 2024-05-09T05:08:13.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f266259c640 1 -- 172.21.15.1:0/2534066440 learned_addr learned my addr 172.21.15.1:0/2534066440 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:13.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f2661d9b640 1 --2- 172.21.15.1:0/2534066440 >> v2:172.21.15.196:3300/0 conn(0x7f266406f1b0 0x7f26640ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:13.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.936+0000 7f2662d9d640 1 --2- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f266415ca90 0x7f266415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:13.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.937+0000 7f2661d9b640 1 -- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f266415ca90 msgr2=0x7f266415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:13.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.937+0000 7f2661d9b640 1 --2- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f266415ca90 0x7f266415cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:13.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.937+0000 7f2661d9b640 1 -- 172.21.15.1:0/2534066440 >> v1:172.21.15.1:6789/0 conn(0x7f26640d16b0 legacy=0x7f26640d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:13.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.937+0000 7f2661d9b640 1 -- 172.21.15.1:0/2534066440 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26640cc7b0 con 0x7f266406f1b0 2024-05-09T05:08:13.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.937+0000 7f2662d9d640 1 --2- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f266415ca90 0x7f266415cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.937+0000 7f2661d9b640 1 --2- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 0x7f26640ce740 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f26580097a0 tx=0x7f265802a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=172539483d1301da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.937+0000 7f266159a640 1 -- 172.21.15.1:0/2534066440 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658037070 con 0x7f266406f1b0 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f266159a640 1 -- 172.21.15.1:0/2534066440 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2658032440 con 0x7f266406f1b0 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f266159a640 1 -- 172.21.15.1:0/2534066440 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2658032810 con 0x7f266406f1b0 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f2668e3b640 1 -- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 msgr2=0x7f26640ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f2668e3b640 1 --2- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 0x7f26640ce740 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f26580097a0 tx=0x7f265802a210 comp rx=0 tx=0).stop 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f2668e3b640 1 -- 172.21.15.1:0/2534066440 shutdown_connections 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f2668e3b640 1 --2- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f266415ca90 0x7f266415cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:13.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f2668e3b640 1 --2- 172.21.15.1:0/2534066440 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 0x7f26640ce740 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:13.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f2668e3b640 1 -- 172.21.15.1:0/2534066440 >> 172.21.15.1:0/2534066440 conn(0x7f26640b3660 msgr2=0x7f26640b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:13.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f2668e3b640 1 -- 172.21.15.1:0/2534066440 shutdown_connections 2024-05-09T05:08:13.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.938+0000 7f2668e3b640 1 -- 172.21.15.1:0/2534066440 wait complete. 2024-05-09T05:08:13.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.940+0000 7f2668e3b640 1 Processor -- start 2024-05-09T05:08:13.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.940+0000 7f2668e3b640 1 -- start start 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.941+0000 7f2668e3b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 0x7f26641f2380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.941+0000 7f2668e3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26640d16b0 0x7f26640cb7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.941+0000 7f2668e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 0x7f26640cbd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.941+0000 7f2668e3b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f266415d3e0 con 0x7f266415ca90 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.941+0000 7f2668e3b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26640ccd70 con 0x7f266406f1b0 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.941+0000 7f2668e3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f26640cec80 con 0x7f26640d16b0 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2662d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 0x7f26640cbd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2662d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 0x7f26640cbd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36580/0 (socket says 172.21.15.1:36580) 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2662d9d640 1 -- 172.21.15.1:0/127536395 learned_addr learned my addr 172.21.15.1:0/127536395 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:13.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f266259c640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 0x7f26641f2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2661d9b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26640d16b0 0x7f26640cb7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2662d9d640 1 -- 172.21.15.1:0/127536395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 msgr2=0x7f26640cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2662d9d640 1 -- 172.21.15.1:0/127536395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 msgr2=0x7f26640cbd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2662d9d640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 0x7f26640cbd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2662d9d640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 0x7f26640cbd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f266259c640 1 -- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26640d16b0 msgr2=0x7f26640cb7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f266259c640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26640d16b0 0x7f26640cb7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f266259c640 1 -- 172.21.15.1:0/127536395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 msgr2=0x7f26640cbd20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f266259c640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 0x7f26640cbd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f266259c640 1 -- 172.21.15.1:0/127536395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26641f58f0 con 0x7f266406f1b0 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2661d9b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26640d16b0 0x7f26640cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f266259c640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 0x7f26641f2380 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f265400c310 tx=0x7f265400c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2652ffd640 1 -- 172.21.15.1:0/127536395 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654017070 con 0x7f266406f1b0 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2652ffd640 1 -- 172.21.15.1:0/127536395 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2654005ce0 con 0x7f266406f1b0 2024-05-09T05:08:13.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26641f5bd0 con 0x7f266406f1b0 2024-05-09T05:08:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2652ffd640 1 -- 172.21.15.1:0/127536395 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2654007310 con 0x7f266406f1b0 2024-05-09T05:08:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.942+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26641f6190 con 0x7f266406f1b0 2024-05-09T05:08:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.943+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2664050870 con 0x7f266406f1b0 2024-05-09T05:08:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.944+0000 7f2652ffd640 1 -- 172.21.15.1:0/127536395 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f265401a070 con 0x7f266406f1b0 2024-05-09T05:08:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.945+0000 7f2652ffd640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26440762d0 0x7f26440787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:13.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.945+0000 7f2652ffd640 1 -- 172.21.15.1:0/127536395 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2654096cf0 con 0x7f266406f1b0 2024-05-09T05:08:13.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.945+0000 7f2661d9b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26440762d0 0x7f26440787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:13.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.945+0000 7f2661d9b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26440762d0 0x7f26440787e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2658034000 tx=0x7f265802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:13.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:13.949+0000 7f2652ffd640 1 -- 172.21.15.1:0/127536395 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f265403a080 con 0x7f266406f1b0 2024-05-09T05:08:14.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:14.216+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f266415d040 con 0x7f266406f1b0 2024-05-09T05:08:14.311 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether build environment is sane... yes 2024-05-09T05:08:14.325 INFO:tasks.workunit.client.0.smithi001.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-05-09T05:08:14.326 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gawk... gawk 2024-05-09T05:08:14.349 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether make sets $(MAKE)... yes 2024-05-09T05:08:14.392 INFO:tasks.workunit.client.0.smithi001.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-05-09T05:08:14.393 INFO:tasks.workunit.client.0.smithi001.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-05-09T05:08:14.411 INFO:tasks.workunit.client.0.smithi001.stdout:checking for style of include used by make... GNU 2024-05-09T05:08:14.416 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gcc... gcc 2024-05-09T05:08:14.503 INFO:tasks.workunit.client.0.smithi001.stdout:checking for C compiler default output file name... a.out 2024-05-09T05:08:14.505 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether the C compiler works... yes 2024-05-09T05:08:14.509 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether we are cross compiling... no 2024-05-09T05:08:14.564 INFO:tasks.workunit.client.0.smithi001.stdout:checking for suffix of executables... 2024-05-09T05:08:14.609 INFO:tasks.workunit.client.0.smithi001.stdout:checking for suffix of object files... o 2024-05-09T05:08:14.652 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether we are using the GNU C compiler... yes 2024-05-09T05:08:14.696 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether gcc accepts -g... yes 2024-05-09T05:08:14.766 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gcc option to accept ISO C89... none needed 2024-05-09T05:08:19.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.139+0000 7f2652ffd640 1 -- 172.21.15.1:0/127536395 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f265405fed0 con 0x7f266406f1b0 2024-05-09T05:08:19.140 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26440762d0 msgr2=0x7f26440787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26440762d0 0x7f26440787e0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2658034000 tx=0x7f265802e000 comp rx=0 tx=0).stop 2024-05-09T05:08:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 msgr2=0x7f26641f2380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 0x7f26641f2380 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f265400c310 tx=0x7f265400c7d0 comp rx=0 tx=0).stop 2024-05-09T05:08:19.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 shutdown_connections 2024-05-09T05:08:19.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26440762d0 0x7f26440787e0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:19.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f266415ca90 0x7f26640cbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:19.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26640d16b0 0x7f26640cb7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:19.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 --2- 172.21.15.1:0/127536395 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f266406f1b0 0x7f26641f2380 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:19.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 >> 172.21.15.1:0/127536395 conn(0x7f26640b3660 msgr2=0x7f2664154ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:19.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.142+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 shutdown_connections 2024-05-09T05:08:19.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:19.143+0000 7f2668e3b640 1 -- 172.21.15.1:0/127536395 wait complete. 2024-05-09T05:08:22.160 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:22.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.288+0000 7f1b54738640 1 Processor -- start 2024-05-09T05:08:22.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.288+0000 7f1b54738640 1 -- start start 2024-05-09T05:08:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b54738640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b4c0ca340 0x7f1b4c0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b54738640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1b4c072120 0x7f1b4c1547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:22.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b54738640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b4c154cf0 con 0x7f1b4c0c62e0 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b54738640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f1b4c0c53f0 con 0x7f1b4c0ca340 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b54738640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f1b4c0c55b0 con 0x7f1b4c072120 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b524d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b4c0ca340 0x7f1b4c0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b524d4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b4c0ca340 0x7f1b4c0c4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49238/0 (socket says 172.21.15.1:49238) 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b524d4640 1 -- 172.21.15.1:0/2905037306 learned_addr learned my addr 172.21.15.1:0/2905037306 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.289+0000 7f1b51cd3640 1 --2- 172.21.15.1:0/2905037306 >> v2:172.21.15.196:3300/0 conn(0x7f1b4c072120 0x7f1b4c1547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.290+0000 7f1b524d4640 1 -- 172.21.15.1:0/2905037306 >> v2:172.21.15.196:3300/0 conn(0x7f1b4c072120 msgr2=0x7f1b4c1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.290+0000 7f1b524d4640 1 --2- 172.21.15.1:0/2905037306 >> v2:172.21.15.196:3300/0 conn(0x7f1b4c072120 0x7f1b4c1547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.290+0000 7f1b524d4640 1 -- 172.21.15.1:0/2905037306 >> v1:172.21.15.196:6789/0 conn(0x7f1b4c0c62e0 legacy=0x7f1b4c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:22.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.290+0000 7f1b524d4640 1 -- 172.21.15.1:0/2905037306 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4c15b110 con 0x7f1b4c0ca340 2024-05-09T05:08:22.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.291+0000 7f1b524d4640 1 --2- 172.21.15.1:0/2905037306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c0ca340 0x7f1b4c0c4eb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0b37a0 tx=0x7f1b3c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec5493d5981cc0c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:22.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.291+0000 7f1b514d2640 1 -- 172.21.15.1:0/2905037306 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c037070 con 0x7f1b4c0ca340 2024-05-09T05:08:22.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.291+0000 7f1b514d2640 1 -- 172.21.15.1:0/2905037306 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3c032440 con 0x7f1b4c0ca340 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.291+0000 7f1b514d2640 1 -- 172.21.15.1:0/2905037306 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c032810 con 0x7f1b4c0ca340 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.291+0000 7f1b54738640 1 -- 172.21.15.1:0/2905037306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c0ca340 msgr2=0x7f1b4c0c4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.291+0000 7f1b54738640 1 --2- 172.21.15.1:0/2905037306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c0ca340 0x7f1b4c0c4eb0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c0b37a0 tx=0x7f1b3c02a1e0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.292+0000 7f1b54738640 1 -- 172.21.15.1:0/2905037306 shutdown_connections 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.292+0000 7f1b54738640 1 --2- 172.21.15.1:0/2905037306 >> v2:172.21.15.196:3300/0 conn(0x7f1b4c072120 0x7f1b4c1547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.292+0000 7f1b54738640 1 --2- 172.21.15.1:0/2905037306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c0ca340 0x7f1b4c0c4eb0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.292+0000 7f1b54738640 1 -- 172.21.15.1:0/2905037306 >> 172.21.15.1:0/2905037306 conn(0x7f1b4c0731d0 msgr2=0x7f1b4c0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.292+0000 7f1b54738640 1 -- 172.21.15.1:0/2905037306 shutdown_connections 2024-05-09T05:08:22.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.292+0000 7f1b54738640 1 -- 172.21.15.1:0/2905037306 wait complete. 2024-05-09T05:08:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.293+0000 7f1b54738640 1 Processor -- start 2024-05-09T05:08:22.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.293+0000 7f1b54738640 1 -- start start 2024-05-09T05:08:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b54738640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c072120 0x7f1b4c158a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:22.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b54738640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b4c0c62e0 0x7f1b4c153480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b54738640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b4c0ca340 0x7f1b4c14d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b54738640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c0c5ae0 con 0x7f1b4c072120 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b54738640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1b4c0c5800 con 0x7f1b4c0c62e0 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b54738640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1b4c0c5dc0 con 0x7f1b4c0ca340 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c072120 0x7f1b4c158a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c072120 0x7f1b4c158a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49240/0 (socket says 172.21.15.1:49240) 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 -- 172.21.15.1:0/2217637896 learned_addr learned my addr 172.21.15.1:0/2217637896 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b51cd3640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b4c0c62e0 0x7f1b4c153480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b52cd5640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b4c0ca340 0x7f1b4c14d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 -- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b4c0ca340 msgr2=0x7f1b4c14d360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b4c0ca340 0x7f1b4c14d360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 -- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b4c0c62e0 msgr2=0x7f1b4c153480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b4c0c62e0 0x7f1b4c153480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 -- 172.21.15.1:0/2217637896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b4c158f60 con 0x7f1b4c072120 2024-05-09T05:08:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b52cd5640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b4c0ca340 0x7f1b4c14d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:08:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.295+0000 7f1b524d4640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c072120 0x7f1b4c158a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c02a220 tx=0x7f1b3c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.296+0000 7f1b4affd640 1 -- 172.21.15.1:0/2217637896 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c037040 con 0x7f1b4c072120 2024-05-09T05:08:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.296+0000 7f1b4affd640 1 -- 172.21.15.1:0/2217637896 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b3c042070 con 0x7f1b4c072120 2024-05-09T05:08:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.296+0000 7f1b4affd640 1 -- 172.21.15.1:0/2217637896 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b3c030880 con 0x7f1b4c072120 2024-05-09T05:08:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.296+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b4c1f0980 con 0x7f1b4c072120 2024-05-09T05:08:22.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.296+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b4c1f0f40 con 0x7f1b4c072120 2024-05-09T05:08:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.296+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b4c050870 con 0x7f1b4c072120 2024-05-09T05:08:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.297+0000 7f1b4affd640 1 -- 172.21.15.1:0/2217637896 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1b3c030a20 con 0x7f1b4c072120 2024-05-09T05:08:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.298+0000 7f1b4affd640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b38076300 0x7f1b38078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.298+0000 7f1b4affd640 1 -- 172.21.15.1:0/2217637896 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1b3c0b7f30 con 0x7f1b4c072120 2024-05-09T05:08:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.298+0000 7f1b51cd3640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b38076300 0x7f1b38078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:22.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.299+0000 7f1b51cd3640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b38076300 0x7f1b38078810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c151070 tx=0x7f1b40008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:22.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.303+0000 7f1b4affd640 1 -- 172.21.15.1:0/2217637896 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1b3c081810 con 0x7f1b4c072120 2024-05-09T05:08:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.570+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1b4c0cba10 con 0x7f1b4c072120 2024-05-09T05:08:22.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.571+0000 7f1b4affd640 1 -- 172.21.15.1:0/2217637896 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f1b3c081110 con 0x7f1b4c072120 2024-05-09T05:08:22.571 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b38076300 msgr2=0x7f1b38078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b38076300 0x7f1b38078810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c151070 tx=0x7f1b40008040 comp rx=0 tx=0).stop 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c072120 msgr2=0x7f1b4c158a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c072120 0x7f1b4c158a20 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1b3c02a220 tx=0x7f1b3c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 shutdown_connections 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b38076300 0x7f1b38078810 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b4c0c62e0 0x7f1b4c153480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b4c0ca340 0x7f1b4c14d360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 --2- 172.21.15.1:0/2217637896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b4c072120 0x7f1b4c158a20 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 >> 172.21.15.1:0/2217637896 conn(0x7f1b4c0731d0 msgr2=0x7f1b4c0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 shutdown_connections 2024-05-09T05:08:22.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:22.573+0000 7f1b54738640 1 -- 172.21.15.1:0/2217637896 wait complete. 2024-05-09T05:08:25.585 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:25.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.715+0000 7fe883896640 1 Processor -- start 2024-05-09T05:08:25.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.716+0000 7fe883896640 1 -- start start 2024-05-09T05:08:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe883896640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe87c0cc4c0 0x7fe87c0c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe883896640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 0x7fe87c1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe883896640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe87c15d3f0 con 0x7fe87c0c6a80 2024-05-09T05:08:25.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe883896640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe87c0c82d0 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe883896640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe87c0c8ca0 con 0x7fe87c0c7500 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe880e31640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 0x7fe87c1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe881632640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe87c0cc4c0 0x7fe87c0c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe881632640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe87c0cc4c0 0x7fe87c0c8760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:55334/0 (socket says 172.21.15.1:55334) 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe881632640 1 -- 172.21.15.1:0/4024815133 learned_addr learned my addr 172.21.15.1:0/4024815133 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe880e31640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 0x7fe87c1569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:51084/0 (socket says 172.21.15.1:51084) 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe880e31640 1 -- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 msgr2=0x7fe87c1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe880e31640 1 -- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 msgr2=0x7fe87c1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe880e31640 1 --2- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 0x7fe87c1569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.717+0000 7fe880e31640 1 --2- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 0x7fe87c1569c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.718+0000 7fe881632640 1 -- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 msgr2=0x7fe87c1569c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.718+0000 7fe881632640 1 --2- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 0x7fe87c1569c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.718+0000 7fe881632640 1 -- 172.21.15.1:0/4024815133 >> v1:172.21.15.196:6789/0 conn(0x7fe87c0c6a80 legacy=0x7fe87c0ce550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:25.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.718+0000 7fe881632640 1 -- 172.21.15.1:0/4024815133 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c0c7c80 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.718+0000 7fe881632640 1 --2- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c0c8760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe874009870 tx=0x7fe87402a300 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99f7113a6f3e53e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe87bfff640 1 -- 172.21.15.1:0/4024815133 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874037070 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe87bfff640 1 -- 172.21.15.1:0/4024815133 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe87402ae50 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe87bfff640 1 -- 172.21.15.1:0/4024815133 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe874032860 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe883896640 1 -- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0cc4c0 msgr2=0x7fe87c0c8760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe883896640 1 --2- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c0c8760 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fe874009870 tx=0x7fe87402a300 comp rx=0 tx=0).stop 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe883896640 1 -- 172.21.15.1:0/4024815133 shutdown_connections 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe883896640 1 --2- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c7500 0x7fe87c1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe883896640 1 --2- 172.21.15.1:0/4024815133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c0c8760 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.719+0000 7fe883896640 1 -- 172.21.15.1:0/4024815133 >> 172.21.15.1:0/4024815133 conn(0x7fe87c068650 msgr2=0x7fe87c068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.720+0000 7fe883896640 1 -- 172.21.15.1:0/4024815133 shutdown_connections 2024-05-09T05:08:25.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.720+0000 7fe883896640 1 -- 172.21.15.1:0/4024815133 wait complete. 2024-05-09T05:08:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.721+0000 7fe883896640 1 Processor -- start 2024-05-09T05:08:25.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.721+0000 7fe883896640 1 -- start start 2024-05-09T05:08:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.722+0000 7fe883896640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 0x7fe87c0ca4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.722+0000 7fe883896640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0c7500 0x7fe87c15f470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:25.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe883896640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c1570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe883896640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c15d6d0 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe883896640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe87c156f00 con 0x7fe87c0c7500 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe883896640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe87c0c9cd0 con 0x7fe87c0c6a80 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c1570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c1570c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49250/0 (socket says 172.21.15.1:49250) 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 -- 172.21.15.1:0/2221039198 learned_addr learned my addr 172.21.15.1:0/2221039198 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881632640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 0x7fe87c0ca4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe880e31640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0c7500 0x7fe87c15f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881632640 1 -- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 msgr2=0x7fe87c0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881632640 1 -- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 msgr2=0x7fe87c0ca4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881632640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 0x7fe87c0ca4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881632640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 0x7fe87c0ca4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 -- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 msgr2=0x7fe87c0ca4f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 0x7fe87c0ca4f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 -- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0c7500 msgr2=0x7fe87c15f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:25.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0c7500 0x7fe87c15f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 -- 172.21.15.1:0/2221039198 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe87c0cae20 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe881e33640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c1570c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00c6e0 tx=0x7fe86c00cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe879ffb640 1 -- 172.21.15.1:0/2221039198 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c003b70 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.723+0000 7fe879ffb640 1 -- 172.21.15.1:0/2221039198 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe86c003d10 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.724+0000 7fe879ffb640 1 -- 172.21.15.1:0/2221039198 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe86c0052f0 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.724+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe87c0cb100 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.724+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe87c15baf0 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.725+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe87c050870 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.725+0000 7fe879ffb640 1 -- 172.21.15.1:0/2221039198 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe86c005490 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.726+0000 7fe879ffb640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe8680762d0 0x7fe8680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.726+0000 7fe879ffb640 1 -- 172.21.15.1:0/2221039198 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe86c0974c0 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.726+0000 7fe881632640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe8680762d0 0x7fe8680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:25.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.727+0000 7fe881632640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe8680762d0 0x7fe8680787e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe874034000 tx=0x7fe87402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:25.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.731+0000 7fe879ffb640 1 -- 172.21.15.1:0/2221039198 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe86c060d20 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.993+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe87c15a1f0 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.994+0000 7fe879ffb640 1 -- 172.21.15.1:0/2221039198 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fe86c060620 con 0x7fe87c0cc4c0 2024-05-09T05:08:25.995 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":20,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":6,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe8680762d0 msgr2=0x7fe8680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe8680762d0 0x7fe8680787e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe874034000 tx=0x7fe87402e000 comp rx=0 tx=0).stop 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c0cc4c0 msgr2=0x7fe87c1570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c1570c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe86c00c6e0 tx=0x7fe86c00cba0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 shutdown_connections 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe8680762d0 0x7fe8680787e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe87c0c7500 0x7fe87c15f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe87c0cc4c0 0x7fe87c1570c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 --2- 172.21.15.1:0/2221039198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe87c0c6a80 0x7fe87c0ca4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 >> 172.21.15.1:0/2221039198 conn(0x7fe87c068650 msgr2=0x7fe87c155310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 shutdown_connections 2024-05-09T05:08:25.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:25.996+0000 7fe883896640 1 -- 172.21.15.1:0/2221039198 wait complete. 2024-05-09T05:08:26.192 INFO:tasks.workunit.client.0.smithi001.stdout:checking dependency style of gcc... gcc3 2024-05-09T05:08:26.274 INFO:tasks.workunit.client.0.smithi001.stdout:checking how to run the C preprocessor... gcc -E 2024-05-09T05:08:26.335 INFO:tasks.workunit.client.0.smithi001.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-05-09T05:08:26.336 INFO:tasks.workunit.client.0.smithi001.stdout:checking for egrep... /usr/bin/grep -E 2024-05-09T05:08:26.350 INFO:tasks.workunit.client.0.smithi001.stdout:checking for AIX... no 2024-05-09T05:08:26.384 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gcc... (cached) gcc 2024-05-09T05:08:26.398 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-05-09T05:08:26.398 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether gcc accepts -g... (cached) yes 2024-05-09T05:08:26.399 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-05-09T05:08:26.400 INFO:tasks.workunit.client.0.smithi001.stdout:checking dependency style of gcc... (cached) gcc3 2024-05-09T05:08:26.505 INFO:tasks.workunit.client.0.smithi001.stdout:checking for main in -lm... yes 2024-05-09T05:08:26.584 INFO:tasks.workunit.client.0.smithi001.stdout:checking for main in -lpthread... yes 2024-05-09T05:08:26.797 INFO:tasks.workunit.client.0.smithi001.stdout:checking for ANSI C header files... yes 2024-05-09T05:08:26.866 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-05-09T05:08:26.926 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/types.h... yes 2024-05-09T05:08:26.993 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/stat.h... yes 2024-05-09T05:08:27.063 INFO:tasks.workunit.client.0.smithi001.stdout:checking for stdlib.h... yes 2024-05-09T05:08:27.133 INFO:tasks.workunit.client.0.smithi001.stdout:checking for string.h... yes 2024-05-09T05:08:27.208 INFO:tasks.workunit.client.0.smithi001.stdout:checking for memory.h... yes 2024-05-09T05:08:27.278 INFO:tasks.workunit.client.0.smithi001.stdout:checking for strings.h... yes 2024-05-09T05:08:27.343 INFO:tasks.workunit.client.0.smithi001.stdout:checking for inttypes.h... yes 2024-05-09T05:08:27.409 INFO:tasks.workunit.client.0.smithi001.stdout:checking for stdint.h... yes 2024-05-09T05:08:27.476 INFO:tasks.workunit.client.0.smithi001.stdout:checking for unistd.h... yes 2024-05-09T05:08:27.549 INFO:tasks.workunit.client.0.smithi001.stdout:checking pthread.h usability... yes 2024-05-09T05:08:27.579 INFO:tasks.workunit.client.0.smithi001.stdout:checking pthread.h presence... yes 2024-05-09T05:08:27.580 INFO:tasks.workunit.client.0.smithi001.stdout:checking for pthread.h... yes 2024-05-09T05:08:27.653 INFO:tasks.workunit.client.0.smithi001.stdout:checking fcntl.h usability... yes 2024-05-09T05:08:27.682 INFO:tasks.workunit.client.0.smithi001.stdout:checking fcntl.h presence... yes 2024-05-09T05:08:27.682 INFO:tasks.workunit.client.0.smithi001.stdout:checking for fcntl.h... yes 2024-05-09T05:08:27.753 INFO:tasks.workunit.client.0.smithi001.stdout:checking limits.h usability... yes 2024-05-09T05:08:27.783 INFO:tasks.workunit.client.0.smithi001.stdout:checking limits.h presence... yes 2024-05-09T05:08:27.783 INFO:tasks.workunit.client.0.smithi001.stdout:checking for limits.h... yes 2024-05-09T05:08:27.811 INFO:tasks.workunit.client.0.smithi001.stdout:checking for stdint.h... (cached) yes 2024-05-09T05:08:27.885 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/time.h usability... yes 2024-05-09T05:08:27.914 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/time.h presence... yes 2024-05-09T05:08:27.915 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/time.h... yes 2024-05-09T05:08:27.941 INFO:tasks.workunit.client.0.smithi001.stdout:checking for unistd.h... (cached) yes 2024-05-09T05:08:28.007 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/vfs.h usability... yes 2024-05-09T05:08:28.037 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/vfs.h presence... yes 2024-05-09T05:08:28.037 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/vfs.h... yes 2024-05-09T05:08:28.105 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/limits.h usability... no 2024-05-09T05:08:28.137 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/limits.h presence... no 2024-05-09T05:08:28.138 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/limits.h... no 2024-05-09T05:08:28.190 INFO:tasks.workunit.client.0.smithi001.stdout:checking for an ANSI C-conforming const... yes 2024-05-09T05:08:28.236 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-05-09T05:08:28.239 INFO:tasks.workunit.client.0.smithi001.stdout:checking for function prototypes... yes 2024-05-09T05:08:28.283 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether setvbuf arguments are reversed... no 2024-05-09T05:08:28.370 INFO:tasks.workunit.client.0.smithi001.stdout:checking for system... yes 2024-05-09T05:08:28.459 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gettimeofday... yes 2024-05-09T05:08:28.549 INFO:tasks.workunit.client.0.smithi001.stdout:checking for mkdir... yes 2024-05-09T05:08:28.638 INFO:tasks.workunit.client.0.smithi001.stdout:checking for strerror... yes 2024-05-09T05:08:28.733 INFO:tasks.workunit.client.0.smithi001.stdout:checking for open64... yes 2024-05-09T05:08:28.821 INFO:tasks.workunit.client.0.smithi001.stdout:checking for stat64... yes 2024-05-09T05:08:28.911 INFO:tasks.workunit.client.0.smithi001.stdout:checking for fseeko64... yes 2024-05-09T05:08:29.004 INFO:tasks.workunit.client.0.smithi001.stdout:checking for lrand48_r... yes 2024-05-09T05:08:29.008 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:29.102 INFO:tasks.workunit.client.0.smithi001.stdout:checking for srand48_r... yes 2024-05-09T05:08:29.132 INFO:tasks.workunit.client.0.smithi001.stdout:configure: creating ./config.status 2024-05-09T05:08:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.132+0000 7f04a06bd640 1 Processor -- start 2024-05-09T05:08:29.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.132+0000 7f04a06bd640 1 -- start start 2024-05-09T05:08:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.134+0000 7f04a06bd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f049815b350 0x7f0498166060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.134+0000 7f04a06bd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04980c9e30 con 0x7f04981548d0 2024-05-09T05:08:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.134+0000 7f04a06bd640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04980c9970 con 0x7f049815af40 2024-05-09T05:08:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.134+0000 7f04a06bd640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f04980cacd0 con 0x7f049815b350 2024-05-09T05:08:29.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.134+0000 7f049e459640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f04981548d0 0x7f049815e680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58216/0 (socket says 172.21.15.1:58216) 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.134+0000 7f049e459640 1 -- 172.21.15.1:0/1040875862 learned_addr learned my addr 172.21.15.1:0/1040875862 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.134+0000 7f049ec5a640 1 --2- 172.21.15.1:0/1040875862 >> v2:172.21.15.196:3300/0 conn(0x7f049815b350 0x7f0498166060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.135+0000 7f049d457640 1 -- 172.21.15.1:0/1040875862 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2216314804 0 0) 0x7f04980c9970 con 0x7f049815af40 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.135+0000 7f049d457640 1 -- 172.21.15.1:0/1040875862 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0484003520 con 0x7f049815af40 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.135+0000 7f049d457640 1 -- 172.21.15.1:0/1040875862 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1034946937 0 0) 0x7f04980c9e30 con 0x7f04981548d0 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.135+0000 7f049d457640 1 -- 172.21.15.1:0/1040875862 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f04980c9970 con 0x7f04981548d0 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.135+0000 7f049ec5a640 1 -- 172.21.15.1:0/1040875862 >> v1:172.21.15.196:6789/0 conn(0x7f049815af40 legacy=0x7f04980c7f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.135+0000 7f049ec5a640 1 -- 172.21.15.1:0/1040875862 >> v1:172.21.15.1:6789/0 conn(0x7f04981548d0 legacy=0x7f049815e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.135+0000 7f049ec5a640 1 -- 172.21.15.1:0/1040875862 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04980cafb0 con 0x7f049815b350 2024-05-09T05:08:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.136+0000 7f049ec5a640 1 --2- 172.21.15.1:0/1040875862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 0x7f0498166060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0490009d90 tx=0x7f049002a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2cdb017d093a17a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.136+0000 7f049d457640 1 -- 172.21.15.1:0/1040875862 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490007740 con 0x7f049815b350 2024-05-09T05:08:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.136+0000 7f049d457640 1 -- 172.21.15.1:0/1040875862 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04900078e0 con 0x7f049815b350 2024-05-09T05:08:29.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.136+0000 7f049d457640 1 -- 172.21.15.1:0/1040875862 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490007cb0 con 0x7f049815b350 2024-05-09T05:08:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.137+0000 7f04a06bd640 1 -- 172.21.15.1:0/1040875862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 msgr2=0x7f0498166060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.137+0000 7f04a06bd640 1 --2- 172.21.15.1:0/1040875862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 0x7f0498166060 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f0490009d90 tx=0x7f049002a6f0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.137+0000 7f04a06bd640 1 -- 172.21.15.1:0/1040875862 shutdown_connections 2024-05-09T05:08:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.137+0000 7f04a06bd640 1 --2- 172.21.15.1:0/1040875862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 0x7f0498166060 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.137+0000 7f04a06bd640 1 -- 172.21.15.1:0/1040875862 >> 172.21.15.1:0/1040875862 conn(0x7f04980745b0 msgr2=0x7f04980719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.137+0000 7f04a06bd640 1 -- 172.21.15.1:0/1040875862 shutdown_connections 2024-05-09T05:08:29.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.137+0000 7f04a06bd640 1 -- 172.21.15.1:0/1040875862 wait complete. 2024-05-09T05:08:29.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.139+0000 7f04a06bd640 1 Processor -- start 2024-05-09T05:08:29.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.140+0000 7f04a06bd640 1 -- start start 2024-05-09T05:08:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f04a06bd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04981548d0 0x7f0498153450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f04a06bd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f049815af40 0x7f04981e52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f04a06bd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 0x7f0498143140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f04a06bd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f049815a6d0 con 0x7f04981548d0 2024-05-09T05:08:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f04a06bd640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f049806cde0 con 0x7f049815b350 2024-05-09T05:08:29.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f04a06bd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0498154ce0 con 0x7f049815af40 2024-05-09T05:08:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f049e459640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04981548d0 0x7f0498153450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f049e459640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04981548d0 0x7f0498153450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49262/0 (socket says 172.21.15.1:49262) 2024-05-09T05:08:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f049e459640 1 -- 172.21.15.1:0/1155155099 learned_addr learned my addr 172.21.15.1:0/1155155099 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.141+0000 7f049dc58640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f049815af40 0x7f04981e52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f049ec5a640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 0x7f0498143140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f049e459640 1 -- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f049815af40 msgr2=0x7f04981e52f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f049e459640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f049815af40 0x7f04981e52f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f049e459640 1 -- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 msgr2=0x7f0498143140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f049e459640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 0x7f0498143140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f049e459640 1 -- 172.21.15.1:0/1155155099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0498146180 con 0x7f04981548d0 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f049e459640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04981548d0 0x7f0498153450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0494010b10 tx=0x7f049400e060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f048effd640 1 -- 172.21.15.1:0/1155155099 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049401b070 con 0x7f04981548d0 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0498146800 con 0x7f04981548d0 2024-05-09T05:08:29.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.142+0000 7f048effd640 1 -- 172.21.15.1:0/1155155099 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f049400ec70 con 0x7f04981548d0 2024-05-09T05:08:29.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.143+0000 7f048effd640 1 -- 172.21.15.1:0/1155155099 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f049400ee60 con 0x7f04981548d0 2024-05-09T05:08:29.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.143+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0498152aa0 con 0x7f04981548d0 2024-05-09T05:08:29.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.144+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0498050870 con 0x7f04981548d0 2024-05-09T05:08:29.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.145+0000 7f048effd640 1 -- 172.21.15.1:0/1155155099 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f049401e070 con 0x7f04981548d0 2024-05-09T05:08:29.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.145+0000 7f048effd640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f047c076300 0x7f047c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.145+0000 7f048effd640 1 -- 172.21.15.1:0/1155155099 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f049409b410 con 0x7f04981548d0 2024-05-09T05:08:29.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.145+0000 7f049dc58640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f047c076300 0x7f047c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.146+0000 7f049dc58640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f047c076300 0x7f047c078810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0488007fd0 tx=0x7f0488009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.153+0000 7f048effd640 1 -- 172.21.15.1:0/1155155099 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0494064590 con 0x7f04981548d0 2024-05-09T05:08:29.289 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: creating Makefile 2024-05-09T05:08:29.309 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: creating config.h 2024-05-09T05:08:29.338 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: executing depfiles commands 2024-05-09T05:08:29.418 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: executing default commands 2024-05-09T05:08:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.470+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f04980ca090 con 0x7f04981548d0 2024-05-09T05:08:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.470+0000 7f048effd640 1 -- 172.21.15.1:0/1155155099 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f0494068890 con 0x7f04981548d0 2024-05-09T05:08:29.471 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:29.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.472+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f047c076300 msgr2=0x7f047c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.472+0000 7f04a06bd640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f047c076300 0x7f047c078810 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f0488007fd0 tx=0x7f0488009040 comp rx=0 tx=0).stop 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.472+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04981548d0 msgr2=0x7f0498153450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.472+0000 7f04a06bd640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04981548d0 0x7f0498153450 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0494010b10 tx=0x7f049400e060 comp rx=0 tx=0).stop 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.473+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 shutdown_connections 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.473+0000 7f04a06bd640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f047c076300 0x7f047c078810 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.473+0000 7f04a06bd640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f049815b350 0x7f0498143140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.473+0000 7f04a06bd640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f049815af40 0x7f04981e52f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.473+0000 7f04a06bd640 1 --2- 172.21.15.1:0/1155155099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04981548d0 0x7f0498153450 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.473+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 >> 172.21.15.1:0/1155155099 conn(0x7f04980745b0 msgr2=0x7f0498158280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.473+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 shutdown_connections 2024-05-09T05:08:29.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.473+0000 7f04a06bd640 1 -- 172.21.15.1:0/1155155099 wait complete. 2024-05-09T05:08:29.484 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:08:29.485 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:08:29.485 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:08:29.486 INFO:tasks.workunit.client.0.smithi001.stderr:+ make 2024-05-09T05:08:29.518 INFO:tasks.workunit.client.0.smithi001.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-05-09T05:08:29.526 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-05-09T05:08:29.528 INFO:tasks.workunit.client.0.smithi001.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-05-09T05:08:29.528 INFO:tasks.workunit.client.0.smithi001.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-05-09T05:08:29.528 INFO:tasks.workunit.client.0.smithi001.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-05-09T05:08:29.528 INFO:tasks.workunit.client.0.smithi001.stderr: any GNU archive site. 2024-05-09T05:08:29.534 INFO:tasks.workunit.client.0.smithi001.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-05-09T05:08:29.540 INFO:tasks.workunit.client.0.smithi001.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-05-09T05:08:29.541 INFO:tasks.workunit.client.0.smithi001.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-05-09T05:08:29.541 INFO:tasks.workunit.client.0.smithi001.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-05-09T05:08:29.541 INFO:tasks.workunit.client.0.smithi001.stderr: You might want to install the `Automake' and `Perl' packages. 2024-05-09T05:08:29.541 INFO:tasks.workunit.client.0.smithi001.stderr: Grab them from any GNU archive site. 2024-05-09T05:08:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.617+0000 7fc0c5b7d640 1 Processor -- start 2024-05-09T05:08:29.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.617+0000 7fc0c5b7d640 1 -- start start 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.618+0000 7fc0c5b7d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c0153800 0x7fc0c00745d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.618+0000 7fc0c5b7d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c00c5510 con 0x7fc0c00c7ac0 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.618+0000 7fc0c5b7d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c0153c10 con 0x7fc0c00c9410 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.618+0000 7fc0c5b7d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c0073e10 con 0x7fc0c0153800 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.618+0000 7fc0bf7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc0c00c7ac0 0x7fc0c00c4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58218/0 (socket says 172.21.15.1:58218) 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.618+0000 7fc0bf7fe640 1 -- 172.21.15.1:0/1240152216 learned_addr learned my addr 172.21.15.1:0/1240152216 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.618+0000 7fc0beffd640 1 --2- 172.21.15.1:0/1240152216 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c0153800 0x7fc0c00745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.619+0000 7fc0be7fc640 1 -- 172.21.15.1:0/1240152216 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3546693466 0 0) 0x7fc0c00c5510 con 0x7fc0c00c7ac0 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.619+0000 7fc0be7fc640 1 -- 172.21.15.1:0/1240152216 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0a8003520 con 0x7fc0c00c7ac0 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.619+0000 7fc0be7fc640 1 -- 172.21.15.1:0/1240152216 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 262312852 0 0) 0x7fc0c0153c10 con 0x7fc0c00c9410 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.619+0000 7fc0be7fc640 1 -- 172.21.15.1:0/1240152216 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0c00c5510 con 0x7fc0c00c9410 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.619+0000 7fc0beffd640 1 -- 172.21.15.1:0/1240152216 >> v1:172.21.15.196:6789/0 conn(0x7fc0c00c9410 legacy=0x7fc0c0163e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.619+0000 7fc0beffd640 1 -- 172.21.15.1:0/1240152216 >> v1:172.21.15.1:6789/0 conn(0x7fc0c00c7ac0 legacy=0x7fc0c00c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.619+0000 7fc0beffd640 1 -- 172.21.15.1:0/1240152216 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c00c5b00 con 0x7fc0c0153800 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.620+0000 7fc0beffd640 1 --2- 172.21.15.1:0/1240152216 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c0153800 0x7fc0c00745d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40097d0 tx=0x7fc0b402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1cda66ab99ac10f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.620+0000 7fc0be7fc640 1 -- 172.21.15.1:0/1240152216 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4037070 con 0x7fc0c0153800 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.620+0000 7fc0be7fc640 1 -- 172.21.15.1:0/1240152216 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b4032440 con 0x7fc0c0153800 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.620+0000 7fc0be7fc640 1 -- 172.21.15.1:0/1240152216 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4032810 con 0x7fc0c0153800 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.621+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/1240152216 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c0153800 msgr2=0x7fc0c00745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.621+0000 7fc0c5b7d640 1 --2- 172.21.15.1:0/1240152216 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c0153800 0x7fc0c00745d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40097d0 tx=0x7fc0b402a210 comp rx=0 tx=0).stop 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.621+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/1240152216 shutdown_connections 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.621+0000 7fc0c5b7d640 1 --2- 172.21.15.1:0/1240152216 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c0153800 0x7fc0c00745d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.621+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/1240152216 >> 172.21.15.1:0/1240152216 conn(0x7fc0c0070f40 msgr2=0x7fc0c006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.621+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/1240152216 shutdown_connections 2024-05-09T05:08:29.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.621+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/1240152216 wait complete. 2024-05-09T05:08:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.623+0000 7fc0c5b7d640 1 Processor -- start 2024-05-09T05:08:29.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.623+0000 7fc0c5b7d640 1 -- start start 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0c5b7d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00c7ac0 0x7fc0c01f2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0c5b7d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c00c9410 0x7fc0c0157270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0c5b7d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c0153800 0x7fc0c014f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0c5b7d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c00c63a0 con 0x7fc0c00c7ac0 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0c5b7d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c00c60c0 con 0x7fc0c0153800 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0c5b7d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c00740b0 con 0x7fc0c00c9410 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00c7ac0 0x7fc0c01f2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0bf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00c7ac0 0x7fc0c01f2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49268/0 (socket says 172.21.15.1:49268) 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0bf7fe640 1 -- 172.21.15.1:0/875066537 learned_addr learned my addr 172.21.15.1:0/875066537 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:29.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0bffff640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c0153800 0x7fc0c014f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.624+0000 7fc0beffd640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c00c9410 0x7fc0c0157270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0beffd640 1 -- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c0153800 msgr2=0x7fc0c014f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0beffd640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c0153800 0x7fc0c014f7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0beffd640 1 -- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00c7ac0 msgr2=0x7fc0c01f2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0beffd640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00c7ac0 0x7fc0c01f2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0beffd640 1 -- 172.21.15.1:0/875066537 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c01f2e70 con 0x7fc0c00c9410 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0bf7fe640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00c7ac0 0x7fc0c01f2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0beffd640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c00c9410 0x7fc0c0157270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4031000 tx=0x7fc0b40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4041020 con 0x7fc0c00c9410 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b403d620 con 0x7fc0c00c9410 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c015a730 con 0x7fc0c00c9410 2024-05-09T05:08:29.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4037040 con 0x7fc0c00c9410 2024-05-09T05:08:29.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.625+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c015acf0 con 0x7fc0c00c9410 2024-05-09T05:08:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.626+0000 7fc0c5b7d640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00ce500 0x7fc0c014e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.626+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c01f3340 con 0x7fc0c00ce500 2024-05-09T05:08:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.627+0000 7fc0bf7fe640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00ce500 0x7fc0c014e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.627+0000 7fc0bf7fe640 1 -- 172.21.15.1:0/875066537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc0c0050870 con 0x7fc0c00ce500 2024-05-09T05:08:29.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.627+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc0b4044020 con 0x7fc0c00c9410 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.627+0000 7fc0bf7fe640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00ce500 0x7fc0c014e180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000fae0 tx=0x7fc0b0001e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0a7fff640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ac0762d0 0x7fc0ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc0b40b7a30 con 0x7fc0c00c9410 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0bffff640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ac0762d0 0x7fc0ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b0005ac0 con 0x7fc0c00ce500 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc0b0018070 con 0x7fc0c00ce500 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00ce500 msgr2=0x7fc0c014e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0a7fff640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00ce500 0x7fc0c014e180 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000fae0 tx=0x7fc0b0001e50 comp rx=0 tx=0).stop 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00ce500 msgr2=0x7fc0c014e180 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0a7fff640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c00ce500 0x7fc0c014e180 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.628+0000 7fc0bffff640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ac0762d0 0x7fc0ac0787e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0150860 tx=0x7fc0b8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.635+0000 7fc0c5b7d640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c014e6c0 0x7fc0c00c7ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.635+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c0073bf0 con 0x7fc0c014e6c0 2024-05-09T05:08:29.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.635+0000 7fc0bf7fe640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c014e6c0 0x7fc0c00c7ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.635+0000 7fc0bf7fe640 1 -- 172.21.15.1:0/875066537 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc0c01577b0 con 0x7fc0c014e6c0 2024-05-09T05:08:29.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0bf7fe640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c014e6c0 0x7fc0c00c7ff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000fae0 tx=0x7fc0b00057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b001eaf0 con 0x7fc0c014e6c0 2024-05-09T05:08:29.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fc0b000d7b0 con 0x7fc0c014e6c0 2024-05-09T05:08:29.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c014e6c0 msgr2=0x7fc0c00c7ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0a7fff640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c014e6c0 0x7fc0c00c7ff0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000fae0 tx=0x7fc0b00057f0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0a7fff640 1 -- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c014e6c0 msgr2=0x7fc0c00c7ff0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:29.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0a7fff640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c014e6c0 0x7fc0c00c7ff0 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fc0b000fae0 tx=0x7fc0b00057f0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.637 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 24, 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15714, 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:29.638 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:29.639 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:29.640 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:08:29.641 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:29.642 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.643 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.644 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.645 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:29.646 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.647 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:08:29.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.649 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:08:29.650 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:08:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ac0762d0 msgr2=0x7fc0ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0c5b7d640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ac0762d0 0x7fc0ac0787e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0150860 tx=0x7fc0b8009040 comp rx=0 tx=0).stop 2024-05-09T05:08:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c00c9410 msgr2=0x7fc0c0157270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0c5b7d640 1 --2- 172.21.15.1:0/875066537 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c00c9410 0x7fc0c0157270 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4031000 tx=0x7fc0b40046c0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0bffff640 1 -- 172.21.15.1:0/875066537 reap_dead start 2024-05-09T05:08:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0bffff640 1 -- 172.21.15.1:0/875066537 reap_dead start 2024-05-09T05:08:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 shutdown_connections 2024-05-09T05:08:29.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.636+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 >> 172.21.15.1:0/875066537 conn(0x7fc0c0070f40 msgr2=0x7fc0c0159850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.637+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 shutdown_connections 2024-05-09T05:08:29.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.637+0000 7fc0c5b7d640 1 -- 172.21.15.1:0/875066537 wait complete. 2024-05-09T05:08:29.652 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:08:29.652 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:08:29.652 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:08:29.732 INFO:tasks.workunit.client.0.smithi001.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-05-09T05:08:29.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.810+0000 7fcb21786640 1 Processor -- start 2024-05-09T05:08:29.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.810+0000 7fcb21786640 1 -- start start 2024-05-09T05:08:29.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.811+0000 7fcb21786640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb1c0b2850 0x7fcb1c154510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.811+0000 7fcb21786640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcb1c0cab00 0x7fcb1c0cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.811+0000 7fcb21786640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcb1c154a50 con 0x7fcb1c0ca4a0 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.811+0000 7fcb21786640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcb1c070c60 con 0x7fcb1c0b2850 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.811+0000 7fcb21786640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fcb1c15a800 con 0x7fcb1c0cab00 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.811+0000 7fcb1affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcb1c0b2850 0x7fcb1c154510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.811+0000 7fcb1b7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcb1c0ca4a0 0x7fcb1c0710f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41408/0 (socket says 172.21.15.1:41408) 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.811+0000 7fcb1b7fe640 1 -- 172.21.15.1:0/3287789083 learned_addr learned my addr 172.21.15.1:0/3287789083 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.812+0000 7fcb127fc640 1 --2- 172.21.15.1:0/3287789083 >> v2:172.21.15.196:3300/0 conn(0x7fcb1c0cab00 0x7fcb1c0cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.812+0000 7fcb1affd640 1 -- 172.21.15.1:0/3287789083 >> v2:172.21.15.196:3300/0 conn(0x7fcb1c0cab00 msgr2=0x7fcb1c0cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.812+0000 7fcb1affd640 1 --2- 172.21.15.1:0/3287789083 >> v2:172.21.15.196:3300/0 conn(0x7fcb1c0cab00 0x7fcb1c0cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.812+0000 7fcb1affd640 1 -- 172.21.15.1:0/3287789083 >> v1:172.21.15.1:6789/0 conn(0x7fcb1c0ca4a0 legacy=0x7fcb1c0710f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.812+0000 7fcb1a7fc640 1 -- 172.21.15.1:0/3287789083 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1004060883 0 0) 0x7fcb1c154a50 con 0x7fcb1c0ca4a0 2024-05-09T05:08:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.812+0000 7fcb1affd640 1 -- 172.21.15.1:0/3287789083 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c15aae0 con 0x7fcb1c0b2850 2024-05-09T05:08:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.812+0000 7fcb1affd640 1 --2- 172.21.15.1:0/3287789083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 0x7fcb1c154510 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c073fa0 tx=0x7fcb0c02a110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7fb13cd99b60f03e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb1a7fc640 1 -- 172.21.15.1:0/3287789083 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c037070 con 0x7fcb1c0b2850 2024-05-09T05:08:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb1a7fc640 1 -- 172.21.15.1:0/3287789083 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0c032440 con 0x7fcb1c0b2850 2024-05-09T05:08:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb1a7fc640 1 -- 172.21.15.1:0/3287789083 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c02ae60 con 0x7fcb1c0b2850 2024-05-09T05:08:29.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb21786640 1 -- 172.21.15.1:0/3287789083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 msgr2=0x7fcb1c154510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb21786640 1 --2- 172.21.15.1:0/3287789083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 0x7fcb1c154510 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c073fa0 tx=0x7fcb0c02a110 comp rx=0 tx=0).stop 2024-05-09T05:08:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb21786640 1 -- 172.21.15.1:0/3287789083 shutdown_connections 2024-05-09T05:08:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb21786640 1 --2- 172.21.15.1:0/3287789083 >> v2:172.21.15.196:3300/0 conn(0x7fcb1c0cab00 0x7fcb1c0cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb21786640 1 --2- 172.21.15.1:0/3287789083 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 0x7fcb1c154510 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb21786640 1 -- 172.21.15.1:0/3287789083 >> 172.21.15.1:0/3287789083 conn(0x7fcb1c072c30 msgr2=0x7fcb1c073040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb21786640 1 -- 172.21.15.1:0/3287789083 shutdown_connections 2024-05-09T05:08:29.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.813+0000 7fcb21786640 1 -- 172.21.15.1:0/3287789083 wait complete. 2024-05-09T05:08:29.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.815+0000 7fcb21786640 1 Processor -- start 2024-05-09T05:08:29.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.815+0000 7fcb21786640 1 -- start start 2024-05-09T05:08:29.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.816+0000 7fcb21786640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 0x7fcb1c1f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.816+0000 7fcb21786640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcb1c0ca4a0 0x7fcb1c14f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb21786640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0cab00 0x7fcb1c0cda90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb21786640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c15b380 con 0x7fcb1c0b2850 2024-05-09T05:08:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb21786640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c15b0a0 con 0x7fcb1c0cab00 2024-05-09T05:08:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb21786640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcb1c071800 con 0x7fcb1c0ca4a0 2024-05-09T05:08:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 0x7fcb1c1f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 0x7fcb1c1f60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60096/0 (socket says 172.21.15.1:60096) 2024-05-09T05:08:29.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 -- 172.21.15.1:0/2844739953 learned_addr learned my addr 172.21.15.1:0/2844739953 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1b7fe640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0cab00 0x7fcb1c0cda90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb127fc640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcb1c0ca4a0 0x7fcb1c14f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 -- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcb1c0ca4a0 msgr2=0x7fcb1c14f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcb1c0ca4a0 0x7fcb1c14f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 -- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0cab00 msgr2=0x7fcb1c0cda90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0cab00 0x7fcb1c0cda90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 -- 172.21.15.1:0/2844739953 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb1c1f6630 con 0x7fcb1c0b2850 2024-05-09T05:08:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1b7fe640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0cab00 0x7fcb1c0cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:08:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb1affd640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 0x7fcb1c1f60f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c009890 tx=0x7fcb0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c037040 con 0x7fcb1c0b2850 2024-05-09T05:08:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcb0c042070 con 0x7fcb1c0b2850 2024-05-09T05:08:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb1c1f5bf0 con 0x7fcb1c0b2850 2024-05-09T05:08:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0c030920 con 0x7fcb1c0b2850 2024-05-09T05:08:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.817+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb1c0d0440 con 0x7fcb1c0b2850 2024-05-09T05:08:29.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.818+0000 7fcb21786640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0b0d70 0x7fcb1c0cdfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.818+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c1f5a70 con 0x7fcb1c0b0d70 2024-05-09T05:08:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.818+0000 7fcb127fc640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0b0d70 0x7fcb1c0cdfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.819+0000 7fcb127fc640 1 -- 172.21.15.1:0/2844739953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcb1c050870 con 0x7fcb1c0b0d70 2024-05-09T05:08:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.819+0000 7fcb127fc640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0b0d70 0x7fcb1c0cdfd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c06fae0 tx=0x7fcb04010c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.819+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcb0c030ac0 con 0x7fcb1c0b2850 2024-05-09T05:08:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb13fff640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcafc0762d0 0x7fcafc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcb0c0b8050 con 0x7fcb1c0b2850 2024-05-09T05:08:29.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb1b7fe640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcafc0762d0 0x7fcafc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0400f040 con 0x7fcb1c0b0d70 2024-05-09T05:08:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcb0400e9c0 con 0x7fcb1c0b0d70 2024-05-09T05:08:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0b0d70 msgr2=0x7fcb1c0cdfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb13fff640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0b0d70 0x7fcb1c0cdfd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c06fae0 tx=0x7fcb04010c20 comp rx=0 tx=0).stop 2024-05-09T05:08:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0b0d70 msgr2=0x7fcb1c0cdfd0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb13fff640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0b0d70 0x7fcb1c0cdfd0 secure :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c06fae0 tx=0x7fcb04010c20 comp rx=0 tx=0).stop 2024-05-09T05:08:29.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.820+0000 7fcb1b7fe640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcafc0762d0 0x7fcafc0787e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb140012e0 tx=0x7fcb14009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.826+0000 7fcb21786640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0d0680 0x7fcb1c1f2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:29.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.826+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcb1c0cff10 con 0x7fcb1c0d0680 2024-05-09T05:08:29.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.827+0000 7fcb127fc640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0d0680 0x7fcb1c1f2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:29.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.827+0000 7fcb127fc640 1 -- 172.21.15.1:0/2844739953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcb1c0cb000 con 0x7fcb1c0d0680 2024-05-09T05:08:29.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.827+0000 7fcb127fc640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0d0680 0x7fcb1c1f2500 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c06fae0 tx=0x7fcb04004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:29.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.827+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb0401cc20 con 0x7fcb1c0d0680 2024-05-09T05:08:29.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5061 (secure 0 0 0) 0x7fcb0401cc00 con 0x7fcb1c0d0680 2024-05-09T05:08:29.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0d0680 msgr2=0x7fcb1c1f2500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb13fff640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0d0680 0x7fcb1c1f2500 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcb1c06fae0 tx=0x7fcb04004730 comp rx=0 tx=0).stop 2024-05-09T05:08:29.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb13fff640 1 -- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0d0680 msgr2=0x7fcb1c1f2500 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:29.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb13fff640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcb1c0d0680 0x7fcb1c1f2500 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:08:29.832 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 24, 2024-05-09T05:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 173917, 2024-05-09T05:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:08:29.833 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:29.834 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:08:29.834 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:08:29.834 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:29.834 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:29.834 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:29.834 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:29.835 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:29.835 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:29.835 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:29.835 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:29.835 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:29.835 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:29.835 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:29.835 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.836 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:08:29.836 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:08:29.836 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:29.836 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:29.836 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:29.836 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:29.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:29.836 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:29.837 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:29.837 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:29.837 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:29.837 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:29.837 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:29.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.837 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:08:29.838 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:08:29.838 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:08:29.838 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:08:29.838 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:08:29.838 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:08:29.838 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:29.838 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:29.838 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:08:29.839 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:29.840 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:29.840 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:29.840 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:29.840 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:29.840 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:29.840 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:29.840 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:29.840 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:29.841 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:29.841 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:29.841 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.841 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:08:29.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.841 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:08:29.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.841 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:29.842 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:29.842 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:29.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:29.842 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.842 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:29.842 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:08:29.842 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.843 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:29.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:08:29.843 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.843 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.843 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.843 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:08:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:08:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:29.844 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.845 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:29.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:08:29.846 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.846 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.846 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.846 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.846 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:08:29.846 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:08:29.846 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.847 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.848 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:08:29.849 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:29.850 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:08:29.851 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcafc0762d0 msgr2=0x7fcafc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb21786640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcafc0762d0 0x7fcafc0787e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fcb140012e0 tx=0x7fcb14009040 comp rx=0 tx=0).stop 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 msgr2=0x7fcb1c1f60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb21786640 1 --2- 172.21.15.1:0/2844739953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcb1c0b2850 0x7fcb1c1f60f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcb0c009890 tx=0x7fcb0c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb1b7fe640 1 -- 172.21.15.1:0/2844739953 reap_dead start 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb1b7fe640 1 -- 172.21.15.1:0/2844739953 reap_dead start 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 shutdown_connections 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 >> 172.21.15.1:0/2844739953 conn(0x7fcb1c072c30 msgr2=0x7fcb1c1582d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 shutdown_connections 2024-05-09T05:08:29.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:29.828+0000 7fcb21786640 1 -- 172.21.15.1:0/2844739953 wait complete. 2024-05-09T05:08:29.854 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:08:29.854 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:08:29.854 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:08:30.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.010+0000 7ff0c01e3640 1 Processor -- start 2024-05-09T05:08:30.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.010+0000 7ff0c01e3640 1 -- start start 2024-05-09T05:08:30.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.011+0000 7ff0c01e3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff0b80b4ec0 0x7ff0b80cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.011+0000 7ff0c01e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 0x7ff0b815fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.011+0000 7ff0c01e3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0b8160290 0x7ff0b81606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.011+0000 7ff0c01e3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b815d0c0 con 0x7ff0b8160290 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.011+0000 7ff0c01e3640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b8160be0 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.011+0000 7ff0c01e3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b815c7b0 con 0x7ff0b80ce010 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.011+0000 7ff0be780640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0b8160290 0x7ff0b81606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0be780640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0b8160290 0x7ff0b81606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60108/0 (socket says 172.21.15.1:60108) 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0bdf7f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff0b80b4ec0 0x7ff0b80cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0be780640 1 -- 172.21.15.1:0/1886541329 learned_addr learned my addr 172.21.15.1:0/1886541329 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0bd77e640 1 --2- 172.21.15.1:0/1886541329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 0x7ff0b815fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0be780640 1 -- 172.21.15.1:0/1886541329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 msgr2=0x7ff0b815fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0be780640 1 --2- 172.21.15.1:0/1886541329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 0x7ff0b815fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0be780640 1 -- 172.21.15.1:0/1886541329 >> v2:172.21.15.196:3300/0 conn(0x7ff0b80b4ec0 msgr2=0x7ff0b80cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:30.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0be780640 1 --2- 172.21.15.1:0/1886541329 >> v2:172.21.15.196:3300/0 conn(0x7ff0b80b4ec0 0x7ff0b80cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0be780640 1 -- 172.21.15.1:0/1886541329 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b80cca50 con 0x7ff0b8160290 2024-05-09T05:08:30.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.012+0000 7ff0bdf7f640 1 --2- 172.21.15.1:0/1886541329 >> v2:172.21.15.196:3300/0 conn(0x7ff0b80b4ec0 0x7ff0b80cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.013+0000 7ff0be780640 1 --2- 172.21.15.1:0/1886541329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b8160290 0x7ff0b81606a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80d15d0 tx=0x7ff0b002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df3b6c356f901d6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.013+0000 7ff0bcf7d640 1 -- 172.21.15.1:0/1886541329 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b0036070 con 0x7ff0b8160290 2024-05-09T05:08:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.013+0000 7ff0bcf7d640 1 -- 172.21.15.1:0/1886541329 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b0031600 con 0x7ff0b8160290 2024-05-09T05:08:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.013+0000 7ff0bcf7d640 1 -- 172.21.15.1:0/1886541329 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b00319d0 con 0x7ff0b8160290 2024-05-09T05:08:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 -- 172.21.15.1:0/1886541329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b8160290 msgr2=0x7ff0b81606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/1886541329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b8160290 0x7ff0b81606a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80d15d0 tx=0x7ff0b002a730 comp rx=0 tx=0).stop 2024-05-09T05:08:30.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 -- 172.21.15.1:0/1886541329 shutdown_connections 2024-05-09T05:08:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/1886541329 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b8160290 0x7ff0b81606a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/1886541329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 0x7ff0b815fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/1886541329 >> v2:172.21.15.196:3300/0 conn(0x7ff0b80b4ec0 0x7ff0b80cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 -- 172.21.15.1:0/1886541329 >> 172.21.15.1:0/1886541329 conn(0x7ff0b80d2360 msgr2=0x7ff0b80ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 -- 172.21.15.1:0/1886541329 shutdown_connections 2024-05-09T05:08:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.014+0000 7ff0c01e3640 1 -- 172.21.15.1:0/1886541329 wait complete. 2024-05-09T05:08:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.016+0000 7ff0c01e3640 1 Processor -- start 2024-05-09T05:08:30.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.016+0000 7ff0c01e3640 1 -- start start 2024-05-09T05:08:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0c01e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b80b4ec0 0x7ff0b8155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0c01e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 0x7ff0b814c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0c01e3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b8160290 0x7ff0b814cb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0c01e3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b80cc750 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0c01e3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b815cd50 con 0x7ff0b8160290 2024-05-09T05:08:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0c01e3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b80cff10 con 0x7ff0b80ce010 2024-05-09T05:08:30.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bdf7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b80b4ec0 0x7ff0b8155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bdf7f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b80b4ec0 0x7ff0b8155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60116/0 (socket says 172.21.15.1:60116) 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bdf7f640 1 -- 172.21.15.1:0/905449196 learned_addr learned my addr 172.21.15.1:0/905449196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bd77e640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 0x7ff0b814c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0be780640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b8160290 0x7ff0b814cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bdf7f640 1 -- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 msgr2=0x7ff0b814c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bdf7f640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 0x7ff0b814c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bdf7f640 1 -- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b8160290 msgr2=0x7ff0b814cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bdf7f640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b8160290 0x7ff0b814cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bdf7f640 1 -- 172.21.15.1:0/905449196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b815ff70 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0bd77e640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b80ce010 0x7ff0b814c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.017+0000 7ff0be780640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b8160290 0x7ff0b814cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.018+0000 7ff0bdf7f640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b80b4ec0 0x7ff0b8155a90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400e910 tx=0x7ff0b400edd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.018+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b4017070 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.018+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0b400ca00 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.018+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b815fc90 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.018+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0b400cdd0 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.018+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b81518e0 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.019+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b806d410 0x7ff0b8154410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.019+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b8050870 con 0x7ff0b806d410 2024-05-09T05:08:30.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.019+0000 7ff0bd77e640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b806d410 0x7ff0b8154410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.019+0000 7ff0bd77e640 1 -- 172.21.15.1:0/905449196 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff0b80cd2b0 con 0x7ff0b806d410 2024-05-09T05:08:30.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.019+0000 7ff0bd77e640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b806d410 0x7ff0b8154410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80b6470 tx=0x7ff0a8011a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:30.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a800f8c0 con 0x7ff0b806d410 2024-05-09T05:08:30.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff0b401a070 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff09c076200 0x7ff09c078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff0b40973a0 con 0x7ff0b80b4ec0 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff0a800fa60 con 0x7ff0b806d410 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b806d410 msgr2=0x7ff0b8154410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b806d410 0x7ff0b8154410 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80b6470 tx=0x7ff0a8011a60 comp rx=0 tx=0).stop 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b806d410 msgr2=0x7ff0b8154410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.020+0000 7ff0ae7fc640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b806d410 0x7ff0b8154410 secure :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80b6470 tx=0x7ff0a8011a60 comp rx=0 tx=0).stop 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.021+0000 7ff0be780640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff09c076200 0x7ff09c078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.021+0000 7ff0be780640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff09c076200 0x7ff09c078710 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0033000 tx=0x7ff0b0009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:30.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.027+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b8154950 0x7ff0b81f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.027+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b80cc000 con 0x7ff0b8154950 2024-05-09T05:08:30.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.027+0000 7ff0bd77e640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b8154950 0x7ff0b81f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.028+0000 7ff0bd77e640 1 -- 172.21.15.1:0/905449196 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff0b8156c80 con 0x7ff0b8154950 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.028+0000 7ff0bd77e640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b8154950 0x7ff0b81f5d70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80b6470 tx=0x7ff0a80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.028+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8004230 con 0x7ff0b8154950 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.028+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7ff0a8026070 con 0x7ff0b8154950 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.028+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b8154950 msgr2=0x7ff0b81f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.028+0000 7ff0ae7fc640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b8154950 0x7ff0b81f5d70 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff0b80b6470 tx=0x7ff0a80046c0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.028+0000 7ff0ae7fc640 1 -- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b8154950 msgr2=0x7ff0b81f5d70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.028+0000 7ff0ae7fc640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b8154950 0x7ff0b81f5d70 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.029 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 24, 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 68025, 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:08:30.030 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:08:30.031 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:08:30.032 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:30.033 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:08:30.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:30.035 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:30.036 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.037 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:08:30.038 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:08:30.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.040 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:08:30.041 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:08:30.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff09c076200 msgr2=0x7ff09c078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff09c076200 0x7ff09c078710 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff0b0033000 tx=0x7ff0b0009000 comp rx=0 tx=0).stop 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b80b4ec0 msgr2=0x7ff0b8155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b80b4ec0 0x7ff0b8155a90 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff0b400e910 tx=0x7ff0b400edd0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0be780640 1 -- 172.21.15.1:0/905449196 reap_dead start 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 shutdown_connections 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 --2- 172.21.15.1:0/905449196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b80b4ec0 0x7ff0b8155a90 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 >> 172.21.15.1:0/905449196 conn(0x7ff0b80d2360 msgr2=0x7ff0b80d0c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 shutdown_connections 2024-05-09T05:08:30.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.029+0000 7ff0c01e3640 1 -- 172.21.15.1:0/905449196 wait complete. 2024-05-09T05:08:30.044 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:08:30.044 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:08:30.044 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:08:30.104 INFO:tasks.workunit.client.0.smithi001.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-05-09T05:08:30.104 INFO:tasks.workunit.client.0.smithi001.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-05-09T05:08:30.105 INFO:tasks.workunit.client.0.smithi001.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-05-09T05:08:30.105 INFO:tasks.workunit.client.0.smithi001.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-05-09T05:08:30.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.166+0000 7f0ccd54d640 1 Processor -- start 2024-05-09T05:08:30.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.166+0000 7f0ccd54d640 1 -- start start 2024-05-09T05:08:30.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.167+0000 7f0ccd54d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0cc80c5ca0 0x7f0cc80ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.167+0000 7f0ccd54d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cc80c86a0 con 0x7f0cc8156a80 2024-05-09T05:08:30.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.167+0000 7f0ccd54d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cc80c8980 con 0x7f0cc8156e90 2024-05-09T05:08:30.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.167+0000 7f0ccd54d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0cc80cb910 con 0x7f0cc80c5ca0 2024-05-09T05:08:30.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.167+0000 7f0cc67fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0cc8156a80 0x7f0cc806df20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41420/0 (socket says 172.21.15.1:41420) 2024-05-09T05:08:30.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.167+0000 7f0cc67fc640 1 -- 172.21.15.1:0/1321237266 learned_addr learned my addr 172.21.15.1:0/1321237266 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:30.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.167+0000 7f0cc6ffd640 1 --2- 172.21.15.1:0/1321237266 >> v2:172.21.15.196:3300/0 conn(0x7f0cc80c5ca0 0x7f0cc80ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.168+0000 7f0cc5ffb640 1 -- 172.21.15.1:0/1321237266 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4231482884 0 0) 0x7f0cc80c86a0 con 0x7f0cc8156a80 2024-05-09T05:08:30.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.168+0000 7f0cc5ffb640 1 -- 172.21.15.1:0/1321237266 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0cb0003520 con 0x7f0cc8156a80 2024-05-09T05:08:30.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.168+0000 7f0cc6ffd640 1 -- 172.21.15.1:0/1321237266 >> v1:172.21.15.196:6789/0 conn(0x7f0cc8156e90 legacy=0x7f0cc80c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.168+0000 7f0cc6ffd640 1 -- 172.21.15.1:0/1321237266 >> v1:172.21.15.1:6789/0 conn(0x7f0cc8156a80 legacy=0x7f0cc806df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.168+0000 7f0cc6ffd640 1 -- 172.21.15.1:0/1321237266 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc80c8c60 con 0x7f0cc80c5ca0 2024-05-09T05:08:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.169+0000 7f0cc6ffd640 1 --2- 172.21.15.1:0/1321237266 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 0x7f0cc80ca4a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0cb80097a0 tx=0x7f0cb802a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a3747c21db404bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.169+0000 7f0cc5ffb640 1 -- 172.21.15.1:0/1321237266 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8037070 con 0x7f0cc80c5ca0 2024-05-09T05:08:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.169+0000 7f0cc5ffb640 1 -- 172.21.15.1:0/1321237266 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cb8032640 con 0x7f0cc80c5ca0 2024-05-09T05:08:30.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.169+0000 7f0cc5ffb640 1 -- 172.21.15.1:0/1321237266 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cb8032a10 con 0x7f0cc80c5ca0 2024-05-09T05:08:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.170+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1321237266 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 msgr2=0x7f0cc80ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.170+0000 7f0ccd54d640 1 --2- 172.21.15.1:0/1321237266 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 0x7f0cc80ca4a0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f0cb80097a0 tx=0x7f0cb802a2c0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.170+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1321237266 shutdown_connections 2024-05-09T05:08:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.170+0000 7f0ccd54d640 1 --2- 172.21.15.1:0/1321237266 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 0x7f0cc80ca4a0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.170+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1321237266 >> 172.21.15.1:0/1321237266 conn(0x7f0cc80745a0 msgr2=0x7f0cc8071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.170+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1321237266 shutdown_connections 2024-05-09T05:08:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.170+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1321237266 wait complete. 2024-05-09T05:08:30.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.172+0000 7f0ccd54d640 1 Processor -- start 2024-05-09T05:08:30.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.172+0000 7f0ccd54d640 1 -- start start 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0ccd54d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 0x7f0cc814fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0ccd54d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cc8156a80 0x7f0cc81554f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0ccd54d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cc8156e90 0x7f0cc8154720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0ccd54d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc80cad60 con 0x7f0cc8156e90 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0ccd54d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0cc80cb5a0 con 0x7f0cc80c5ca0 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0ccd54d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0cc80ca9e0 con 0x7f0cc8156a80 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0cc67fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cc8156a80 0x7f0cc81554f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0cc6ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 0x7f0cc814fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0cc67fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cc8156a80 0x7f0cc81554f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:56614/0 (socket says 172.21.15.1:56614) 2024-05-09T05:08:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0cc6ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 0x7f0cc814fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:53918/0 (socket says 172.21.15.1:53918) 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0cc67fc640 1 -- 172.21.15.1:0/1059869756 learned_addr learned my addr 172.21.15.1:0/1059869756 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.174+0000 7f0cc77fe640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cc8156e90 0x7f0cc8154720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0cc67fc640 1 -- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 msgr2=0x7f0cc814fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0cc67fc640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 0x7f0cc814fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0cc67fc640 1 -- 172.21.15.1:0/1059869756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cc8156e90 msgr2=0x7f0cc8154720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0cc67fc640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cc8156e90 0x7f0cc8154720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0cc67fc640 1 -- 172.21.15.1:0/1059869756 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cc8155a30 con 0x7f0cc8156a80 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0cc67fc640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cc8156a80 0x7f0cc81554f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c390 tx=0x7f0cbc00c850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0caf7fe640 1 -- 172.21.15.1:0/1059869756 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc017070 con 0x7f0cc8156a80 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0caf7fe640 1 -- 172.21.15.1:0/1059869756 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cbc005d00 con 0x7f0cc8156a80 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0caf7fe640 1 -- 172.21.15.1:0/1059869756 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cbc007310 con 0x7f0cc8156a80 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.175+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cc81f2bd0 con 0x7f0cc8156a80 2024-05-09T05:08:30.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.176+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cc81f3110 con 0x7f0cc8156a80 2024-05-09T05:08:30.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.177+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cc8050870 con 0x7f0cc8156a80 2024-05-09T05:08:30.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.178+0000 7f0caf7fe640 1 -- 172.21.15.1:0/1059869756 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0cbc01a070 con 0x7f0cc8156a80 2024-05-09T05:08:30.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.179+0000 7f0caf7fe640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cb40762d0 0x7f0cb40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:30.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.179+0000 7f0caf7fe640 1 -- 172.21.15.1:0/1059869756 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0cbc0969d0 con 0x7f0cc8156a80 2024-05-09T05:08:30.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.180+0000 7f0cc6ffd640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cb40762d0 0x7f0cb40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:30.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.181+0000 7f0cc6ffd640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cb40762d0 0x7f0cb40787e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8034000 tx=0x7f0cb802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:30.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.185+0000 7f0caf7fe640 1 -- 172.21.15.1:0/1059869756 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0cbc05fbc0 con 0x7f0cc8156a80 2024-05-09T05:08:30.272 INFO:tasks.workunit.client.0.smithi001.stdout:/bin/sh ./config.status --recheck 2024-05-09T05:08:30.342 INFO:tasks.workunit.client.0.smithi001.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-05-09T05:08:30.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.497+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f0cc80cb000 con 0x7f0cc8156a80 2024-05-09T05:08:30.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.499+0000 7f0caf7fe640 1 -- 172.21.15.1:0/1059869756 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f0cbc068030 con 0x7f0cc8156a80 2024-05-09T05:08:30.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cb40762d0 msgr2=0x7f0cb40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cb40762d0 0x7f0cb40787e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f0cb8034000 tx=0x7f0cb802e000 comp rx=0 tx=0).stop 2024-05-09T05:08:30.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cc8156a80 msgr2=0x7f0cc81554f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:30.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cc8156a80 0x7f0cc81554f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc00c390 tx=0x7f0cbc00c850 comp rx=0 tx=0).stop 2024-05-09T05:08:30.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 shutdown_connections 2024-05-09T05:08:30.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cb40762d0 0x7f0cb40787e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cc8156e90 0x7f0cc8154720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cc8156a80 0x7f0cc81554f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 --2- 172.21.15.1:0/1059869756 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cc80c5ca0 0x7f0cc814fe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:30.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 >> 172.21.15.1:0/1059869756 conn(0x7f0cc80745a0 msgr2=0x7f0cc80c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:30.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 shutdown_connections 2024-05-09T05:08:30.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:30.501+0000 7f0ccd54d640 1 -- 172.21.15.1:0/1059869756 wait complete. 2024-05-09T05:08:30.505 INFO:tasks.workunit.client.0.smithi001.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-05-09T05:08:30.514 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:08:31.516 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether build environment is sane... yes 2024-05-09T05:08:31.527 INFO:tasks.workunit.client.0.smithi001.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-05-09T05:08:31.528 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gawk... gawk 2024-05-09T05:08:31.553 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether make sets $(MAKE)... yes 2024-05-09T05:08:31.603 INFO:tasks.workunit.client.0.smithi001.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-05-09T05:08:31.603 INFO:tasks.workunit.client.0.smithi001.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-05-09T05:08:31.624 INFO:tasks.workunit.client.0.smithi001.stdout:checking for style of include used by make... GNU 2024-05-09T05:08:31.630 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gcc... gcc 2024-05-09T05:08:31.756 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether the C compiler works... yes 2024-05-09T05:08:31.757 INFO:tasks.workunit.client.0.smithi001.stdout:checking for C compiler default output file name... a.out 2024-05-09T05:08:31.811 INFO:tasks.workunit.client.0.smithi001.stdout:checking for suffix of executables... 2024-05-09T05:08:31.867 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether we are cross compiling... no 2024-05-09T05:08:31.916 INFO:tasks.workunit.client.0.smithi001.stdout:checking for suffix of object files... o 2024-05-09T05:08:31.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:08:31.930 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:08:31.950 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether we are using the GNU C compiler... yes 2024-05-09T05:08:31.984 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether gcc accepts -g... yes 2024-05-09T05:08:32.031 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gcc option to accept ISO C89... none needed 2024-05-09T05:08:32.185 INFO:tasks.workunit.client.0.smithi001.stdout:checking dependency style of gcc... gcc3 2024-05-09T05:08:32.251 INFO:tasks.workunit.client.0.smithi001.stdout:checking how to run the C preprocessor... gcc -E 2024-05-09T05:08:32.317 INFO:tasks.workunit.client.0.smithi001.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-05-09T05:08:32.318 INFO:tasks.workunit.client.0.smithi001.stdout:checking for egrep... /usr/bin/grep -E 2024-05-09T05:08:32.492 INFO:tasks.workunit.client.0.smithi001.stdout:checking for ANSI C header files... yes 2024-05-09T05:08:32.540 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/types.h... yes 2024-05-09T05:08:32.595 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/stat.h... yes 2024-05-09T05:08:32.651 INFO:tasks.workunit.client.0.smithi001.stdout:checking for stdlib.h... yes 2024-05-09T05:08:32.709 INFO:tasks.workunit.client.0.smithi001.stdout:checking for string.h... yes 2024-05-09T05:08:32.766 INFO:tasks.workunit.client.0.smithi001.stdout:checking for memory.h... yes 2024-05-09T05:08:32.821 INFO:tasks.workunit.client.0.smithi001.stdout:checking for strings.h... yes 2024-05-09T05:08:32.876 INFO:tasks.workunit.client.0.smithi001.stdout:checking for inttypes.h... yes 2024-05-09T05:08:32.933 INFO:tasks.workunit.client.0.smithi001.stdout:checking for stdint.h... yes 2024-05-09T05:08:32.991 INFO:tasks.workunit.client.0.smithi001.stdout:checking for unistd.h... yes 2024-05-09T05:08:33.052 INFO:tasks.workunit.client.0.smithi001.stdout:checking minix/config.h usability... no 2024-05-09T05:08:33.092 INFO:tasks.workunit.client.0.smithi001.stdout:checking minix/config.h presence... no 2024-05-09T05:08:33.092 INFO:tasks.workunit.client.0.smithi001.stdout:checking for minix/config.h... no 2024-05-09T05:08:33.136 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-05-09T05:08:33.147 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gcc... (cached) gcc 2024-05-09T05:08:33.199 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-05-09T05:08:33.199 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether gcc accepts -g... (cached) yes 2024-05-09T05:08:33.200 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-05-09T05:08:33.200 INFO:tasks.workunit.client.0.smithi001.stdout:checking dependency style of gcc... (cached) gcc3 2024-05-09T05:08:33.264 INFO:tasks.workunit.client.0.smithi001.stdout:checking for main in -lm... yes 2024-05-09T05:08:33.330 INFO:tasks.workunit.client.0.smithi001.stdout:checking for main in -lpthread... yes 2024-05-09T05:08:33.335 INFO:tasks.workunit.client.0.smithi001.stdout:checking for ANSI C header files... (cached) yes 2024-05-09T05:08:33.387 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-05-09T05:08:33.444 INFO:tasks.workunit.client.0.smithi001.stdout:checking pthread.h usability... yes 2024-05-09T05:08:33.470 INFO:tasks.workunit.client.0.smithi001.stdout:checking pthread.h presence... yes 2024-05-09T05:08:33.471 INFO:tasks.workunit.client.0.smithi001.stdout:checking for pthread.h... yes 2024-05-09T05:08:33.534 INFO:tasks.workunit.client.0.smithi001.stdout:checking fcntl.h usability... yes 2024-05-09T05:08:33.558 INFO:tasks.workunit.client.0.smithi001.stdout:checking fcntl.h presence... yes 2024-05-09T05:08:33.559 INFO:tasks.workunit.client.0.smithi001.stdout:checking for fcntl.h... yes 2024-05-09T05:08:33.624 INFO:tasks.workunit.client.0.smithi001.stdout:checking limits.h usability... yes 2024-05-09T05:08:33.646 INFO:tasks.workunit.client.0.smithi001.stdout:checking limits.h presence... yes 2024-05-09T05:08:33.647 INFO:tasks.workunit.client.0.smithi001.stdout:checking for limits.h... yes 2024-05-09T05:08:33.668 INFO:tasks.workunit.client.0.smithi001.stdout:checking for stdint.h... (cached) yes 2024-05-09T05:08:33.723 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/time.h usability... yes 2024-05-09T05:08:33.745 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/time.h presence... yes 2024-05-09T05:08:33.746 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/time.h... yes 2024-05-09T05:08:33.766 INFO:tasks.workunit.client.0.smithi001.stdout:checking for unistd.h... (cached) yes 2024-05-09T05:08:33.820 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/vfs.h usability... yes 2024-05-09T05:08:33.843 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/vfs.h presence... yes 2024-05-09T05:08:33.843 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/vfs.h... yes 2024-05-09T05:08:33.913 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/limits.h usability... no 2024-05-09T05:08:33.952 INFO:tasks.workunit.client.0.smithi001.stdout:checking sys/limits.h presence... no 2024-05-09T05:08:33.952 INFO:tasks.workunit.client.0.smithi001.stdout:checking for sys/limits.h... no 2024-05-09T05:08:33.988 INFO:tasks.workunit.client.0.smithi001.stdout:checking for an ANSI C-conforming const... yes 2024-05-09T05:08:34.028 INFO:tasks.workunit.client.0.smithi001.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-05-09T05:08:34.114 INFO:tasks.workunit.client.0.smithi001.stdout:checking for system... yes 2024-05-09T05:08:34.205 INFO:tasks.workunit.client.0.smithi001.stdout:checking for gettimeofday... yes 2024-05-09T05:08:34.293 INFO:tasks.workunit.client.0.smithi001.stdout:checking for mkdir... yes 2024-05-09T05:08:34.376 INFO:tasks.workunit.client.0.smithi001.stdout:checking for strerror... yes 2024-05-09T05:08:34.459 INFO:tasks.workunit.client.0.smithi001.stdout:checking for open64... yes 2024-05-09T05:08:34.542 INFO:tasks.workunit.client.0.smithi001.stdout:checking for stat64... yes 2024-05-09T05:08:34.625 INFO:tasks.workunit.client.0.smithi001.stdout:checking for fseeko64... yes 2024-05-09T05:08:34.706 INFO:tasks.workunit.client.0.smithi001.stdout:checking for lrand48_r... yes 2024-05-09T05:08:34.789 INFO:tasks.workunit.client.0.smithi001.stdout:checking for srand48_r... yes 2024-05-09T05:08:34.817 INFO:tasks.workunit.client.0.smithi001.stdout:configure: creating ./config.status 2024-05-09T05:08:34.929 INFO:tasks.workunit.client.0.smithi001.stdout: /bin/sh ./config.status 2024-05-09T05:08:34.995 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: creating Makefile 2024-05-09T05:08:35.013 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: creating config.h 2024-05-09T05:08:35.042 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: executing depfiles commands 2024-05-09T05:08:35.062 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: executing default commands 2024-05-09T05:08:35.119 INFO:tasks.workunit.client.0.smithi001.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-05-09T05:08:35.308 INFO:tasks.workunit.client.0.smithi001.stdout:rm -f stamp-h1 2024-05-09T05:08:35.311 INFO:tasks.workunit.client.0.smithi001.stdout:touch config.h.in 2024-05-09T05:08:35.313 INFO:tasks.workunit.client.0.smithi001.stdout:cd . && /bin/sh ./config.status config.h 2024-05-09T05:08:35.369 INFO:tasks.workunit.client.0.smithi001.stdout:config.status: creating config.h 2024-05-09T05:08:35.407 INFO:tasks.workunit.client.0.smithi001.stdout:make all-am 2024-05-09T05:08:35.434 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-05-09T05:08:35.434 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-05-09T05:08:35.612 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-05-09T05:08:35.617 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-05-09T05:08:35.725 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-05-09T05:08:35.730 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-05-09T05:08:35.839 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-05-09T05:08:35.844 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-05-09T05:08:35.960 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-05-09T05:08:35.965 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-05-09T05:08:36.109 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-05-09T05:08:36.114 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-05-09T05:08:36.193 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-05-09T05:08:36.200 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-05-09T05:08:36.255 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-05-09T05:08:36.262 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-05-09T05:08:36.328 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-05-09T05:08:36.335 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-05-09T05:08:36.561 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-05-09T05:08:36.567 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-05-09T05:08:36.722 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-05-09T05:08:36.727 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-05-09T05:08:36.899 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-05-09T05:08:36.904 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-05-09T05:08:36.995 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-05-09T05:08:37.002 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-05-09T05:08:37.098 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-05-09T05:08:37.103 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-05-09T05:08:37.239 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-05-09T05:08:37.245 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-05-09T05:08:37.291 INFO:tasks.workunit.client.0.smithi001.stderr:In file included from parser.h:22, 2024-05-09T05:08:37.291 INFO:tasks.workunit.client.0.smithi001.stderr: from parser.c:25: 2024-05-09T05:08:37.291 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c: In function ‘init_filesys’: 2024-05-09T05:08:37.293 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T05:08:37.293 INFO:tasks.workunit.client.0.smithi001.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T05:08:37.293 INFO:tasks.workunit.client.0.smithi001.stderr: | ^ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: 36 | container_of(ptr, type, member) 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T05:08:37.294 INFO:tasks.workunit.client.0.smithi001.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: | ^ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: 36 | container_of(ptr, type, member) 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T05:08:37.295 INFO:tasks.workunit.client.0.smithi001.stderr: | ^ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: 36 | container_of(ptr, type, member) 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T05:08:37.296 INFO:tasks.workunit.client.0.smithi001.stderr: | ^ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: 36 | container_of(ptr, type, member) 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c: In function ‘init_tg_stats’: 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T05:08:37.297 INFO:tasks.workunit.client.0.smithi001.stderr: | ^ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr: 36 | container_of(ptr, type, member) 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr: 971 | list_for_each_entry(tmp_list, 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.298 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: | ^ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: 36 | container_of(ptr, type, member) 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: 971 | list_for_each_entry(tmp_list, 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.299 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: | ^ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: 36 | container_of(ptr, type, member) 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: 980 | list_for_each_entry(tmp_list, 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.300 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: | ^ 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~ 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: 36 | container_of(ptr, type, member) 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~ 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~ 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: 980 | list_for_each_entry(tmp_list, 2024-05-09T05:08:37.301 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.341 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c: In function ‘get_optstr’: 2024-05-09T05:08:37.341 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-05-09T05:08:37.341 INFO:tasks.workunit.client.0.smithi001.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-05-09T05:08:37.342 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.342 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:200:14: note: source object declared here 2024-05-09T05:08:37.342 INFO:tasks.workunit.client.0.smithi001.stderr: 200 | char temp[BUFSIZE]; 2024-05-09T05:08:37.342 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~ 2024-05-09T05:08:37.348 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c: In function ‘size64_convert’: 2024-05-09T05:08:37.348 INFO:tasks.workunit.client.0.smithi001.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-05-09T05:08:37.348 INFO:tasks.workunit.client.0.smithi001.stderr: 119 | memcpy(unit, "\0", 3); 2024-05-09T05:08:37.348 INFO:tasks.workunit.client.0.smithi001.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-05-09T05:08:37.656 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-05-09T05:08:37.663 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-05-09T05:08:37.744 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-05-09T05:08:37.750 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-05-09T05:08:37.861 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-05-09T05:08:37.868 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-05-09T05:08:37.909 INFO:tasks.workunit.client.0.smithi001.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-05-09T05:08:37.917 INFO:tasks.workunit.client.0.smithi001.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-05-09T05:08:38.027 INFO:tasks.workunit.client.0.smithi001.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-05-09T05:08:38.028 INFO:tasks.workunit.client.0.smithi001.stderr:+ cd .. 2024-05-09T05:08:38.029 INFO:tasks.workunit.client.0.smithi001.stderr:+ mkdir tmp 2024-05-09T05:08:38.033 INFO:tasks.workunit.client.0.smithi001.stderr:+ cd tmp 2024-05-09T05:08:38.034 INFO:tasks.workunit.client.0.smithi001.stderr:+ for f in $mydir/*.ffsb 2024-05-09T05:08:38.034 INFO:tasks.workunit.client.0.smithi001.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-05-09T05:08:38.035 INFO:tasks.workunit.client.0.smithi001.stdout:FFSB version 6.0-RC2 started 2024-05-09T05:08:38.035 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout:benchmark time = 300 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout:ThreadGroup 0 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout:================ 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout: num_threads = 32 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout: read_random = off 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout: read_size = 0 (0B) 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout: read_blocksize = 0 (0B) 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout: read_skip = off 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout: read_skipsize = 0 (0B) 2024-05-09T05:08:38.038 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: write_random = on 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: write_size = 5242880 (5MB) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: fsync_file = 0 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: write_blocksize = 4096 (4KB) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: wait time = 0 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: op weights 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: read = 0 (0.00%) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: readall = 0 (0.00%) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: write = 1 (100.00%) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: create = 0 (0.00%) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: append = 0 (0.00%) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: delete = 0 (0.00%) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: metaop = 0 (0.00%) 2024-05-09T05:08:38.039 INFO:tasks.workunit.client.0.smithi001.stdout: createdir = 0 (0.00%) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: stat = 0 (0.00%) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: writeall = 0 (0.00%) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: writeall_fsync = 0 (0.00%) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: open_close = 0 (0.00%) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: write_fsync = 0 (0.00%) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: create_fsync = 0 (0.00%) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: append_fsync = 0 (0.00%) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout:FileSystem . 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout:========== 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: num_dirs = 0 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: starting files = 128 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: min file size = 104857600 (100MB) 2024-05-09T05:08:38.040 INFO:tasks.workunit.client.0.smithi001.stdout: max file size = 104857600 (100MB) 2024-05-09T05:08:38.041 INFO:tasks.workunit.client.0.smithi001.stdout: directio = off 2024-05-09T05:08:38.041 INFO:tasks.workunit.client.0.smithi001.stdout: alignedio = on 2024-05-09T05:08:38.041 INFO:tasks.workunit.client.0.smithi001.stdout: bufferedio = off 2024-05-09T05:08:38.041 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:08:38.041 INFO:tasks.workunit.client.0.smithi001.stdout: aging is off 2024-05-09T05:08:38.041 INFO:tasks.workunit.client.0.smithi001.stdout: current utilization = 0.00% 2024-05-09T05:08:38.041 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:08:38.042 INFO:tasks.workunit.client.0.smithi001.stdout:checking existing fs: . 2024-05-09T05:08:38.042 INFO:tasks.workunit.client.0.smithi001.stdout:recreating new fileset 2024-05-09T05:08:38.042 INFO:tasks.workunit.client.0.smithi001.stderr:opendir: No such file or directory 2024-05-09T05:08:40.515 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:08:40.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:40.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.649+0000 7f816d914640 1 Processor -- start 2024-05-09T05:08:40.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.649+0000 7f816d914640 1 -- start start 2024-05-09T05:08:40.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f816d914640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8168156c40 0x7f81680ce810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:40.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f816d914640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f81680ced50 0x7f81680c7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:40.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f816d914640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f81680ca770 con 0x7f816806d8f0 2024-05-09T05:08:40.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f816d914640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f81680ca540 con 0x7f81680ced50 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f816d914640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f81680caa30 con 0x7f8168156c40 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f81677fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f81680ced50 0x7f81680c7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f81667fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8168156c40 0x7f81680ce810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f81667fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8168156c40 0x7f81680ce810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:53300/0 (socket says 172.21.15.1:53300) 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f81677fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f81680ced50 0x7f81680c7ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:42042/0 (socket says 172.21.15.1:42042) 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.650+0000 7f81667fc640 1 -- 172.21.15.1:0/3775992698 learned_addr learned my addr 172.21.15.1:0/3775992698 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.651+0000 7f81677fe640 1 -- 172.21.15.1:0/3775992698 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8168156c40 msgr2=0x7f81680ce810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.651+0000 7f81677fe640 1 --2- 172.21.15.1:0/3775992698 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8168156c40 0x7f81680ce810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.651+0000 7f81677fe640 1 -- 172.21.15.1:0/3775992698 >> v1:172.21.15.196:6789/0 conn(0x7f816806d8f0 legacy=0x7f81680c5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.651+0000 7f81677fe640 1 -- 172.21.15.1:0/3775992698 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f816806de40 con 0x7f81680ced50 2024-05-09T05:08:40.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.651+0000 7f81667fc640 1 --2- 172.21.15.1:0/3775992698 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8168156c40 0x7f81680ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.652+0000 7f81677fe640 1 --2- 172.21.15.1:0/3775992698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81680ced50 0x7f81680c7ad0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8160009d90 tx=0x7f816002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38d168c59526172b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.652+0000 7f8165ffb640 1 -- 172.21.15.1:0/3775992698 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160007740 con 0x7f81680ced50 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.652+0000 7f8165ffb640 1 -- 172.21.15.1:0/3775992698 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f81600078e0 con 0x7f81680ced50 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.652+0000 7f8165ffb640 1 -- 172.21.15.1:0/3775992698 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160007cb0 con 0x7f81680ced50 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.653+0000 7f816d914640 1 -- 172.21.15.1:0/3775992698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81680ced50 msgr2=0x7f81680c7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.653+0000 7f816d914640 1 --2- 172.21.15.1:0/3775992698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81680ced50 0x7f81680c7ad0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f8160009d90 tx=0x7f816002a660 comp rx=0 tx=0).stop 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.653+0000 7f816d914640 1 -- 172.21.15.1:0/3775992698 shutdown_connections 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.653+0000 7f816d914640 1 --2- 172.21.15.1:0/3775992698 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f81680ced50 0x7f81680c7ad0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.653+0000 7f816d914640 1 --2- 172.21.15.1:0/3775992698 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8168156c40 0x7f81680ce810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.653+0000 7f816d914640 1 -- 172.21.15.1:0/3775992698 >> 172.21.15.1:0/3775992698 conn(0x7f81680745b0 msgr2=0x7f81680719a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.653+0000 7f816d914640 1 -- 172.21.15.1:0/3775992698 shutdown_connections 2024-05-09T05:08:40.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.653+0000 7f816d914640 1 -- 172.21.15.1:0/3775992698 wait complete. 2024-05-09T05:08:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.654+0000 7f816d914640 1 Processor -- start 2024-05-09T05:08:40.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.655+0000 7f816d914640 1 -- start start 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f816d914640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816806d8f0 0x7f81681529c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f816d914640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81680ced50 0x7f816815ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f816d914640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168156c40 0x7f816814b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f816d914640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f816806e6e0 con 0x7f8168156c40 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f816d914640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f816806e400 con 0x7f816806d8f0 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f816d914640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8168072240 con 0x7f81680ced50 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f81677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168156c40 0x7f816814b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f81677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168156c40 0x7f816814b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53206/0 (socket says 172.21.15.1:53206) 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f81677fe640 1 -- 172.21.15.1:0/1861276171 learned_addr learned my addr 172.21.15.1:0/1861276171 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f81667fc640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81680ced50 0x7f816815ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.656+0000 7f8166ffd640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816806d8f0 0x7f81681529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:40.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f81677fe640 1 -- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81680ced50 msgr2=0x7f816815ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f81677fe640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81680ced50 0x7f816815ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f81677fe640 1 -- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816806d8f0 msgr2=0x7f81681529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f81677fe640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816806d8f0 0x7f81681529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f81677fe640 1 -- 172.21.15.1:0/1861276171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81680ce160 con 0x7f8168156c40 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f81677fe640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168156c40 0x7f816814b330 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f816002ac50 tx=0x7f8160004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f814ffff640 1 -- 172.21.15.1:0/1861276171 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f816003f070 con 0x7f8168156c40 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f814ffff640 1 -- 172.21.15.1:0/1861276171 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8160031bc0 con 0x7f8168156c40 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81680cdd70 con 0x7f8168156c40 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f814ffff640 1 -- 172.21.15.1:0/1861276171 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8160009450 con 0x7f8168156c40 2024-05-09T05:08:40.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.657+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81680ce420 con 0x7f8168156c40 2024-05-09T05:08:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.658+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8168050870 con 0x7f8168156c40 2024-05-09T05:08:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.659+0000 7f814ffff640 1 -- 172.21.15.1:0/1861276171 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8160044460 con 0x7f8168156c40 2024-05-09T05:08:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.660+0000 7f814ffff640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81540762d0 0x7f81540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.660+0000 7f814ffff640 1 -- 172.21.15.1:0/1861276171 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f81600b8d60 con 0x7f8168156c40 2024-05-09T05:08:40.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.660+0000 7f8166ffd640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81540762d0 0x7f81540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:40.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.661+0000 7f8166ffd640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81540762d0 0x7f81540787e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8158001720 tx=0x7f8158009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:40.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.666+0000 7f814ffff640 1 -- 172.21.15.1:0/1861276171 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8160081f60 con 0x7f8168156c40 2024-05-09T05:08:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.959+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f816806cef0 con 0x7f8168156c40 2024-05-09T05:08:40.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.960+0000 7f814ffff640 1 -- 172.21.15.1:0/1861276171 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8160086260 con 0x7f8168156c40 2024-05-09T05:08:40.960 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81540762d0 msgr2=0x7f81540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81540762d0 0x7f81540787e0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f8158001720 tx=0x7f8158009040 comp rx=0 tx=0).stop 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168156c40 msgr2=0x7f816814b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168156c40 0x7f816814b330 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f816002ac50 tx=0x7f8160004a40 comp rx=0 tx=0).stop 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 shutdown_connections 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f81540762d0 0x7f81540787e0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8168156c40 0x7f816814b330 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f81680ced50 0x7f816815ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 --2- 172.21.15.1:0/1861276171 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f816806d8f0 0x7f81681529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:40.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 >> 172.21.15.1:0/1861276171 conn(0x7f81680745b0 msgr2=0x7f816806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:40.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 shutdown_connections 2024-05-09T05:08:40.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:40.962+0000 7f816d914640 1 -- 172.21.15.1:0/1861276171 wait complete. 2024-05-09T05:08:40.974 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:08:40.974 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:08:40.974 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:08:40.975 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:08:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.097+0000 7f3e83f0d640 1 Processor -- start 2024-05-09T05:08:41.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.097+0000 7f3e83f0d640 1 -- start start 2024-05-09T05:08:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.098+0000 7f3e83f0d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3e7c0ce010 0x7f3e7c0c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.098+0000 7f3e83f0d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e7c0c77c0 0x7f3e7c0c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.098+0000 7f3e83f0d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7c15c7b0 con 0x7f3e7c0b4ec0 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.098+0000 7f3e83f0d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7c15d0c0 con 0x7f3e7c0c77c0 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.098+0000 7f3e83f0d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f3e7c0cca50 con 0x7f3e7c0ce010 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.098+0000 7f3e824aa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3e7c0c77c0 0x7f3e7c0c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.098+0000 7f3e81ca9640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3e7c0b4ec0 0x7f3e7c0cf9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44226/0 (socket says 172.21.15.1:44226) 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.098+0000 7f3e81ca9640 1 -- 172.21.15.1:0/2762680633 learned_addr learned my addr 172.21.15.1:0/2762680633 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.099+0000 7f3e814a8640 1 --2- 172.21.15.1:0/2762680633 >> v2:172.21.15.196:3300/0 conn(0x7f3e7c0ce010 0x7f3e7c0c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.099+0000 7f3e824aa640 1 -- 172.21.15.1:0/2762680633 >> v2:172.21.15.196:3300/0 conn(0x7f3e7c0ce010 msgr2=0x7f3e7c0c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.099+0000 7f3e824aa640 1 --2- 172.21.15.1:0/2762680633 >> v2:172.21.15.196:3300/0 conn(0x7f3e7c0ce010 0x7f3e7c0c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.099+0000 7f3e824aa640 1 -- 172.21.15.1:0/2762680633 >> v1:172.21.15.1:6789/0 conn(0x7f3e7c0b4ec0 legacy=0x7f3e7c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.099+0000 7f3e80ca7640 1 -- 172.21.15.1:0/2762680633 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 272547056 0 0) 0x7f3e7c15c7b0 con 0x7f3e7c0b4ec0 2024-05-09T05:08:41.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.099+0000 7f3e824aa640 1 -- 172.21.15.1:0/2762680633 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c0ccd30 con 0x7f3e7c0c77c0 2024-05-09T05:08:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.100+0000 7f3e824aa640 1 --2- 172.21.15.1:0/2762680633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0c77c0 0x7f3e7c0c7bd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0c8b40 tx=0x7f3e7402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc40c1170d8a6a88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.100+0000 7f3e80ca7640 1 -- 172.21.15.1:0/2762680633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74007740 con 0x7f3e7c0c77c0 2024-05-09T05:08:41.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.100+0000 7f3e80ca7640 1 -- 172.21.15.1:0/2762680633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e740078e0 con 0x7f3e7c0c77c0 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.100+0000 7f3e80ca7640 1 -- 172.21.15.1:0/2762680633 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74007cb0 con 0x7f3e7c0c77c0 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.101+0000 7f3e83f0d640 1 -- 172.21.15.1:0/2762680633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0c77c0 msgr2=0x7f3e7c0c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.101+0000 7f3e83f0d640 1 --2- 172.21.15.1:0/2762680633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0c77c0 0x7f3e7c0c7bd0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3e7c0c8b40 tx=0x7f3e7402a710 comp rx=0 tx=0).stop 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.101+0000 7f3e83f0d640 1 -- 172.21.15.1:0/2762680633 shutdown_connections 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.101+0000 7f3e83f0d640 1 --2- 172.21.15.1:0/2762680633 >> v2:172.21.15.196:3300/0 conn(0x7f3e7c0ce010 0x7f3e7c0c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.101+0000 7f3e83f0d640 1 --2- 172.21.15.1:0/2762680633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0c77c0 0x7f3e7c0c7bd0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.101+0000 7f3e83f0d640 1 -- 172.21.15.1:0/2762680633 >> 172.21.15.1:0/2762680633 conn(0x7f3e7c0d2360 msgr2=0x7f3e7c0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.101+0000 7f3e83f0d640 1 -- 172.21.15.1:0/2762680633 shutdown_connections 2024-05-09T05:08:41.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.101+0000 7f3e83f0d640 1 -- 172.21.15.1:0/2762680633 wait complete. 2024-05-09T05:08:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.103+0000 7f3e83f0d640 1 Processor -- start 2024-05-09T05:08:41.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.104+0000 7f3e83f0d640 1 -- start start 2024-05-09T05:08:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e83f0d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e7c0b4ec0 0x7f3e7c15b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e83f0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e7c0c77c0 0x7f3e7c15f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e83f0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0ce010 0x7f3e7c15b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e83f0d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c0cc750 con 0x7f3e7c0ce010 2024-05-09T05:08:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e83f0d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c15cd50 con 0x7f3e7c0b4ec0 2024-05-09T05:08:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e83f0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7c0c8110 con 0x7f3e7c0c77c0 2024-05-09T05:08:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e824aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0ce010 0x7f3e7c15b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e824aa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0ce010 0x7f3e7c15b6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53212/0 (socket says 172.21.15.1:53212) 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.105+0000 7f3e824aa640 1 -- 172.21.15.1:0/662998429 learned_addr learned my addr 172.21.15.1:0/662998429 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e81ca9640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e7c0b4ec0 0x7f3e7c15b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e824aa640 1 -- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e7c0c77c0 msgr2=0x7f3e7c15f340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e824aa640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e7c0c77c0 0x7f3e7c15f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e824aa640 1 -- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e7c0b4ec0 msgr2=0x7f3e7c15b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e824aa640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e7c0b4ec0 0x7f3e7c15b0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e824aa640 1 -- 172.21.15.1:0/662998429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7c160530 con 0x7f3e7c0ce010 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e824aa640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0ce010 0x7f3e7c15b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009e50 tx=0x7f3e74004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74040030 con 0x7f3e7c0ce010 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7c160ad0 con 0x7f3e7c0ce010 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e740319f0 con 0x7f3e7c0ce010 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e74031da0 con 0x7f3e7c0ce010 2024-05-09T05:08:41.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.106+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7c15a8f0 con 0x7f3e7c0ce010 2024-05-09T05:08:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.108+0000 7f3e83f0d640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c14acc0 0x7f3e7c15bc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.108+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c15aac0 con 0x7f3e7c14acc0 2024-05-09T05:08:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.108+0000 7f3e81ca9640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c14acc0 0x7f3e7c15bc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.108+0000 7f3e81ca9640 1 -- 172.21.15.1:0/662998429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e7c0b7e10 con 0x7f3e7c14acc0 2024-05-09T05:08:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.108+0000 7f3e81ca9640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c14acc0 0x7f3e7c15bc00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e78001380 tx=0x7f3e78001840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.109+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e78019070 con 0x7f3e7c14acc0 2024-05-09T05:08:41.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.109+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3e78010510 con 0x7f3e7c14acc0 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.109+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c14acc0 msgr2=0x7f3e7c15bc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.109+0000 7f3e727fc640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c14acc0 0x7f3e7c15bc00 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f3e78001380 tx=0x7f3e78001840 comp rx=0 tx=0).stop 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.109+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c14acc0 msgr2=0x7f3e7c15bc00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.109+0000 7f3e727fc640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c14acc0 0x7f3e7c15bc00 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.109+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3e74043030 con 0x7f3e7c0ce010 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.110+0000 7f3e727fc640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e60075d40 0x7f3e60078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.110+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3e740b8700 con 0x7f3e7c0ce010 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.110+0000 7f3e81ca9640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e60075d40 0x7f3e60078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.111+0000 7f3e81ca9640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e60075d40 0x7f3e60078250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3e78001d90 tx=0x7f3e78009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.118+0000 7f3e83f0d640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15e9f0 0x7f3e7c15c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.118+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7c14b100 con 0x7f3e7c15e9f0 2024-05-09T05:08:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.118+0000 7f3e814a8640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15e9f0 0x7f3e7c15c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.119+0000 7f3e814a8640 1 -- 172.21.15.1:0/662998429 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3e7c154e30 con 0x7f3e7c15e9f0 2024-05-09T05:08:41.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.119+0000 7f3e814a8640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15e9f0 0x7f3e7c15c140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00d400 tx=0x7f3e6c00d8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.119+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6c004020 con 0x7f3e7c15e9f0 2024-05-09T05:08:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.119+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f3e6c004020 con 0x7f3e7c15e9f0 2024-05-09T05:08:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.119+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15e9f0 msgr2=0x7f3e7c15c140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.119+0000 7f3e727fc640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15e9f0 0x7f3e7c15c140 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00d400 tx=0x7f3e6c00d8c0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e727fc640 1 -- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15e9f0 msgr2=0x7f3e7c15c140 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e727fc640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c15e9f0 0x7f3e7c15c140 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f3e6c00d400 tx=0x7f3e6c00d8c0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 24, 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 27197, 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:08:41.121 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 14, 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.122 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.123 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:41.124 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.125 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.126 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:08:41.127 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:41.129 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:41.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:08:41.132 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:08:41.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e60075d40 msgr2=0x7f3e60078250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e60075d40 0x7f3e60078250 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f3e78001d90 tx=0x7f3e78009030 comp rx=0 tx=0).stop 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e824aa640 1 -- 172.21.15.1:0/662998429 reap_dead start 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0ce010 msgr2=0x7f3e7c15b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0ce010 0x7f3e7c15b6c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f3e74009e50 tx=0x7f3e74004a40 comp rx=0 tx=0).stop 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 shutdown_connections 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 --2- 172.21.15.1:0/662998429 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e7c0ce010 0x7f3e7c15b6c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 >> 172.21.15.1:0/662998429 conn(0x7f3e7c0d2360 msgr2=0x7f3e7c0cf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 shutdown_connections 2024-05-09T05:08:41.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.120+0000 7f3e83f0d640 1 -- 172.21.15.1:0/662998429 wait complete. 2024-05-09T05:08:41.134 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:08:41.135 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:08:41.135 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:08:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.252+0000 7f8814489640 1 Processor -- start 2024-05-09T05:08:41.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.252+0000 7f8814489640 1 -- start start 2024-05-09T05:08:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8814489640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f880c0cbd10 0x7f880c15afd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8814489640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c154980 0x7f880c0c5000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8814489640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f880c0c4b00 con 0x7f880c0cc830 2024-05-09T05:08:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8814489640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f880c15e510 con 0x7f880c0cbd10 2024-05-09T05:08:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8814489640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f880c0cafd0 con 0x7f880c154980 2024-05-09T05:08:41.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8811a24640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c154980 0x7f880c0c5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8811a24640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c154980 0x7f880c0c5000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:53332/0 (socket says 172.21.15.1:53332) 2024-05-09T05:08:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8811a24640 1 -- 172.21.15.1:0/4034405506 learned_addr learned my addr 172.21.15.1:0/4034405506 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.254+0000 7f8812225640 1 --2- 172.21.15.1:0/4034405506 >> v2:172.21.15.196:3300/0 conn(0x7f880c0cbd10 0x7f880c15afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.255+0000 7f8811a24640 1 -- 172.21.15.1:0/4034405506 >> v2:172.21.15.196:3300/0 conn(0x7f880c0cbd10 msgr2=0x7f880c15afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.255+0000 7f8811a24640 1 --2- 172.21.15.1:0/4034405506 >> v2:172.21.15.196:3300/0 conn(0x7f880c0cbd10 0x7f880c15afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.255+0000 7f8811a24640 1 -- 172.21.15.1:0/4034405506 >> v1:172.21.15.196:6789/0 conn(0x7f880c0cc830 legacy=0x7f880c070ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.255+0000 7f8811a24640 1 -- 172.21.15.1:0/4034405506 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c0cb570 con 0x7f880c154980 2024-05-09T05:08:41.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.255+0000 7f8812225640 1 --2- 172.21.15.1:0/4034405506 >> v2:172.21.15.196:3300/0 conn(0x7f880c0cbd10 0x7f880c15afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.256+0000 7f8811a24640 1 --2- 172.21.15.1:0/4034405506 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c154980 0x7f880c0c5000 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f880c15e690 tx=0x7f87fc02a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=af7fe0dd2b0ad817 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.256+0000 7f8811223640 1 -- 172.21.15.1:0/4034405506 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc037070 con 0x7f880c154980 2024-05-09T05:08:41.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.256+0000 7f8811223640 1 -- 172.21.15.1:0/4034405506 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f87fc032440 con 0x7f880c154980 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.256+0000 7f8811223640 1 -- 172.21.15.1:0/4034405506 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f87fc032810 con 0x7f880c154980 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.257+0000 7f8814489640 1 -- 172.21.15.1:0/4034405506 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c154980 msgr2=0x7f880c0c5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.257+0000 7f8814489640 1 --2- 172.21.15.1:0/4034405506 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c154980 0x7f880c0c5000 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f880c15e690 tx=0x7f87fc02a1c0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.257+0000 7f8814489640 1 -- 172.21.15.1:0/4034405506 shutdown_connections 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.257+0000 7f8814489640 1 --2- 172.21.15.1:0/4034405506 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c154980 0x7f880c0c5000 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.257+0000 7f8814489640 1 --2- 172.21.15.1:0/4034405506 >> v2:172.21.15.196:3300/0 conn(0x7f880c0cbd10 0x7f880c15afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.257+0000 7f8814489640 1 -- 172.21.15.1:0/4034405506 >> 172.21.15.1:0/4034405506 conn(0x7f880c072e50 msgr2=0x7f880c073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.257+0000 7f8814489640 1 -- 172.21.15.1:0/4034405506 shutdown_connections 2024-05-09T05:08:41.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.257+0000 7f8814489640 1 -- 172.21.15.1:0/4034405506 wait complete. 2024-05-09T05:08:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.259+0000 7f8814489640 1 Processor -- start 2024-05-09T05:08:41.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.259+0000 7f8814489640 1 -- start start 2024-05-09T05:08:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8814489640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c0cbd10 0x7f880c1e6dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8814489640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c0cc830 0x7f880c0cf8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8814489640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f880c154980 0x7f880c0c4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8814489640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f880c15dec0 con 0x7f880c154980 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8814489640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f880c15dbe0 con 0x7f880c0cbd10 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8814489640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f880c15e1a0 con 0x7f880c0cc830 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f880c154980 0x7f880c0c4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f880c154980 0x7f880c0c4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53250/0 (socket says 172.21.15.1:53250) 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 -- 172.21.15.1:0/390205262 learned_addr learned my addr 172.21.15.1:0/390205262 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8811a24640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c0cc830 0x7f880c0cf8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812225640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c0cbd10 0x7f880c1e6dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 -- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c0cc830 msgr2=0x7f880c0cf8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c0cc830 0x7f880c0cf8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 -- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c0cbd10 msgr2=0x7f880c1e6dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c0cbd10 0x7f880c1e6dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 -- 172.21.15.1:0/390205262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f880c1f0630 con 0x7f880c154980 2024-05-09T05:08:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8811a24640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f880c0cc830 0x7f880c0cf8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.261+0000 7f8812a26640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f880c154980 0x7f880c0c4d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f880400c730 tx=0x7f880400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.262+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804003b90 con 0x7f880c154980 2024-05-09T05:08:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.262+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8804003d50 con 0x7f880c154980 2024-05-09T05:08:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.262+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8804005310 con 0x7f880c154980 2024-05-09T05:08:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.262+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f880c1e7750 con 0x7f880c154980 2024-05-09T05:08:41.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.262+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f880c1f0b90 con 0x7f880c154980 2024-05-09T05:08:41.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.263+0000 7f8814489640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c14b0a0 0x7f880c0c52c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.263+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f880c14c140 con 0x7f880c14b0a0 2024-05-09T05:08:41.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.263+0000 7f8812225640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c14b0a0 0x7f880c0c52c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.263+0000 7f8812225640 1 -- 172.21.15.1:0/390205262 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f880c1e74a0 con 0x7f880c14b0a0 2024-05-09T05:08:41.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.264+0000 7f8812225640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c14b0a0 0x7f880c0c52c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8808011890 tx=0x7f8808011d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.264+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f880800faa0 con 0x7f880c14b0a0 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.264+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f88040056c0 con 0x7f880c154980 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.264+0000 7f8802ffd640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f87f00761d0 0x7f87f00786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.265+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f88040975d0 con 0x7f880c154980 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.265+0000 7f8811a24640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f87f00761d0 0x7f87f00786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.265+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f880800fc40 con 0x7f880c14b0a0 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.265+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c14b0a0 msgr2=0x7f880c0c52c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.265+0000 7f8802ffd640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c14b0a0 0x7f880c0c52c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8808011890 tx=0x7f8808011d50 comp rx=0 tx=0).stop 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.265+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c14b0a0 msgr2=0x7f880c0c52c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.265+0000 7f8802ffd640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c14b0a0 0x7f880c0c52c0 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8808011890 tx=0x7f8808011d50 comp rx=0 tx=0).stop 2024-05-09T05:08:41.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.265+0000 7f8811a24640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f87f00761d0 0x7f87f00786e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f87fc034000 tx=0x7f87fc02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.273+0000 7f8814489640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c1f3ba0 0x7f880c1e5a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.273+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f880c050870 con 0x7f880c1f3ba0 2024-05-09T05:08:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.273+0000 7f8812225640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c1f3ba0 0x7f880c1e5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.274+0000 7f8812225640 1 -- 172.21.15.1:0/390205262 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f880c153eb0 con 0x7f880c1f3ba0 2024-05-09T05:08:41.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.274+0000 7f8812225640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c1f3ba0 0x7f880c1e5a20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8808011890 tx=0x7f88080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.274+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8808014640 con 0x7f880c1f3ba0 2024-05-09T05:08:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f880802c420 con 0x7f880c1f3ba0 2024-05-09T05:08:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c1f3ba0 msgr2=0x7f880c1e5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8802ffd640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c1f3ba0 0x7f880c1e5a20 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8808011890 tx=0x7f88080046c0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8802ffd640 1 -- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c1f3ba0 msgr2=0x7f880c1e5a20 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:41.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8802ffd640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f880c1f3ba0 0x7f880c1e5a20 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8808011890 tx=0x7f88080046c0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 24, 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 185365, 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.276 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 15, 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.277 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.278 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:08:41.279 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.280 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:08:41.281 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.282 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:41.284 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:08:41.285 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:41.286 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.287 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.288 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.289 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:08:41.289 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:08:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f87f00761d0 msgr2=0x7f87f00786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f87f00761d0 0x7f87f00786e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f87fc034000 tx=0x7f87fc02e000 comp rx=0 tx=0).stop 2024-05-09T05:08:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8812a26640 1 -- 172.21.15.1:0/390205262 reap_dead start 2024-05-09T05:08:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f880c154980 msgr2=0x7f880c0c4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f880c154980 0x7f880c0c4d80 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f880400c730 tx=0x7f880400cbf0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 shutdown_connections 2024-05-09T05:08:41.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 --2- 172.21.15.1:0/390205262 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f880c154980 0x7f880c0c4d80 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 >> 172.21.15.1:0/390205262 conn(0x7f880c072e50 msgr2=0x7f880c0c5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 shutdown_connections 2024-05-09T05:08:41.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.275+0000 7f8814489640 1 -- 172.21.15.1:0/390205262 wait complete. 2024-05-09T05:08:41.291 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:08:41.291 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:08:41.291 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:08:41.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.428+0000 7f277902b640 1 Processor -- start 2024-05-09T05:08:41.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.428+0000 7f277902b640 1 -- start start 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.429+0000 7f277902b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740736b0 0x7f27740c7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.429+0000 7f277902b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27740689e0 0x7f27740c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.429+0000 7f277902b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f27740cce20 con 0x7f27740cc970 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.429+0000 7f277902b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f27740cd100 con 0x7f27740689e0 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.429+0000 7f277902b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f27740cd3e0 con 0x7f27740736b0 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27740689e0 0x7f27740c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f27740689e0 0x7f27740c7d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53260/0 (socket says 172.21.15.1:53260) 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277359e640 1 -- 172.21.15.1:0/782441225 learned_addr learned my addr 172.21.15.1:0/782441225 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277259c640 1 --2- 172.21.15.1:0/782441225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740736b0 0x7f27740c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277359e640 1 -- 172.21.15.1:0/782441225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740736b0 msgr2=0x7f27740c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277359e640 1 --2- 172.21.15.1:0/782441225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740736b0 0x7f27740c7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277359e640 1 -- 172.21.15.1:0/782441225 >> v1:172.21.15.196:6789/0 conn(0x7f27740cc970 legacy=0x7f277415b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277359e640 1 -- 172.21.15.1:0/782441225 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f277415a8c0 con 0x7f27740689e0 2024-05-09T05:08:41.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.430+0000 7f277259c640 1 --2- 172.21.15.1:0/782441225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740736b0 0x7f27740c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.431+0000 7f277359e640 1 --2- 172.21.15.1:0/782441225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 0x7f27740c7d50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f276c009e50 tx=0x7f276c02a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c42c6cc5f0841cf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.431+0000 7f2771d9b640 1 -- 172.21.15.1:0/782441225 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c036070 con 0x7f27740689e0 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.431+0000 7f2771d9b640 1 -- 172.21.15.1:0/782441225 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f276c031600 con 0x7f27740689e0 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.431+0000 7f2771d9b640 1 -- 172.21.15.1:0/782441225 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276c0319d0 con 0x7f27740689e0 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.432+0000 7f277902b640 1 -- 172.21.15.1:0/782441225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 msgr2=0x7f27740c7d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.432+0000 7f277902b640 1 --2- 172.21.15.1:0/782441225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 0x7f27740c7d50 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f276c009e50 tx=0x7f276c02a710 comp rx=0 tx=0).stop 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.432+0000 7f277902b640 1 -- 172.21.15.1:0/782441225 shutdown_connections 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.432+0000 7f277902b640 1 --2- 172.21.15.1:0/782441225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 0x7f27740c7d50 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.432+0000 7f277902b640 1 --2- 172.21.15.1:0/782441225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740736b0 0x7f27740c7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.432+0000 7f277902b640 1 -- 172.21.15.1:0/782441225 >> 172.21.15.1:0/782441225 conn(0x7f27740ce180 msgr2=0x7f27740ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.432+0000 7f277902b640 1 -- 172.21.15.1:0/782441225 shutdown_connections 2024-05-09T05:08:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.432+0000 7f277902b640 1 -- 172.21.15.1:0/782441225 wait complete. 2024-05-09T05:08:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.434+0000 7f277902b640 1 Processor -- start 2024-05-09T05:08:41.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.434+0000 7f277902b640 1 -- start start 2024-05-09T05:08:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.435+0000 7f277902b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 0x7f277414f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.435+0000 7f277902b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27740736b0 0x7f27740c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.435+0000 7f277902b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740cc970 0x7f27740c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.435+0000 7f277902b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f277406c9c0 con 0x7f27740689e0 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.435+0000 7f277902b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2774154d20 con 0x7f27740736b0 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.435+0000 7f277902b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f277406cb40 con 0x7f27740cc970 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 0x7f277414f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 0x7f277414f5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53274/0 (socket says 172.21.15.1:53274) 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 -- 172.21.15.1:0/2917889696 learned_addr learned my addr 172.21.15.1:0/2917889696 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f277259c640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27740736b0 0x7f27740c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f277359e640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740cc970 0x7f27740c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 -- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740cc970 msgr2=0x7f27740c68c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740cc970 0x7f27740c68c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 -- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27740736b0 msgr2=0x7f27740c6380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f27740736b0 0x7f27740c6380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 -- 172.21.15.1:0/2917889696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27741e6c90 con 0x7f27740689e0 2024-05-09T05:08:41.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f2772d9d640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 0x7f277414f5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f276400c3d0 tx=0x7f276400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764017070 con 0x7f27740689e0 2024-05-09T05:08:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27640052d0 con 0x7f27740689e0 2024-05-09T05:08:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27741e64d0 con 0x7f27740689e0 2024-05-09T05:08:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.436+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2764003830 con 0x7f27740689e0 2024-05-09T05:08:41.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.437+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27740c5a70 con 0x7f27740689e0 2024-05-09T05:08:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.438+0000 7f277902b640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740c5600 0x7f27741e66a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.438+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2774050870 con 0x7f27740c5600 2024-05-09T05:08:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.438+0000 7f277259c640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740c5600 0x7f27741e66a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.438+0000 7f277259c640 1 -- 172.21.15.1:0/2917889696 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f277406c6f0 con 0x7f27740c5600 2024-05-09T05:08:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.438+0000 7f277259c640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740c5600 0x7f27741e66a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f277414ffe0 tx=0x7f2768011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.439+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2768019070 con 0x7f27740c5600 2024-05-09T05:08:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.439+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f276401a070 con 0x7f27740689e0 2024-05-09T05:08:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.440+0000 7f275b7fe640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f27600761d0 0x7f27600786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.440+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2768004240 con 0x7f27740c5600 2024-05-09T05:08:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.440+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740c5600 msgr2=0x7f27741e66a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.440+0000 7f275b7fe640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740c5600 0x7f27741e66a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f277414ffe0 tx=0x7f2768011cf0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.440+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740c5600 msgr2=0x7f27741e66a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.440+0000 7f275b7fe640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27740c5600 0x7f27741e66a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.440+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2764096890 con 0x7f27740689e0 2024-05-09T05:08:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.440+0000 7f277359e640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f27600761d0 0x7f27600786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.442+0000 7f277359e640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f27600761d0 0x7f27600786e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f276c033000 tx=0x7f276c009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.451+0000 7f277902b640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f277406e0c0 0x7f27741e4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.451+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f277406cd40 con 0x7f277406e0c0 2024-05-09T05:08:41.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.451+0000 7f277259c640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f277406e0c0 0x7f27741e4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.451+0000 7f277259c640 1 -- 172.21.15.1:0/2917889696 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f27740b9d70 con 0x7f277406e0c0 2024-05-09T05:08:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.452+0000 7f277259c640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f277406e0c0 0x7f27741e4de0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f27740cab60 tx=0x7f27680046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.452+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f276801e650 con 0x7f277406e0c0 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.452+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7f2768003e50 con 0x7f277406e0c0 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.452+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f277406e0c0 msgr2=0x7f27741e4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.452+0000 7f275b7fe640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f277406e0c0 0x7f27741e4de0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f27740cab60 tx=0x7f27680046c0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.452+0000 7f275b7fe640 1 -- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f277406e0c0 msgr2=0x7f27741e4de0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.452+0000 7f275b7fe640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f277406e0c0 0x7f27741e4de0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:41.453 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 24, 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 79450, 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 15, 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.454 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.455 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:08:41.456 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:08:41.457 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:08:41.458 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.459 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:41.460 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:08:41.461 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.464 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:41.465 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:41.466 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:08:41.466 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:08:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f27600761d0 msgr2=0x7f27600786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277902b640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f27600761d0 0x7f27600786e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f276c033000 tx=0x7f276c009000 comp rx=0 tx=0).stop 2024-05-09T05:08:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 msgr2=0x7f277414f5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277902b640 1 --2- 172.21.15.1:0/2917889696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27740689e0 0x7f277414f5a0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f276400c3d0 tx=0x7f276400c890 comp rx=0 tx=0).stop 2024-05-09T05:08:41.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277359e640 1 -- 172.21.15.1:0/2917889696 reap_dead start 2024-05-09T05:08:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277359e640 1 -- 172.21.15.1:0/2917889696 reap_dead start 2024-05-09T05:08:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 shutdown_connections 2024-05-09T05:08:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 >> 172.21.15.1:0/2917889696 conn(0x7f27740ce180 msgr2=0x7f27741531c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 shutdown_connections 2024-05-09T05:08:41.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.453+0000 7f277902b640 1 -- 172.21.15.1:0/2917889696 wait complete. 2024-05-09T05:08:41.468 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:08:41.469 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-05-09T05:08:41.469 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:08:41.469 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-05-09T05:08:41.469 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:08:41.469 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T05:08:41.469 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:08:41.487 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T05:08:41.487 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:08:41.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.614+0000 7fafe7484640 1 Processor -- start 2024-05-09T05:08:41.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.614+0000 7fafe7484640 1 -- start start 2024-05-09T05:08:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.616+0000 7fafe7484640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe006c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.616+0000 7fafe7484640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafe0068cd0 con 0x7fafe00688c0 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.616+0000 7fafe7484640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fafe00c89d0 con 0x7fafe0154760 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.616+0000 7fafe7484640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fafe00c8c70 con 0x7fafe00b9f50 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.616+0000 7fafe5220640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fafe0154760 legacy=0x7fafe00c76a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.616+0000 7fafe5a21640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fafe00688c0 0x7fafe015a630 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44242/0 (socket says 172.21.15.1:44242) 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.616+0000 7fafe5a21640 1 -- 172.21.15.1:0/444369780 learned_addr learned my addr 172.21.15.1:0/444369780 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.616+0000 7fafe4a1f640 1 --2- 172.21.15.1:0/444369780 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe006c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.617+0000 7fafd7fff640 1 -- 172.21.15.1:0/444369780 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 942499820 0 0) 0x7fafd8001490 con 0x7fafe00688c0 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.617+0000 7fafd7fff640 1 -- 172.21.15.1:0/444369780 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fafcc003520 con 0x7fafe00688c0 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.617+0000 7fafe4a1f640 1 -- 172.21.15.1:0/444369780 >> v1:172.21.15.196:6789/0 conn(0x7fafe0154760 legacy=0x7fafe00c76a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.617+0000 7fafe4a1f640 1 -- 172.21.15.1:0/444369780 >> v1:172.21.15.1:6789/0 conn(0x7fafe00688c0 legacy=0x7fafe015a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.617+0000 7fafe4a1f640 1 -- 172.21.15.1:0/444369780 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe00c93f0 con 0x7fafe00b9f50 2024-05-09T05:08:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.618+0000 7fafe4a1f640 1 --2- 172.21.15.1:0/444369780 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe006c6f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fafd00097d0 tx=0x7fafd002a160 comp rx=0 tx=0).ready entity=mon.2 client_cookie=495f79c790eb9c07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.618+0000 7fafd7fff640 1 -- 172.21.15.1:0/444369780 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0037070 con 0x7fafe00b9f50 2024-05-09T05:08:41.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.618+0000 7fafd7fff640 1 -- 172.21.15.1:0/444369780 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafd0032440 con 0x7fafe00b9f50 2024-05-09T05:08:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.618+0000 7fafd7fff640 1 -- 172.21.15.1:0/444369780 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd0032810 con 0x7fafe00b9f50 2024-05-09T05:08:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.619+0000 7fafe7484640 1 -- 172.21.15.1:0/444369780 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 msgr2=0x7fafe006c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.619+0000 7fafe7484640 1 --2- 172.21.15.1:0/444369780 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe006c6f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fafd00097d0 tx=0x7fafd002a160 comp rx=0 tx=0).stop 2024-05-09T05:08:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.619+0000 7fafe7484640 1 -- 172.21.15.1:0/444369780 shutdown_connections 2024-05-09T05:08:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.619+0000 7fafe7484640 1 --2- 172.21.15.1:0/444369780 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe006c6f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.619+0000 7fafe7484640 1 -- 172.21.15.1:0/444369780 >> 172.21.15.1:0/444369780 conn(0x7fafe00caec0 msgr2=0x7fafe00cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.619+0000 7fafe7484640 1 -- 172.21.15.1:0/444369780 shutdown_connections 2024-05-09T05:08:41.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.619+0000 7fafe7484640 1 -- 172.21.15.1:0/444369780 wait complete. 2024-05-09T05:08:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.621+0000 7fafe7484640 1 Processor -- start 2024-05-09T05:08:41.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.621+0000 7fafe7484640 1 -- start start 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe7484640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe00688c0 0x7fafe0141c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe7484640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe014d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe7484640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe0154760 0x7fafe014db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe7484640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fafe00c9c90 con 0x7fafe00688c0 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe7484640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fafe00c99b0 con 0x7fafe0154760 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe7484640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fafe00c8f50 con 0x7fafe00b9f50 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5220640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe00688c0 0x7fafe0141c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5a21640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe0154760 msgr2=0x7fafe014db00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:41.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5220640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe00688c0 0x7fafe0141c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53336/0 (socket says 172.21.15.1:53336) 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5220640 1 -- 172.21.15.1:0/2603097060 learned_addr learned my addr 172.21.15.1:0/2603097060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5a21640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe0154760 0x7fafe014db00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe4a1f640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe014d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5220640 1 -- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 msgr2=0x7fafe014d5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5220640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe014d5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5220640 1 -- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe0154760 msgr2=0x7fafe014db00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5220640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe0154760 0x7fafe014db00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.622+0000 7fafe5220640 1 -- 172.21.15.1:0/2603097060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafe0152130 con 0x7fafe00688c0 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.623+0000 7fafe5220640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe00688c0 0x7fafe0141c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00b4a0 tx=0x7fafdc00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.623+0000 7fafd5ffb640 1 -- 172.21.15.1:0/2603097060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafdc003b40 con 0x7fafe00688c0 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.623+0000 7fafd5ffb640 1 -- 172.21.15.1:0/2603097060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fafdc015070 con 0x7fafe00688c0 2024-05-09T05:08:41.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.623+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafe0143af0 con 0x7fafe00688c0 2024-05-09T05:08:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.623+0000 7fafd5ffb640 1 -- 172.21.15.1:0/2603097060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafdc005070 con 0x7fafe00688c0 2024-05-09T05:08:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.623+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafe0152b40 con 0x7fafe00688c0 2024-05-09T05:08:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.624+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fafe0050870 con 0x7fafe00688c0 2024-05-09T05:08:41.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.625+0000 7fafd5ffb640 1 -- 172.21.15.1:0/2603097060 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fafdc005210 con 0x7fafe00688c0 2024-05-09T05:08:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.626+0000 7fafd5ffb640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fafc40762b0 0x7fafc40787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.626+0000 7fafd5ffb640 1 -- 172.21.15.1:0/2603097060 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fafdc0970d0 con 0x7fafe00688c0 2024-05-09T05:08:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.626+0000 7fafe4a1f640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fafc40762b0 0x7fafc40787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:41.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.627+0000 7fafe4a1f640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fafc40762b0 0x7fafc40787c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fafd0034000 tx=0x7fafd002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:41.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.633+0000 7fafd5ffb640 1 -- 172.21.15.1:0/2603097060 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fafdc0608a0 con 0x7fafe00688c0 2024-05-09T05:08:41.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.931+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fafe006cc60 con 0x7fafe00688c0 2024-05-09T05:08:41.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.931+0000 7fafd5ffb640 1 -- 172.21.15.1:0/2603097060 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fafdc0601a0 con 0x7fafe00688c0 2024-05-09T05:08:41.932 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:41.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fafc40762b0 msgr2=0x7fafc40787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fafc40762b0 0x7fafc40787c0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fafd0034000 tx=0x7fafd002e000 comp rx=0 tx=0).stop 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe00688c0 msgr2=0x7fafe0141c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe00688c0 0x7fafe0141c30 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fafdc00b4a0 tx=0x7fafdc00b960 comp rx=0 tx=0).stop 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 shutdown_connections 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fafc40762b0 0x7fafc40787c0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fafe0154760 0x7fafe014db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fafe00b9f50 0x7fafe014d5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 --2- 172.21.15.1:0/2603097060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fafe00688c0 0x7fafe0141c30 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 >> 172.21.15.1:0/2603097060 conn(0x7fafe00caec0 msgr2=0x7fafe0146fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 shutdown_connections 2024-05-09T05:08:41.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:41.934+0000 7fafe7484640 1 -- 172.21.15.1:0/2603097060 wait complete. 2024-05-09T05:08:44.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:45.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.077+0000 7f04d5b08640 1 Processor -- start 2024-05-09T05:08:45.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.077+0000 7f04d5b08640 1 -- start start 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04d5b08640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f04d0159440 0x7f04d0070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04d5b08640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0153830 0x7f04d00cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04d5b08640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f04d00c4430 con 0x7f04d0070530 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04d5b08640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f04d00c3b90 con 0x7f04d0159440 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04d5b08640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f04d00c3e70 con 0x7f04d0153830 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04cf7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f04d0159440 0x7f04d0070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04ceffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f04d0070530 0x7f04d015d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44248/0 (socket says 172.21.15.1:44248) 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04ceffd640 1 -- 172.21.15.1:0/789357397 learned_addr learned my addr 172.21.15.1:0/789357397 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04cffff640 1 --2- 172.21.15.1:0/789357397 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0153830 0x7f04d00cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04cf7fe640 1 -- 172.21.15.1:0/789357397 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0153830 msgr2=0x7f04d00cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04cf7fe640 1 --2- 172.21.15.1:0/789357397 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0153830 0x7f04d00cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04cf7fe640 1 -- 172.21.15.1:0/789357397 >> v1:172.21.15.1:6789/0 conn(0x7f04d0070530 legacy=0x7f04d015d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:45.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04cf7fe640 1 -- 172.21.15.1:0/789357397 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d00c4150 con 0x7f04d0159440 2024-05-09T05:08:45.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04ce7fc640 1 -- 172.21.15.1:0/789357397 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3703459372 0 0) 0x7f04d00c4430 con 0x7f04d0070530 2024-05-09T05:08:45.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.079+0000 7f04cffff640 1 --2- 172.21.15.1:0/789357397 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0153830 0x7f04d00cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.080+0000 7f04cf7fe640 1 --2- 172.21.15.1:0/789357397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d0070ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04c00097a0 tx=0x7f04c002a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b35d5d561fc2c208 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.080+0000 7f04ce7fc640 1 -- 172.21.15.1:0/789357397 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0037070 con 0x7f04d0159440 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.080+0000 7f04ce7fc640 1 -- 172.21.15.1:0/789357397 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c0032440 con 0x7f04d0159440 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.080+0000 7f04ce7fc640 1 -- 172.21.15.1:0/789357397 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c0032810 con 0x7f04d0159440 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.081+0000 7f04d5b08640 1 -- 172.21.15.1:0/789357397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 msgr2=0x7f04d0070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.081+0000 7f04d5b08640 1 --2- 172.21.15.1:0/789357397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d0070ae0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f04c00097a0 tx=0x7f04c002a210 comp rx=0 tx=0).stop 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.081+0000 7f04d5b08640 1 -- 172.21.15.1:0/789357397 shutdown_connections 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.081+0000 7f04d5b08640 1 --2- 172.21.15.1:0/789357397 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0153830 0x7f04d00cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.081+0000 7f04d5b08640 1 --2- 172.21.15.1:0/789357397 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d0070ae0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.081+0000 7f04d5b08640 1 -- 172.21.15.1:0/789357397 >> 172.21.15.1:0/789357397 conn(0x7f04d0071b40 msgr2=0x7f04d0071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:45.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.081+0000 7f04d5b08640 1 -- 172.21.15.1:0/789357397 shutdown_connections 2024-05-09T05:08:45.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.081+0000 7f04d5b08640 1 -- 172.21.15.1:0/789357397 wait complete. 2024-05-09T05:08:45.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.084+0000 7f04d5b08640 1 Processor -- start 2024-05-09T05:08:45.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.084+0000 7f04d5b08640 1 -- start start 2024-05-09T05:08:45.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.085+0000 7f04d5b08640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0070530 0x7f04d00c5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.085+0000 7f04d5b08640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04d0153830 0x7f04d00ca3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04d5b08640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d01390f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04d5b08640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f04d00c3870 con 0x7f04d0159440 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04d5b08640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f04d015ceb0 con 0x7f04d0153830 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04d5b08640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f04d006fad0 con 0x7f04d0070530 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04ceffd640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04d0153830 msgr2=0x7f04d00ca3f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04ceffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04d0153830 0x7f04d00ca3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cf7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0070530 0x7f04d00c5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d01390f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cf7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0070530 0x7f04d00c5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:53458/0 (socket says 172.21.15.1:53458) 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cf7fe640 1 -- 172.21.15.1:0/2290752430 learned_addr learned my addr 172.21.15.1:0/2290752430 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cffff640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d01390f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53342/0 (socket says 172.21.15.1:53342) 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cffff640 1 -- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0070530 msgr2=0x7f04d00c5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:45.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cffff640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0070530 0x7f04d00c5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cffff640 1 -- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04d0153830 msgr2=0x7f04d00ca3f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cffff640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04d0153830 0x7f04d00ca3f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cffff640 1 -- 172.21.15.1:0/2290752430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04d00c69f0 con 0x7f04d0159440 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.086+0000 7f04cf7fe640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0070530 0x7f04d00c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.087+0000 7f04cffff640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d01390f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04c800b330 tx=0x7f04c800b7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.087+0000 7f04b7fff640 1 -- 172.21.15.1:0/2290752430 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c800be00 con 0x7f04d0159440 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.087+0000 7f04b7fff640 1 -- 172.21.15.1:0/2290752430 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f04c8003990 con 0x7f04d0159440 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.087+0000 7f04b7fff640 1 -- 172.21.15.1:0/2290752430 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04c8003d80 con 0x7f04d0159440 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.087+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04d00c6520 con 0x7f04d0159440 2024-05-09T05:08:45.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.087+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04d0154b80 con 0x7f04d0159440 2024-05-09T05:08:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.088+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04d0050870 con 0x7f04d0159440 2024-05-09T05:08:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.090+0000 7f04b7fff640 1 -- 172.21.15.1:0/2290752430 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f04c8005d10 con 0x7f04d0159440 2024-05-09T05:08:45.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.090+0000 7f04b7fff640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f04bc076300 0x7f04bc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:45.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.090+0000 7f04b7fff640 1 -- 172.21.15.1:0/2290752430 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f04c809b640 con 0x7f04d0159440 2024-05-09T05:08:45.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.090+0000 7f04cf7fe640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f04bc076300 0x7f04bc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:45.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.091+0000 7f04cf7fe640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f04bc076300 0x7f04bc078810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f04c00097a0 tx=0x7f04c002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:45.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.096+0000 7f04b7fff640 1 -- 172.21.15.1:0/2290752430 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f04c8065000 con 0x7f04d0159440 2024-05-09T05:08:45.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.433+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f04d00c94c0 con 0x7f04d0159440 2024-05-09T05:08:45.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.434+0000 7f04b7fff640 1 -- 172.21.15.1:0/2290752430 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f04c8064900 con 0x7f04d0159440 2024-05-09T05:08:45.435 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:45.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f04bc076300 msgr2=0x7f04bc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:45.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f04bc076300 0x7f04bc078810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f04c00097a0 tx=0x7f04c002e000 comp rx=0 tx=0).stop 2024-05-09T05:08:45.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 msgr2=0x7f04d01390f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d01390f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f04c800b330 tx=0x7f04c800b7f0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 shutdown_connections 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f04bc076300 0x7f04bc078810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f04d0153830 0x7f04d00ca3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f04d0159440 0x7f04d01390f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 --2- 172.21.15.1:0/2290752430 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f04d0070530 0x7f04d00c5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 >> 172.21.15.1:0/2290752430 conn(0x7f04d0071b40 msgr2=0x7f04d013fda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 shutdown_connections 2024-05-09T05:08:45.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:45.437+0000 7f04d5b08640 1 -- 172.21.15.1:0/2290752430 wait complete. 2024-05-09T05:08:48.450 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:48.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.577+0000 7f2c65fbe640 1 Processor -- start 2024-05-09T05:08:48.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.577+0000 7f2c65fbe640 1 -- start start 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.578+0000 7f2c65fbe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c60072120 0x7f2c60156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.578+0000 7f2c65fbe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c600caab0 0x7f2c600caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.578+0000 7f2c65fbe640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2c600c5e00 con 0x7f2c6015d1d0 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.578+0000 7f2c65fbe640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2c600ca340 con 0x7f2c600caab0 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.578+0000 7f2c65fbe640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2c600ca620 con 0x7f2c60072120 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2c600caab0 0x7f2c600caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2c6015d1d0 0x7f2c60070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44252/0 (socket says 172.21.15.1:44252) 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5effd640 1 -- 172.21.15.1:0/4122857835 learned_addr learned my addr 172.21.15.1:0/4122857835 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5ffff640 1 --2- 172.21.15.1:0/4122857835 >> v2:172.21.15.1:3300/0 conn(0x7f2c600caab0 0x7f2c600caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53352/0 (socket says 172.21.15.1:53352) 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5f7fe640 1 --2- 172.21.15.1:0/4122857835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c60072120 0x7f2c60156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5ffff640 1 -- 172.21.15.1:0/4122857835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c60072120 msgr2=0x7f2c60156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5ffff640 1 --2- 172.21.15.1:0/4122857835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c60072120 0x7f2c60156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5ffff640 1 -- 172.21.15.1:0/4122857835 >> v1:172.21.15.1:6789/0 conn(0x7f2c6015d1d0 legacy=0x7f2c60070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:48.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.579+0000 7f2c5ffff640 1 -- 172.21.15.1:0/4122857835 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c600c60e0 con 0x7f2c600caab0 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.580+0000 7f2c5ffff640 1 --2- 172.21.15.1:0/4122857835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c600caab0 0x7f2c600caec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009d90 tx=0x7f2c5802a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc34072d86b929e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.580+0000 7f2c5e7fc640 1 -- 172.21.15.1:0/4122857835 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58007740 con 0x7f2c600caab0 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.580+0000 7f2c5e7fc640 1 -- 172.21.15.1:0/4122857835 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c580078e0 con 0x7f2c600caab0 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.580+0000 7f2c5e7fc640 1 -- 172.21.15.1:0/4122857835 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c58007cb0 con 0x7f2c600caab0 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.581+0000 7f2c65fbe640 1 -- 172.21.15.1:0/4122857835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c600caab0 msgr2=0x7f2c600caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.581+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/4122857835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c600caab0 0x7f2c600caec0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f2c58009d90 tx=0x7f2c5802a660 comp rx=0 tx=0).stop 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.581+0000 7f2c65fbe640 1 -- 172.21.15.1:0/4122857835 shutdown_connections 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.581+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/4122857835 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c600caab0 0x7f2c600caec0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.581+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/4122857835 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c60072120 0x7f2c60156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.581+0000 7f2c65fbe640 1 -- 172.21.15.1:0/4122857835 >> 172.21.15.1:0/4122857835 conn(0x7f2c600731d0 msgr2=0x7f2c600cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:48.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.581+0000 7f2c65fbe640 1 -- 172.21.15.1:0/4122857835 shutdown_connections 2024-05-09T05:08:48.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.581+0000 7f2c65fbe640 1 -- 172.21.15.1:0/4122857835 wait complete. 2024-05-09T05:08:48.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.583+0000 7f2c65fbe640 1 Processor -- start 2024-05-09T05:08:48.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.583+0000 7f2c65fbe640 1 -- start start 2024-05-09T05:08:48.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.584+0000 7f2c65fbe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60072120 0x7f2c6015db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:48.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.584+0000 7f2c65fbe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c600caab0 0x7f2c6015e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:48.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.584+0000 7f2c65fbe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c6015d1d0 0x7f2c601f5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:48.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.584+0000 7f2c65fbe640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2c600cb400 con 0x7f2c60072120 2024-05-09T05:08:48.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.584+0000 7f2c65fbe640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2c600c66a0 con 0x7f2c6015d1d0 2024-05-09T05:08:48.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.584+0000 7f2c65fbe640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2c6015d5e0 con 0x7f2c600caab0 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5ffff640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c6015d1d0 msgr2=0x7f2c601f5fe0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60072120 0x7f2c6015db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c6015d1d0 0x7f2c601f5fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60072120 0x7f2c6015db50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53360/0 (socket says 172.21.15.1:53360) 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 -- 172.21.15.1:0/1481349227 learned_addr learned my addr 172.21.15.1:0/1481349227 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5effd640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c600caab0 0x7f2c6015e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 -- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c600caab0 msgr2=0x7f2c6015e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c600caab0 0x7f2c6015e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 -- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c6015d1d0 msgr2=0x7f2c601f5fe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c6015d1d0 0x7f2c601f5fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 -- 172.21.15.1:0/1481349227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c6015e090 con 0x7f2c60072120 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5effd640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c600caab0 0x7f2c6015e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c5f7fe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60072120 0x7f2c6015db50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000e870 tx=0x7f2c5000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c47fff640 1 -- 172.21.15.1:0/1481349227 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c50017070 con 0x7f2c60072120 2024-05-09T05:08:48.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c47fff640 1 -- 172.21.15.1:0/1481349227 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2c5000c940 con 0x7f2c60072120 2024-05-09T05:08:48.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c60159b40 con 0x7f2c60072120 2024-05-09T05:08:48.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c47fff640 1 -- 172.21.15.1:0/1481349227 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c5000cd10 con 0x7f2c60072120 2024-05-09T05:08:48.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.585+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c600ce0e0 con 0x7f2c60072120 2024-05-09T05:08:48.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.587+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c60050870 con 0x7f2c60072120 2024-05-09T05:08:48.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.587+0000 7f2c47fff640 1 -- 172.21.15.1:0/1481349227 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2c5001a070 con 0x7f2c60072120 2024-05-09T05:08:48.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.588+0000 7f2c47fff640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2c4c076280 0x7f2c4c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:48.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.588+0000 7f2c47fff640 1 -- 172.21.15.1:0/1481349227 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2c500974e0 con 0x7f2c60072120 2024-05-09T05:08:48.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.588+0000 7f2c5effd640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2c4c076280 0x7f2c4c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:48.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.589+0000 7f2c5effd640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2c4c076280 0x7f2c4c078790 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2c54000f60 tx=0x7f2c54009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:48.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.593+0000 7f2c47fff640 1 -- 172.21.15.1:0/1481349227 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2c50060dc0 con 0x7f2c60072120 2024-05-09T05:08:48.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.923+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2c6015b520 con 0x7f2c60072120 2024-05-09T05:08:48.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.923+0000 7f2c47fff640 1 -- 172.21.15.1:0/1481349227 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f2c500606c0 con 0x7f2c60072120 2024-05-09T05:08:48.924 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":24,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2c4c076280 msgr2=0x7f2c4c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2c4c076280 0x7f2c4c078790 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2c54000f60 tx=0x7f2c54009040 comp rx=0 tx=0).stop 2024-05-09T05:08:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60072120 msgr2=0x7f2c6015db50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60072120 0x7f2c6015db50 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f2c5000e870 tx=0x7f2c5000ed30 comp rx=0 tx=0).stop 2024-05-09T05:08:48.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 shutdown_connections 2024-05-09T05:08:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2c4c076280 0x7f2c4c078790 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2c6015d1d0 0x7f2c601f5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2c600caab0 0x7f2c6015e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 --2- 172.21.15.1:0/1481349227 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2c60072120 0x7f2c6015db50 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 >> 172.21.15.1:0/1481349227 conn(0x7f2c600731d0 msgr2=0x7f2c600cc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 shutdown_connections 2024-05-09T05:08:48.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:48.926+0000 7f2c65fbe640 1 -- 172.21.15.1:0/1481349227 wait complete. 2024-05-09T05:08:51.941 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:08:52.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.064+0000 7f08b46f0640 1 Processor -- start 2024-05-09T05:08:52.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.064+0000 7f08b46f0640 1 -- start start 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b46f0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b46f0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ac1594f0 con 0x7f08ac1538b0 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b46f0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08ac159770 con 0x7f08ac15cc90 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b46f0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f08ac15a380 con 0x7f08ac0c87d0 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b2c8d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f08ac15cc90 legacy=0x7f08ac15d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b1c8b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f08ac1538b0 0x7f08ac0c92d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46458/0 (socket says 172.21.15.1:46458) 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b1c8b640 1 -- 172.21.15.1:0/1497553246 learned_addr learned my addr 172.21.15.1:0/1497553246 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:52.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.066+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:52.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.267+0000 7f08b2c8d640 1 -- 172.21.15.1:0/1497553246 >> v1:172.21.15.196:6789/0 conn(0x7f08ac15cc90 legacy=0x7f08ac15d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:08:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.267+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.267+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:08:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.267+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:08:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.267+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:52.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.267+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:08:52.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.668+0000 7f08b2c8d640 1 -- 172.21.15.1:0/1497553246 >> v1:172.21.15.196:6789/0 conn(0x7f08ac15cc90 legacy=0x7f08ac15d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:08:52.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.668+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:52.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.668+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:08:52.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.668+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:08:52.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.668+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:52.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:52.668+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:08:53.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:53.469+0000 7f08b2c8d640 1 -- 172.21.15.1:0/1497553246 >> v1:172.21.15.196:6789/0 conn(0x7f08ac15cc90 legacy=0x7f08ac15d0a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:08:53.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:53.469+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:53.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:53.469+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:08:53.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:53.469+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:08:53.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:53.469+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:53.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:53.469+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:08:55.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08a3fff640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 msgr2=0x7f08ac0c5ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:08:55.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08a3fff640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08ac0c87d0 0x7f08ac0c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08a3fff640 1 -- 172.21.15.1:0/1497553246 >> v1:172.21.15.196:6789/0 conn(0x7f08ac15cc90 legacy=0x7f08ac15d0a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08a3fff640 1 -- 172.21.15.1:0/1497553246 >> v1:172.21.15.1:6789/0 conn(0x7f08ac1538b0 legacy=0x7f08ac0c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08a3fff640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08a3fff640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 0x7f089000a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08a3fff640 1 -- 172.21.15.1:0/1497553246 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f08ac15a380 con 0x7f08900058e0 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08a3fff640 1 -- 172.21.15.1:0/1497553246 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f089000afc0 con 0x7f0890008400 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08b1c8b640 1 -- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 msgr2=0x7f089000a910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08b1c8b640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 0x7f089000a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 msgr2=0x7f0890007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 msgr2=0x7f0890007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:55.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.066+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.267+0000 7f08b1c8b640 1 -- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 msgr2=0x7f089000a910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.267+0000 7f08b1c8b640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 0x7f089000a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:08:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.267+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.267+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 msgr2=0x7f0890007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:08:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.267+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 msgr2=0x7f0890007db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:08:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.267+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:08:55.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.267+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:08:55.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.668+0000 7f08b1c8b640 1 -- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 msgr2=0x7f089000a910 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:55.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.668+0000 7f08b1c8b640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 0x7f089000a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:08:55.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.668+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.1:3300/0 conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:55.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.668+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 msgr2=0x7f089000a910 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:08:55.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.668+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> v2:172.21.15.196:3300/0 conn(0x7f0890008400 0x7f089000a910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.668+0000 7f08b248c640 1 -- 172.21.15.1:0/1497553246 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ac159a50 con 0x7f08900058e0 2024-05-09T05:08:55.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.669+0000 7f08b248c640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08900058e0 0x7f0890007db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f08900083d0 tx=0x7f08a802bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=227c407b0cea3b3a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:55.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.676+0000 7f08b148a640 1 -- 172.21.15.1:0/1497553246 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a802d500 con 0x7f08900058e0 2024-05-09T05:08:55.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.680+0000 7f08b148a640 1 -- 172.21.15.1:0/1497553246 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a802d6a0 con 0x7f08900058e0 2024-05-09T05:08:55.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.680+0000 7f08b148a640 1 -- 172.21.15.1:0/1497553246 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a802da90 con 0x7f08900058e0 2024-05-09T05:08:55.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.681+0000 7f08b46f0640 1 -- 172.21.15.1:0/1497553246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08900058e0 msgr2=0x7f0890007db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:55.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.681+0000 7f08b46f0640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08900058e0 0x7f0890007db0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f08900083d0 tx=0x7f08a802bbf0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.681+0000 7f08b2c8d640 1 -- 172.21.15.1:0/1497553246 reap_dead start 2024-05-09T05:08:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.681+0000 7f08b46f0640 1 -- 172.21.15.1:0/1497553246 shutdown_connections 2024-05-09T05:08:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.681+0000 7f08b46f0640 1 --2- 172.21.15.1:0/1497553246 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08900058e0 0x7f0890007db0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.681+0000 7f08b46f0640 1 -- 172.21.15.1:0/1497553246 >> 172.21.15.1:0/1497553246 conn(0x7f08ac0745a0 msgr2=0x7f08ac071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.682+0000 7f08b46f0640 1 -- 172.21.15.1:0/1497553246 shutdown_connections 2024-05-09T05:08:55.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.682+0000 7f08b46f0640 1 -- 172.21.15.1:0/1497553246 wait complete. 2024-05-09T05:08:55.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.684+0000 7f08b46f0640 1 Processor -- start 2024-05-09T05:08:55.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.685+0000 7f08b46f0640 1 -- start start 2024-05-09T05:08:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.686+0000 7f08b46f0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900058e0 0x7f08ac158a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.686+0000 7f08b46f0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08ac06c950 0x7f08ac15b030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.686+0000 7f08b46f0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1470c0 0x7f08ac1474d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.686+0000 7f08b46f0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f08ac0c61e0 con 0x7f08ac1470c0 2024-05-09T05:08:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.686+0000 7f08b46f0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f08ac15a010 con 0x7f08ac06c950 2024-05-09T05:08:55.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.686+0000 7f08b46f0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f08ac0c99e0 con 0x7f08900058e0 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.686+0000 7f08b1c8b640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08ac06c950 msgr2=0x7f08ac15b030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b1c8b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08ac06c950 0x7f08ac15b030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1470c0 0x7f08ac1474d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1470c0 0x7f08ac1474d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48954/0 (socket says 172.21.15.1:48954) 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 -- 172.21.15.1:0/938100775 learned_addr learned my addr 172.21.15.1:0/938100775 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b248c640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900058e0 0x7f08ac158a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 -- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900058e0 msgr2=0x7f08ac158a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900058e0 0x7f08ac158a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 -- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08ac06c950 msgr2=0x7f08ac15b030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08ac06c950 0x7f08ac15b030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 -- 172.21.15.1:0/938100775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08ac1f2b40 con 0x7f08ac1470c0 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b248c640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900058e0 0x7f08ac158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:55.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.687+0000 7f08b2c8d640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1470c0 0x7f08ac1474d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f08ac15dc00 tx=0x7f08a400be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.688+0000 7f08a2ffd640 1 -- 172.21.15.1:0/938100775 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a400ce30 con 0x7f08ac1470c0 2024-05-09T05:08:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.688+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08ac1f1d80 con 0x7f08ac1470c0 2024-05-09T05:08:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.688+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08ac1f2340 con 0x7f08ac1470c0 2024-05-09T05:08:55.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.688+0000 7f08a2ffd640 1 -- 172.21.15.1:0/938100775 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08a4010030 con 0x7f08ac1470c0 2024-05-09T05:08:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.688+0000 7f08a2ffd640 1 -- 172.21.15.1:0/938100775 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08a401b060 con 0x7f08ac1470c0 2024-05-09T05:08:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.689+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0894005290 con 0x7f08ac1470c0 2024-05-09T05:08:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.690+0000 7f08a2ffd640 1 -- 172.21.15.1:0/938100775 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f08a40155e0 con 0x7f08ac1470c0 2024-05-09T05:08:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.691+0000 7f08a2ffd640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08900792e0 0x7f089007b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:55.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.691+0000 7f08a2ffd640 1 -- 172.21.15.1:0/938100775 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f08a40973b0 con 0x7f08ac1470c0 2024-05-09T05:08:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.691+0000 7f08b248c640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08900792e0 0x7f089007b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.692+0000 7f08b248c640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08900792e0 0x7f089007b7f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f08a8036000 tx=0x7f08a802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:55.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.696+0000 7f08a2ffd640 1 -- 172.21.15.1:0/938100775 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f08a4060650 con 0x7f08ac1470c0 2024-05-09T05:08:55.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.963+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0894005850 con 0x7f08ac1470c0 2024-05-09T05:08:55.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.963+0000 7f08a2ffd640 1 -- 172.21.15.1:0/938100775 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f08a4064950 con 0x7f08ac1470c0 2024-05-09T05:08:55.964 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:08:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08900792e0 msgr2=0x7f089007b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08900792e0 0x7f089007b7f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f08a8036000 tx=0x7f08a802f000 comp rx=0 tx=0).stop 2024-05-09T05:08:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1470c0 msgr2=0x7f08ac1474d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1470c0 0x7f08ac1474d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f08ac15dc00 tx=0x7f08a400be60 comp rx=0 tx=0).stop 2024-05-09T05:08:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 shutdown_connections 2024-05-09T05:08:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08900792e0 0x7f089007b7f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08ac06c950 0x7f08ac15b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08ac1470c0 0x7f08ac1474d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 --2- 172.21.15.1:0/938100775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900058e0 0x7f08ac158a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:55.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 >> 172.21.15.1:0/938100775 conn(0x7f08ac0745a0 msgr2=0x7f08ac0cd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:55.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 shutdown_connections 2024-05-09T05:08:55.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:55.965+0000 7f08b46f0640 1 -- 172.21.15.1:0/938100775 wait complete. 2024-05-09T05:08:55.977 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:08:55.977 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:08:55.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:08:56.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.109+0000 7f803159f640 1 Processor -- start 2024-05-09T05:08:56.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.109+0000 7f803159f640 1 -- start start 2024-05-09T05:08:56.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f803159f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f802c159f40 0x7f802c0c9250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f803159f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f802c0ca950 con 0x7f802c153800 2024-05-09T05:08:56.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f803159f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f802c0cac30 con 0x7f802c0cb2e0 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f803159f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f802c0caf10 con 0x7f802c159f40 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f802b7fe640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f802c159f40 msgr2=0x7f802c0c9250 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f8030d9e640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f802c0cb2e0 legacy=0x7f802c163d60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f802b7fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f802c159f40 0x7f802c0c9250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f802bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f802c153800 0x7f802c071ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46466/0 (socket says 172.21.15.1:46466) 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.110+0000 7f802bfff640 1 -- 172.21.15.1:0/797027796 learned_addr learned my addr 172.21.15.1:0/797027796 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2123116369 0 0) 0x7f802c0ca950 con 0x7f802c153800 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8014003520 con 0x7f802c153800 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4200739521 0 0) 0x7f8014003520 con 0x7f802c153800 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 >> v2:172.21.15.196:3300/0 conn(0x7f802c159f40 msgr2=0x7f802c0c9250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 --2- 172.21.15.1:0/797027796 >> v2:172.21.15.196:3300/0 conn(0x7f802c159f40 0x7f802c0c9250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 >> v1:172.21.15.196:6789/0 conn(0x7f802c0cb2e0 legacy=0x7f802c163d60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:08:56.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c0716f0 con 0x7f802c153800 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f801c0016f0 con 0x7f802c153800 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 >> v1:172.21.15.1:6789/0 conn(0x7f802c153800 legacy=0x7f802c071ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f8014003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8014004800 0x7f8014004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8014005350 0x7f801400b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0caf10 con 0x7f80140039b0 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f801400c2b0 con 0x7f8014005350 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f801400c4b0 con 0x7f8014004800 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f802bfff640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f8014003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f8030d9e640 1 -- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8014005350 msgr2=0x7f801400b9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.111+0000 7f8030d9e640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8014005350 0x7f801400b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f802bfff640 1 -- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8014004800 msgr2=0x7f8014004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:56.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f802b7fe640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8014004800 0x7f8014004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f802bfff640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8014004800 0x7f8014004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f802bfff640 1 -- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8014005350 msgr2=0x7f801400b9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f802bfff640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8014005350 0x7f801400b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f8030d9e640 1 -- 172.21.15.1:0/797027796 reap_dead start 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f802bfff640 1 -- 172.21.15.1:0/797027796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8014003520 con 0x7f80140039b0 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f802b7fe640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8014004800 0x7f8014004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.112+0000 7f802bfff640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f8014003e00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c240 tx=0x7f8014005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=887935c9f99241ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.113+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c007ce0 con 0x7f80140039b0 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.113+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f801c005ad0 con 0x7f80140039b0 2024-05-09T05:08:56.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.113+0000 7f802affd640 1 -- 172.21.15.1:0/797027796 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c007070 con 0x7f80140039b0 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.114+0000 7f803159f640 1 -- 172.21.15.1:0/797027796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 msgr2=0x7f8014003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.114+0000 7f803159f640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f8014003e00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c240 tx=0x7f8014005830 comp rx=0 tx=0).stop 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.114+0000 7f803159f640 1 -- 172.21.15.1:0/797027796 shutdown_connections 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.114+0000 7f803159f640 1 --2- 172.21.15.1:0/797027796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f8014003e00 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.114+0000 7f803159f640 1 -- 172.21.15.1:0/797027796 >> 172.21.15.1:0/797027796 conn(0x7f802c073fc0 msgr2=0x7f802c06d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.114+0000 7f803159f640 1 -- 172.21.15.1:0/797027796 shutdown_connections 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.114+0000 7f803159f640 1 -- 172.21.15.1:0/797027796 wait complete. 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.115+0000 7f803159f640 1 Processor -- start 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.116+0000 7f803159f640 1 -- start start 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f803159f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f802c0c8620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f803159f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f802c1524a0 0x7f802c0c40b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f803159f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f802c15ccd0 0x7f802c15d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f803159f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f802c0c3b40 con 0x7f80140039b0 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f803159f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f802c15a350 con 0x7f802c15ccd0 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f803159f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f802c0c3cc0 con 0x7f802c1524a0 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f802bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f802c0c8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f8030d9e640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f802c15ccd0 msgr2=0x7f802c15d0e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f802bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f802c0c8620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48972/0 (socket says 172.21.15.1:48972) 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f802bfff640 1 -- 172.21.15.1:0/704720921 learned_addr learned my addr 172.21.15.1:0/704720921 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.117+0000 7f8030d9e640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f802c15ccd0 0x7f802c15d0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f802b7fe640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f802c1524a0 0x7f802c0c40b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f802bfff640 1 -- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f802c1524a0 msgr2=0x7f802c0c40b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f802bfff640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f802c1524a0 0x7f802c0c40b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f802bfff640 1 -- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f802c15ccd0 msgr2=0x7f802c15d0e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:08:56.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f802bfff640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f802c15ccd0 0x7f802c15d0e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f802bfff640 1 -- 172.21.15.1:0/704720921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f802c152ff0 con 0x7f80140039b0 2024-05-09T05:08:56.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f802b7fe640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f802c1524a0 0x7f802c0c40b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:08:56.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f802bfff640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f802c0c8620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c240 tx=0x7f802c0c5430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c037040 con 0x7f80140039b0 2024-05-09T05:08:56.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f801c041070 con 0x7f80140039b0 2024-05-09T05:08:56.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f802c1532d0 con 0x7f80140039b0 2024-05-09T05:08:56.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f801c0016f0 con 0x7f80140039b0 2024-05-09T05:08:56.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.118+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f802c0c7e20 con 0x7f80140039b0 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.119+0000 7f803159f640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c1f2a80 0x7f802c150f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.119+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f802c152da0 con 0x7f802c1f2a80 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.120+0000 7f802b7fe640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c1f2a80 0x7f802c150f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.120+0000 7f802b7fe640 1 -- 172.21.15.1:0/704720921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f802c050870 con 0x7f802c1f2a80 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.120+0000 7f802b7fe640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c1f2a80 0x7f802c150f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f802c1508f0 tx=0x7f802000ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.120+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020017030 con 0x7f802c1f2a80 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.120+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f801c0049e0 con 0x7f80140039b0 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.121+0000 7f8028ff9640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8018076200 0x7f8018078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.121+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f801c0b6f80 con 0x7f80140039b0 2024-05-09T05:08:56.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.121+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f80200039b0 con 0x7f802c1f2a80 2024-05-09T05:08:56.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.121+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c1f2a80 msgr2=0x7f802c150f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.121+0000 7f8028ff9640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c1f2a80 0x7f802c150f00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f802c1508f0 tx=0x7f802000ec00 comp rx=0 tx=0).stop 2024-05-09T05:08:56.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.121+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c1f2a80 msgr2=0x7f802c150f00 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:56.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.121+0000 7f8028ff9640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c1f2a80 0x7f802c150f00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.121+0000 7f8030d9e640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8018076200 0x7f8018078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.122+0000 7f8030d9e640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8018076200 0x7f8018078710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f802c0690b0 tx=0x7f8024009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.131+0000 7f803159f640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c151440 0x7f802c151850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.131+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f802c151d90 con 0x7f802c151440 2024-05-09T05:08:56.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.131+0000 7f802b7fe640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c151440 0x7f802c151850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.132+0000 7f802b7fe640 1 -- 172.21.15.1:0/704720921 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f802c0c9b50 con 0x7f802c151440 2024-05-09T05:08:56.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.132+0000 7f802b7fe640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c151440 0x7f802c151850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8020015000 tx=0x7f8020005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.132+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8020005630 con 0x7f802c151440 2024-05-09T05:08:56.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.132+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f8020005630 con 0x7f802c151440 2024-05-09T05:08:56.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.132+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c151440 msgr2=0x7f802c151850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.132+0000 7f8028ff9640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c151440 0x7f802c151850 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8020015000 tx=0x7f8020005a10 comp rx=0 tx=0).stop 2024-05-09T05:08:56.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.132+0000 7f8028ff9640 1 -- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c151440 msgr2=0x7f802c151850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:56.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.132+0000 7f8028ff9640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f802c151440 0x7f802c151850 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 28, 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 42210, 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:08:56.134 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:08:56.135 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.136 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:08:56.137 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:56.138 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:08:56.141 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:56.142 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.143 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.144 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:08:56.145 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:08:56.146 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:08:56.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8018076200 msgr2=0x7f8018078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8018076200 0x7f8018078710 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f802c0690b0 tx=0x7f8024009190 comp rx=0 tx=0).stop 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 msgr2=0x7f802c0c8620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f802c0c8620 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f801c00c240 tx=0x7f802c0c5430 comp rx=0 tx=0).stop 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f8030d9e640 1 -- 172.21.15.1:0/704720921 reap_dead start 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 shutdown_connections 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 --2- 172.21.15.1:0/704720921 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80140039b0 0x7f802c0c8620 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 >> 172.21.15.1:0/704720921 conn(0x7f802c073fc0 msgr2=0x7f802c158350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 shutdown_connections 2024-05-09T05:08:56.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.133+0000 7f803159f640 1 -- 172.21.15.1:0/704720921 wait complete. 2024-05-09T05:08:56.148 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:08:56.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:08:56.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:08:56.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.275+0000 7febada61640 1 Processor -- start 2024-05-09T05:08:56.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.276+0000 7febada61640 1 -- start start 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.277+0000 7febada61640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feba8073410 0x7feba8154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7febada61640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 0x7feba806cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7febada61640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7feba815b560 con 0x7feba80748a0 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7febada61640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7feba80c9070 con 0x7feba8073410 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7febada61640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7feba8073da0 con 0x7feba806c6f0 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba6ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7feba8073410 0x7feba8154a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba67fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7feba80748a0 0x7feba80c9500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46472/0 (socket says 172.21.15.1:46472) 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba67fc640 1 -- 172.21.15.1:0/4018890978 learned_addr learned my addr 172.21.15.1:0/4018890978 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba77fe640 1 --2- 172.21.15.1:0/4018890978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 0x7feba806cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba6ffd640 1 -- 172.21.15.1:0/4018890978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 msgr2=0x7feba806cb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba6ffd640 1 --2- 172.21.15.1:0/4018890978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 0x7feba806cb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba6ffd640 1 -- 172.21.15.1:0/4018890978 >> v1:172.21.15.1:6789/0 conn(0x7feba80748a0 legacy=0x7feba80c9500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba6ffd640 1 -- 172.21.15.1:0/4018890978 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba8074080 con 0x7feba8073410 2024-05-09T05:08:56.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.278+0000 7feba5ffb640 1 -- 172.21.15.1:0/4018890978 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3505308561 0 0) 0x7feba815b560 con 0x7feba80748a0 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.279+0000 7feba6ffd640 1 --2- 172.21.15.1:0/4018890978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 0x7feba8154a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb980097a0 tx=0x7feb9802a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e30feb6e481712d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.279+0000 7feba5ffb640 1 -- 172.21.15.1:0/4018890978 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98037070 con 0x7feba8073410 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.279+0000 7feba5ffb640 1 -- 172.21.15.1:0/4018890978 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb98032440 con 0x7feba8073410 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.279+0000 7feba5ffb640 1 -- 172.21.15.1:0/4018890978 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98032810 con 0x7feba8073410 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.280+0000 7febada61640 1 -- 172.21.15.1:0/4018890978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 msgr2=0x7feba8154a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.280+0000 7febada61640 1 --2- 172.21.15.1:0/4018890978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 0x7feba8154a60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7feb980097a0 tx=0x7feb9802a180 comp rx=0 tx=0).stop 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.280+0000 7febada61640 1 -- 172.21.15.1:0/4018890978 shutdown_connections 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.280+0000 7febada61640 1 --2- 172.21.15.1:0/4018890978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 0x7feba806cb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.280+0000 7febada61640 1 --2- 172.21.15.1:0/4018890978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 0x7feba8154a60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.280+0000 7febada61640 1 -- 172.21.15.1:0/4018890978 >> 172.21.15.1:0/4018890978 conn(0x7feba80cdcd0 msgr2=0x7feba80ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.280+0000 7febada61640 1 -- 172.21.15.1:0/4018890978 shutdown_connections 2024-05-09T05:08:56.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.280+0000 7febada61640 1 -- 172.21.15.1:0/4018890978 wait complete. 2024-05-09T05:08:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.282+0000 7febada61640 1 Processor -- start 2024-05-09T05:08:56.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.282+0000 7febada61640 1 -- start start 2024-05-09T05:08:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.283+0000 7febada61640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 0x7feba80c61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.283+0000 7febada61640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 0x7feba814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.283+0000 7febada61640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba80748a0 0x7feba814f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.283+0000 7febada61640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7feba80c9d90 con 0x7feba8073410 2024-05-09T05:08:56.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.283+0000 7febada61640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7feba80c9c10 con 0x7feba80748a0 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.283+0000 7febada61640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7feba815b1b0 con 0x7feba806c6f0 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba77fe640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba80748a0 msgr2=0x7feba814f100 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba77fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba80748a0 0x7feba814f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 0x7feba814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 0x7feba814ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48992/0 (socket says 172.21.15.1:48992) 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 -- 172.21.15.1:0/448216432 learned_addr learned my addr 172.21.15.1:0/448216432 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba6ffd640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 0x7feba80c61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 -- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 msgr2=0x7feba80c61c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 0x7feba80c61c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 -- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba80748a0 msgr2=0x7feba814f100 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7feba80748a0 0x7feba814f100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 -- 172.21.15.1:0/448216432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feba80c6700 con 0x7feba8073410 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba6ffd640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba806c6f0 0x7feba80c61c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:08:56.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feba67fc640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 0x7feba814ff80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c410 tx=0x7feb9c00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c0074e0 con 0x7feba8073410 2024-05-09T05:08:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7feb9c0047e0 con 0x7feba8073410 2024-05-09T05:08:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb9c004bd0 con 0x7feba8073410 2024-05-09T05:08:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.284+0000 7febada61640 1 -- 172.21.15.1:0/448216432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feba8150bb0 con 0x7feba8073410 2024-05-09T05:08:56.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.285+0000 7febada61640 1 -- 172.21.15.1:0/448216432 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feba80c4a70 con 0x7feba8073410 2024-05-09T05:08:56.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.286+0000 7febada61640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba80738e0 0x7feba80cbbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.286+0000 7febada61640 1 -- 172.21.15.1:0/448216432 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7feba80b30c0 con 0x7feba80738e0 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.286+0000 7feba6ffd640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba80738e0 0x7feba80cbbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.286+0000 7feba6ffd640 1 -- 172.21.15.1:0/448216432 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7feba81533c0 con 0x7feba80738e0 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.286+0000 7feba6ffd640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba80738e0 0x7feba80cbbd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feba80ca730 tx=0x7feb980046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.287+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98042070 con 0x7feba80738e0 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.287+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7feb9c011070 con 0x7feba8073410 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.287+0000 7feb8ffff640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7feb940761d0 0x7feb940786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.288+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7feb9c097290 con 0x7feba8073410 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.288+0000 7feba77fe640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7feb940761d0 0x7feb940786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.288+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7feb98032440 con 0x7feba80738e0 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.288+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba80738e0 msgr2=0x7feba80cbbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.288+0000 7feb8ffff640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba80738e0 0x7feba80cbbd0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7feba80ca730 tx=0x7feb980046c0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.288+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba80738e0 msgr2=0x7feba80cbbd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:56.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.288+0000 7feb8ffff640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba80738e0 0x7feba80cbbd0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.288+0000 7feba77fe640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7feb940761d0 0x7feb940786e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feba80c8c20 tx=0x7feba000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.296+0000 7febada61640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba814f640 0x7feba80cc110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:08:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.296+0000 7febada61640 1 -- 172.21.15.1:0/448216432 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7feba80c6bd0 con 0x7feba814f640 2024-05-09T05:08:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.297+0000 7feba6ffd640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba814f640 0x7feba80cc110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:08:56.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.297+0000 7feba6ffd640 1 -- 172.21.15.1:0/448216432 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7feba8050870 con 0x7feba814f640 2024-05-09T05:08:56.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.297+0000 7feba6ffd640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba814f640 0x7feba80cc110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb9802a1c0 tx=0x7feb98008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:08:56.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7feb98037040 con 0x7feba814f640 2024-05-09T05:08:56.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5049 (secure 0 0 0) 0x7feb9803e870 con 0x7feba814f640 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba814f640 msgr2=0x7feba80cc110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7feb8ffff640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba814f640 0x7feba80cc110 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb9802a1c0 tx=0x7feb98008030 comp rx=0 tx=0).stop 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7feb8ffff640 1 -- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba814f640 msgr2=0x7feba80cc110 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7feb8ffff640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7feba814f640 0x7feba80cc110 secure :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7feb9802a1c0 tx=0x7feb98008030 comp rx=0 tx=0).stop 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 28, 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 94296, 2024-05-09T05:08:56.299 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:56.300 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:56.301 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:08:56.302 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:08:56.303 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:56.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:08:56.305 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.307 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.308 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:08:56.309 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:08:56.310 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:08:56.311 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:08:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:08:56.312 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:08:56.312 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:08:56.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7febada61640 1 -- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7feb940761d0 msgr2=0x7feb940786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7febada61640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7feb940761d0 0x7feb940786e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7feba80c8c20 tx=0x7feba000b040 comp rx=0 tx=0).stop 2024-05-09T05:08:56.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7febada61640 1 -- 172.21.15.1:0/448216432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 msgr2=0x7feba814ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:08:56.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7febada61640 1 --2- 172.21.15.1:0/448216432 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7feba8073410 0x7feba814ff80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7feb9c00c410 tx=0x7feb9c00c8d0 comp rx=0 tx=0).stop 2024-05-09T05:08:56.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7feba77fe640 1 -- 172.21.15.1:0/448216432 reap_dead start 2024-05-09T05:08:56.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7feba77fe640 1 -- 172.21.15.1:0/448216432 reap_dead start 2024-05-09T05:08:56.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7febada61640 1 -- 172.21.15.1:0/448216432 shutdown_connections 2024-05-09T05:08:56.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.298+0000 7febada61640 1 -- 172.21.15.1:0/448216432 >> 172.21.15.1:0/448216432 conn(0x7feba80cdcd0 msgr2=0x7feba80ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:08:56.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.299+0000 7febada61640 1 -- 172.21.15.1:0/448216432 shutdown_connections 2024-05-09T05:08:56.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:08:56.299+0000 7febada61640 1 -- 172.21.15.1:0/448216432 wait complete. 2024-05-09T05:08:56.314 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:08:56.314 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:09:02.057 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:09:02.063 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:09:16.315 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:09:16.315 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-05-09T05:09:16.315 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T05:09:16.315 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T05:09:16.318 INFO:tasks.ceph.mon.b:Started 2024-05-09T05:09:16.318 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:09:16.318 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:09:16.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.443+0000 7f8efb07e640 1 Processor -- start 2024-05-09T05:09:16.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.443+0000 7f8efb07e640 1 -- start start 2024-05-09T05:09:16.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8efb07e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8ef40cb1c0 0x7f8ef415d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:16.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8efb07e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ef40c5e00 con 0x7f8ef4072120 2024-05-09T05:09:16.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8efb07e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8ef40c60e0 con 0x7f8ef4156a80 2024-05-09T05:09:16.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8efb07e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8ef415d710 con 0x7f8ef40cb1c0 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef8e1a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8ef4072120 0x7f8ef4070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58806/0 (socket says 172.21.15.1:58806) 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef8e1a640 1 -- 172.21.15.1:0/3454833589 learned_addr learned my addr 172.21.15.1:0/3454833589 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef961b640 1 --2- 172.21.15.1:0/3454833589 >> v2:172.21.15.196:3300/0 conn(0x7f8ef40cb1c0 0x7f8ef415d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef961b640 1 -- 172.21.15.1:0/3454833589 >> v2:172.21.15.196:3300/0 conn(0x7f8ef40cb1c0 msgr2=0x7f8ef415d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef961b640 1 -- 172.21.15.1:0/3454833589 >> v2:172.21.15.196:3300/0 conn(0x7f8ef40cb1c0 msgr2=0x7f8ef415d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef961b640 1 --2- 172.21.15.1:0/3454833589 >> v2:172.21.15.196:3300/0 conn(0x7f8ef40cb1c0 0x7f8ef415d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3952875908 0 0) 0x7f8ef40c5e00 con 0x7f8ef4072120 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef961b640 1 --2- 172.21.15.1:0/3454833589 >> v2:172.21.15.196:3300/0 conn(0x7f8ef40cb1c0 0x7f8ef415d1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8ee4003520 con 0x7f8ef4072120 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1233516105 0 0) 0x7f8ee4003520 con 0x7f8ef4072120 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 >> v2:172.21.15.196:3300/0 conn(0x7f8ef40cb1c0 msgr2=0x7f8ef415d1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 --2- 172.21.15.1:0/3454833589 >> v2:172.21.15.196:3300/0 conn(0x7f8ef40cb1c0 0x7f8ef415d1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 >> v1:172.21.15.196:6789/0 conn(0x7f8ef4156a80 legacy=0x7f8ef40caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:16.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef40c63c0 con 0x7f8ef4072120 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f8ee80016f0 con 0x7f8ef4072120 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.445+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 >> v1:172.21.15.1:6789/0 conn(0x7f8ef4072120 legacy=0x7f8ef4070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef37fe640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee4003aa0 0x7f8ee4003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef37fe640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee4004800 0x7f8ee4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef37fe640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ee4005350 0x7f8ee400ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee400c2b0 con 0x7f8ee4003aa0 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee400c460 con 0x7f8ee4004800 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8ee400c660 con 0x7f8ee4005350 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee4003aa0 0x7f8ee4003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef3fff640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee4004800 0x7f8ee4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef961b640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ee4005350 0x7f8ee400ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef8e1a640 1 -- 172.21.15.1:0/3454833589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ee4005350 msgr2=0x7f8ee400ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ee4005350 0x7f8ee400ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef8e1a640 1 -- 172.21.15.1:0/3454833589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee4004800 msgr2=0x7f8ee4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:16.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ee4004800 0x7f8ee4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef8e1a640 1 -- 172.21.15.1:0/3454833589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee4003520 con 0x7f8ee4003aa0 2024-05-09T05:09:16.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.446+0000 7f8ef961b640 1 -- 172.21.15.1:0/3454833589 reap_dead start 2024-05-09T05:09:16.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.447+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee4003aa0 0x7f8ee4003f10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800c160 tx=0x7f8ee4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c24e51da3e898ed0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:16.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.447+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8007c60 con 0x7f8ee4003aa0 2024-05-09T05:09:16.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.447+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8ee8005ad0 con 0x7f8ee4003aa0 2024-05-09T05:09:16.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.447+0000 7f8ef37fe640 1 -- 172.21.15.1:0/3454833589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ee8007070 con 0x7f8ee4003aa0 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.448+0000 7f8efb07e640 1 -- 172.21.15.1:0/3454833589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee4003aa0 msgr2=0x7f8ee4003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.448+0000 7f8efb07e640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee4003aa0 0x7f8ee4003f10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f8ee800c160 tx=0x7f8ee4005830 comp rx=0 tx=0).stop 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.448+0000 7f8efb07e640 1 -- 172.21.15.1:0/3454833589 shutdown_connections 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.448+0000 7f8efb07e640 1 --2- 172.21.15.1:0/3454833589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ee4003aa0 0x7f8ee4003f10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.448+0000 7f8efb07e640 1 -- 172.21.15.1:0/3454833589 >> 172.21.15.1:0/3454833589 conn(0x7f8ef40731d0 msgr2=0x7f8ef40cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.448+0000 7f8efb07e640 1 -- 172.21.15.1:0/3454833589 shutdown_connections 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.448+0000 7f8efb07e640 1 -- 172.21.15.1:0/3454833589 wait complete. 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.450+0000 7f8efb07e640 1 Processor -- start 2024-05-09T05:09:16.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.451+0000 7f8efb07e640 1 -- start start 2024-05-09T05:09:16.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.452+0000 7f8efb07e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 0x7f8ef4154910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:16.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.452+0000 7f8efb07e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ef4155d70 0x7f8ef40b11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:16.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.452+0000 7f8efb07e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ef4156180 0x7f8ef40b1700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:16.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.452+0000 7f8efb07e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef40ca620 con 0x7f8ef4156180 2024-05-09T05:09:16.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.452+0000 7f8efb07e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8ef40ca340 con 0x7f8ef4152d90 2024-05-09T05:09:16.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.452+0000 7f8efb07e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8ef4156e90 con 0x7f8ef4155d70 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ef4156180 0x7f8ef40b1700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ef4156180 0x7f8ef40b1700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:52522/0 (socket says 172.21.15.1:52522) 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 -- 172.21.15.1:0/3988587418 learned_addr learned my addr 172.21.15.1:0/3988587418 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 0x7f8ef4154910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef3fff640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ef4155d70 0x7f8ef40b11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef8e1a640 1 -- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 msgr2=0x7f8ef4154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef8e1a640 1 -- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 msgr2=0x7f8ef4154910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 0x7f8ef4154910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 0x7f8ef4154910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 -- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ef4155d70 msgr2=0x7f8ef40b11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ef4155d70 0x7f8ef40b11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 -- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 msgr2=0x7f8ef4154910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:09:16.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 0x7f8ef4154910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 -- 172.21.15.1:0/3988587418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ef41536e0 con 0x7f8ef4156180 2024-05-09T05:09:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef961b640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ef4156180 0x7f8ef40b1700 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009610 tx=0x7f8eec00f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.453+0000 7f8ef17fa640 1 -- 172.21.15.1:0/3988587418 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec003990 con 0x7f8ef4156180 2024-05-09T05:09:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.454+0000 7f8ef17fa640 1 -- 172.21.15.1:0/3988587418 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8eec003b50 con 0x7f8ef4156180 2024-05-09T05:09:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.454+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ef41531a0 con 0x7f8ef4156180 2024-05-09T05:09:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.454+0000 7f8ef17fa640 1 -- 172.21.15.1:0/3988587418 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8eec004030 con 0x7f8ef4156180 2024-05-09T05:09:16.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.454+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ef40b0400 con 0x7f8ef4156180 2024-05-09T05:09:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.455+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ef4050870 con 0x7f8ef4156180 2024-05-09T05:09:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.455+0000 7f8ef17fa640 1 -- 172.21.15.1:0/3988587418 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8eec0041d0 con 0x7f8ef4156180 2024-05-09T05:09:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.456+0000 7f8ef17fa640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8ed80762d0 0x7f8ed80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:16.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.456+0000 7f8ef17fa640 1 -- 172.21.15.1:0/3988587418 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8eec098090 con 0x7f8ef4156180 2024-05-09T05:09:16.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.456+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8ed80762d0 0x7f8ed80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:16.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.457+0000 7f8ef8e1a640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8ed80762d0 0x7f8ed80787e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8032000 tx=0x7f8ee8009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:16.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.461+0000 7f8ef17fa640 1 -- 172.21.15.1:0/3988587418 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8eec023060 con 0x7f8ef4156180 2024-05-09T05:09:16.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.821+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8ef40cba10 con 0x7f8ef4156180 2024-05-09T05:09:16.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.821+0000 7f8ef17fa640 1 -- 172.21.15.1:0/3988587418 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f8ef41531a0 con 0x7f8ef4156180 2024-05-09T05:09:16.823 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8ed80762d0 msgr2=0x7f8ed80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8ed80762d0 0x7f8ed80787e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f8ee8032000 tx=0x7f8ee8009030 comp rx=0 tx=0).stop 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ef4156180 msgr2=0x7f8ef40b1700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ef4156180 0x7f8ef40b1700 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f8eec009610 tx=0x7f8eec00f520 comp rx=0 tx=0).stop 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 shutdown_connections 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8ed80762d0 0x7f8ed80787e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8ef4156180 0x7f8ef40b1700 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8ef4155d70 0x7f8ef40b11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 --2- 172.21.15.1:0/3988587418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8ef4152d90 0x7f8ef4154910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 >> 172.21.15.1:0/3988587418 conn(0x7f8ef40731d0 msgr2=0x7f8ef41511e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 shutdown_connections 2024-05-09T05:09:16.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:16.824+0000 7f8efb07e640 1 -- 172.21.15.1:0/3988587418 wait complete. 2024-05-09T05:09:19.841 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:09:19.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.972+0000 7f80ef89a640 1 Processor -- start 2024-05-09T05:09:19.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.972+0000 7f80ef89a640 1 -- start start 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.973+0000 7f80ef89a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e806c910 0x7f80e80cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.973+0000 7f80ef89a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e80cccf0 con 0x7f80e8154ab0 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.973+0000 7f80ef89a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f80e80ccfd0 con 0x7f80e80c77a0 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.973+0000 7f80ef89a640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f80e8068d10 con 0x7f80e806c910 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80ece35640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f80e8154ab0 0x7f80e815e4d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41146/0 (socket says 172.21.15.1:41146) 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80ece35640 1 -- 172.21.15.1:0/4224453020 learned_addr learned my addr 172.21.15.1:0/4224453020 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80ed636640 1 --2- 172.21.15.1:0/4224453020 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e806c910 0x7f80e80cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80dffff640 1 -- 172.21.15.1:0/4224453020 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 769172336 0 0) 0x7f80e80cccf0 con 0x7f80e8154ab0 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80dffff640 1 -- 172.21.15.1:0/4224453020 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f80d4003520 con 0x7f80e8154ab0 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80dffff640 1 -- 172.21.15.1:0/4224453020 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1091233319 0 0) 0x7f80d4003520 con 0x7f80e8154ab0 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80ed636640 1 -- 172.21.15.1:0/4224453020 >> v1:172.21.15.196:6789/0 conn(0x7f80e80c77a0 legacy=0x7f80e8164fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80ed636640 1 -- 172.21.15.1:0/4224453020 >> v1:172.21.15.1:6789/0 conn(0x7f80e8154ab0 legacy=0x7f80e815e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:19.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.974+0000 7f80ed636640 1 -- 172.21.15.1:0/4224453020 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e80cd2b0 con 0x7f80e806c910 2024-05-09T05:09:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.975+0000 7f80ed636640 1 --2- 172.21.15.1:0/4224453020 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e806c910 0x7f80e80cbef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f80e40097d0 tx=0x7f80e402a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=adbef0d16e6f9078 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.975+0000 7f80dffff640 1 -- 172.21.15.1:0/4224453020 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4037070 con 0x7f80e806c910 2024-05-09T05:09:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.976+0000 7f80dffff640 1 -- 172.21.15.1:0/4224453020 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e402ac30 con 0x7f80e806c910 2024-05-09T05:09:19.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.976+0000 7f80dffff640 1 -- 172.21.15.1:0/4224453020 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80d80016f0 con 0x7f80e806c910 2024-05-09T05:09:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.976+0000 7f80ef89a640 1 -- 172.21.15.1:0/4224453020 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e806c910 msgr2=0x7f80e80cbef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.976+0000 7f80ef89a640 1 --2- 172.21.15.1:0/4224453020 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e806c910 0x7f80e80cbef0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f80e40097d0 tx=0x7f80e402a1a0 comp rx=0 tx=0).stop 2024-05-09T05:09:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.976+0000 7f80ef89a640 1 -- 172.21.15.1:0/4224453020 shutdown_connections 2024-05-09T05:09:19.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.976+0000 7f80ef89a640 1 --2- 172.21.15.1:0/4224453020 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e806c910 0x7f80e80cbef0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.976+0000 7f80ef89a640 1 -- 172.21.15.1:0/4224453020 >> 172.21.15.1:0/4224453020 conn(0x7f80e80ce110 msgr2=0x7f80e80ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.977+0000 7f80ef89a640 1 -- 172.21.15.1:0/4224453020 shutdown_connections 2024-05-09T05:09:19.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.977+0000 7f80ef89a640 1 -- 172.21.15.1:0/4224453020 wait complete. 2024-05-09T05:09:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.979+0000 7f80ef89a640 1 Processor -- start 2024-05-09T05:09:19.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.979+0000 7f80ef89a640 1 -- start start 2024-05-09T05:09:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ef89a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80e806c910 0x7f80e81f1450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:19.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ef89a640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 0x7f80e8157e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ef89a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e8154ab0 0x7f80e8153470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ef89a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f80e815dfb0 con 0x7f80e806c910 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ef89a640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f80e815ebe0 con 0x7f80e80c77a0 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ef89a640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f80e80cc430 con 0x7f80e8154ab0 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80e806c910 0x7f80e81f1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80e806c910 0x7f80e81f1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34366/0 (socket says 172.21.15.1:34366) 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 -- 172.21.15.1:0/954757359 learned_addr learned my addr 172.21.15.1:0/954757359 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ede37640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e8154ab0 0x7f80e8153470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ece35640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 0x7f80e8157e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ece35640 1 -- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 msgr2=0x7f80e8157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ece35640 1 -- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 msgr2=0x7f80e8157e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ece35640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 0x7f80e8157e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ece35640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 0x7f80e8157e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 -- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e8154ab0 msgr2=0x7f80e8153470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:19.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e8154ab0 0x7f80e8153470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 -- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 msgr2=0x7f80e8157e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 0x7f80e8157e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 -- 172.21.15.1:0/954757359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80e80bb460 con 0x7f80e806c910 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.981+0000 7f80ed636640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80e806c910 0x7f80e81f1450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80e40098f0 tx=0x7f80e40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.982+0000 7f80ddffb640 1 -- 172.21.15.1:0/954757359 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4042020 con 0x7f80e806c910 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.982+0000 7f80ddffb640 1 -- 172.21.15.1:0/954757359 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f80e4003c40 con 0x7f80e806c910 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.982+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80e80baea0 con 0x7f80e806c910 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.982+0000 7f80ddffb640 1 -- 172.21.15.1:0/954757359 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f80e4037040 con 0x7f80e806c910 2024-05-09T05:09:19.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.982+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80e81f0c50 con 0x7f80e806c910 2024-05-09T05:09:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.983+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80e81f0a40 con 0x7f80e806c910 2024-05-09T05:09:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.984+0000 7f80ddffb640 1 -- 172.21.15.1:0/954757359 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f80e4045020 con 0x7f80e806c910 2024-05-09T05:09:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.985+0000 7f80ddffb640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f80cc0762d0 0x7f80cc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:19.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.985+0000 7f80ddffb640 1 -- 172.21.15.1:0/954757359 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f80e40b7bd0 con 0x7f80e806c910 2024-05-09T05:09:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.985+0000 7f80ece35640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f80cc0762d0 0x7f80cc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:19.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.986+0000 7f80ece35640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f80cc0762d0 0x7f80cc0787e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f80e8157980 tx=0x7f80d8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:19.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:19.990+0000 7f80ddffb640 1 -- 172.21.15.1:0/954757359 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f80e404c020 con 0x7f80e806c910 2024-05-09T05:09:20.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.317+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f80e8050870 con 0x7f80e806c910 2024-05-09T05:09:20.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.317+0000 7f80ddffb640 1 -- 172.21.15.1:0/954757359 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f80e4081430 con 0x7f80e806c910 2024-05-09T05:09:20.319 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":28,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:09:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f80cc0762d0 msgr2=0x7f80cc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f80cc0762d0 0x7f80cc0787e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f80e8157980 tx=0x7f80d8009040 comp rx=0 tx=0).stop 2024-05-09T05:09:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80e806c910 msgr2=0x7f80e81f1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:20.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80e806c910 0x7f80e81f1450 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f80e40098f0 tx=0x7f80e40046c0 comp rx=0 tx=0).stop 2024-05-09T05:09:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 shutdown_connections 2024-05-09T05:09:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f80cc0762d0 0x7f80cc0787e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80e80c77a0 0x7f80e8157e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80e8154ab0 0x7f80e8153470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 --2- 172.21.15.1:0/954757359 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80e806c910 0x7f80e81f1450 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 >> 172.21.15.1:0/954757359 conn(0x7f80e80ce110 msgr2=0x7f80e8159560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 shutdown_connections 2024-05-09T05:09:20.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:20.320+0000 7f80ef89a640 1 -- 172.21.15.1:0/954757359 wait complete. 2024-05-09T05:09:23.333 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:09:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.464+0000 7fc637c49640 1 Processor -- start 2024-05-09T05:09:23.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.464+0000 7fc637c49640 1 -- start start 2024-05-09T05:09:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.465+0000 7fc637c49640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 0x7fc6300d01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.465+0000 7fc637c49640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6300c55f0 con 0x7fc63015b170 2024-05-09T05:09:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.465+0000 7fc637c49640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6300cb830 con 0x7fc630154680 2024-05-09T05:09:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.465+0000 7fc637c49640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc6300cb590 con 0x7fc6300cfdc0 2024-05-09T05:09:23.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.465+0000 7fc6359e5640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc63015b170 0x7fc6300c4ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41162/0 (socket says 172.21.15.1:41162) 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.465+0000 7fc6359e5640 1 -- 172.21.15.1:0/2221257104 learned_addr learned my addr 172.21.15.1:0/2221257104 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6361e6640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 0x7fc6300d01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4008358658 0 0) 0x7fc6300c55f0 con 0x7fc63015b170 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc61c003520 con 0x7fc63015b170 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3554263790 0 0) 0x7fc61c003520 con 0x7fc63015b170 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 msgr2=0x7fc6300d01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 0x7fc6300d01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 >> v1:172.21.15.196:6789/0 conn(0x7fc630154680 legacy=0x7fc6300cae80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6300cc1b0 con 0x7fc63015b170 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fc62c0016f0 con 0x7fc63015b170 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 >> v1:172.21.15.1:6789/0 conn(0x7fc63015b170 legacy=0x7fc6300c4ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 0x7fc61c003a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61c004000 0x7fc61c004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc61c004d40 0x7fc61c0051f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc6300cb590 con 0x7fc61c004000 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc61c00bc90 con 0x7fc61c004d40 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc61c00be90 con 0x7fc6300cfdc0 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6359e5640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 0x7fc61c003a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.466+0000 7fc6351e4640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61c004000 0x7fc61c004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.467+0000 7fc6361e6640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc61c004d40 0x7fc61c0051f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.467+0000 7fc6351e4640 1 -- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 msgr2=0x7fc61c003a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.467+0000 7fc6351e4640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 0x7fc61c003a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.467+0000 7fc6351e4640 1 -- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc61c004d40 msgr2=0x7fc61c0051f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.467+0000 7fc6351e4640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc61c004d40 0x7fc61c0051f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.467+0000 7fc6351e4640 1 -- 172.21.15.1:0/2221257104 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc61c003520 con 0x7fc61c004000 2024-05-09T05:09:23.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.467+0000 7fc6359e5640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300cfdc0 0x7fc61c003a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:23.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.468+0000 7fc6351e4640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61c004000 0x7fc61c004800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc620001700 tx=0x7fc61c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a3ba118b8adcfcb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:23.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.468+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc620001350 con 0x7fc61c004000 2024-05-09T05:09:23.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.468+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6200052d0 con 0x7fc61c004000 2024-05-09T05:09:23.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.468+0000 7fc6349e3640 1 -- 172.21.15.1:0/2221257104 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62002ab90 con 0x7fc61c004000 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.469+0000 7fc637c49640 1 -- 172.21.15.1:0/2221257104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61c004000 msgr2=0x7fc61c004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.469+0000 7fc637c49640 1 --2- 172.21.15.1:0/2221257104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc61c004000 0x7fc61c004800 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc620001700 tx=0x7fc61c005830 comp rx=0 tx=0).stop 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.469+0000 7fc6361e6640 1 -- 172.21.15.1:0/2221257104 reap_dead start 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.469+0000 7fc637c49640 1 -- 172.21.15.1:0/2221257104 shutdown_connections 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.469+0000 7fc637c49640 1 -- 172.21.15.1:0/2221257104 >> 172.21.15.1:0/2221257104 conn(0x7fc630068830 msgr2=0x7fc630068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.469+0000 7fc637c49640 1 -- 172.21.15.1:0/2221257104 shutdown_connections 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.469+0000 7fc637c49640 1 -- 172.21.15.1:0/2221257104 wait complete. 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.471+0000 7fc637c49640 1 Processor -- start 2024-05-09T05:09:23.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.471+0000 7fc637c49640 1 -- start start 2024-05-09T05:09:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.472+0000 7fc637c49640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6301e6490 0x7fc630151750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.472+0000 7fc637c49640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300c9770 0x7fc6301f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.472+0000 7fc637c49640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300c9b80 0x7fc6301f22b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.472+0000 7fc637c49640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc6300cca50 con 0x7fc6300c9b80 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.472+0000 7fc637c49640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc6300cc770 con 0x7fc6301e6490 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.472+0000 7fc637c49640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc6300cd250 con 0x7fc6300c9770 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.472+0000 7fc6361e6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300c9b80 0x7fc6301f22b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6359e5640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6301e6490 0x7fc630151750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6361e6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300c9b80 0x7fc6301f22b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34386/0 (socket says 172.21.15.1:34386) 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6361e6640 1 -- 172.21.15.1:0/1028881895 learned_addr learned my addr 172.21.15.1:0/1028881895 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6351e4640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300c9770 0x7fc6301f1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6361e6640 1 -- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300c9770 msgr2=0x7fc6301f1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6361e6640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300c9770 0x7fc6301f1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6361e6640 1 -- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6301e6490 msgr2=0x7fc630151750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6361e6640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6301e6490 0x7fc630151750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6361e6640 1 -- 172.21.15.1:0/1028881895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6301f0900 con 0x7fc6300c9b80 2024-05-09T05:09:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6361e6640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300c9b80 0x7fc6301f22b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc63006fac0 tx=0x7fc62800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6267fc640 1 -- 172.21.15.1:0/1028881895 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc62c007d10 con 0x7fc6300c9b80 2024-05-09T05:09:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6267fc640 1 -- 172.21.15.1:0/1028881895 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc62800fde0 con 0x7fc6300c9b80 2024-05-09T05:09:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc630152070 con 0x7fc6300c9b80 2024-05-09T05:09:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc6267fc640 1 -- 172.21.15.1:0/1028881895 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc628015670 con 0x7fc6300c9b80 2024-05-09T05:09:23.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.473+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6301f0ea0 con 0x7fc6300c9b80 2024-05-09T05:09:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.474+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc630050870 con 0x7fc6300c9b80 2024-05-09T05:09:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.476+0000 7fc6267fc640 1 -- 172.21.15.1:0/1028881895 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc628015810 con 0x7fc6300c9b80 2024-05-09T05:09:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.476+0000 7fc6267fc640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6140762d0 0x7fc6140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.476+0000 7fc6267fc640 1 -- 172.21.15.1:0/1028881895 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc628098220 con 0x7fc6300c9b80 2024-05-09T05:09:23.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.477+0000 7fc6359e5640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6140762d0 0x7fc6140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.477+0000 7fc6359e5640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6140762d0 0x7fc6140787e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc62c001eb0 tx=0x7fc62c0078f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:23.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.482+0000 7fc6267fc640 1 -- 172.21.15.1:0/1028881895 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc628024020 con 0x7fc6300c9b80 2024-05-09T05:09:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.829+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc6300cba30 con 0x7fc6300c9b80 2024-05-09T05:09:23.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.830+0000 7fc6267fc640 1 -- 172.21.15.1:0/1028881895 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fc628019030 con 0x7fc6300c9b80 2024-05-09T05:09:23.831 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:09:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6140762d0 msgr2=0x7fc6140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6140762d0 0x7fc6140787e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fc62c001eb0 tx=0x7fc62c0078f0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300c9b80 msgr2=0x7fc6301f22b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300c9b80 0x7fc6301f22b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc63006fac0 tx=0x7fc62800f580 comp rx=0 tx=0).stop 2024-05-09T05:09:23.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 shutdown_connections 2024-05-09T05:09:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6140762d0 0x7fc6140787e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6300c9b80 0x7fc6301f22b0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6300c9770 0x7fc6301f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 --2- 172.21.15.1:0/1028881895 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6301e6490 0x7fc630151750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 >> 172.21.15.1:0/1028881895 conn(0x7fc630068830 msgr2=0x7fc630153200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 shutdown_connections 2024-05-09T05:09:23.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.832+0000 7fc637c49640 1 -- 172.21.15.1:0/1028881895 wait complete. 2024-05-09T05:09:23.844 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:09:23.845 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:09:23.845 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:09:23.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.974+0000 7f964656c640 1 Processor -- start 2024-05-09T05:09:23.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.974+0000 7f964656c640 1 -- start start 2024-05-09T05:09:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.975+0000 7f964656c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96400c8210 0x7f96400c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.975+0000 7f964656c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f96400cc1f0 0x7f96400cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.975+0000 7f964656c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9640153ba0 con 0x7f964014ff20 2024-05-09T05:09:23.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.975+0000 7f964656c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9640159980 con 0x7f96400c8210 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.975+0000 7f964656c640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f96400c62b0 con 0x7f96400cc1f0 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f963ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f96400c8210 0x7f96400c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f963f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f964014ff20 0x7f9640159e10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41178/0 (socket says 172.21.15.1:41178) 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f963f7fe640 1 -- 172.21.15.1:0/1138751644 learned_addr learned my addr 172.21.15.1:0/1138751644 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f9644b09640 1 --2- 172.21.15.1:0/1138751644 >> v2:172.21.15.196:3300/0 conn(0x7f96400cc1f0 0x7f96400cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f963ffff640 1 -- 172.21.15.1:0/1138751644 >> v2:172.21.15.196:3300/0 conn(0x7f96400cc1f0 msgr2=0x7f96400cc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f963ffff640 1 --2- 172.21.15.1:0/1138751644 >> v2:172.21.15.196:3300/0 conn(0x7f96400cc1f0 0x7f96400cc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f963ffff640 1 -- 172.21.15.1:0/1138751644 >> v1:172.21.15.1:6789/0 conn(0x7f964014ff20 legacy=0x7f9640159e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f963ffff640 1 -- 172.21.15.1:0/1138751644 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96400c6590 con 0x7f96400c8210 2024-05-09T05:09:23.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.976+0000 7f963effd640 1 -- 172.21.15.1:0/1138751644 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3317949037 0 0) 0x7f9640153ba0 con 0x7f964014ff20 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.977+0000 7f963ffff640 1 --2- 172.21.15.1:0/1138751644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c8210 0x7f96400c5600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9640159b00 tx=0x7f963002a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7b5055fbc177df9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.977+0000 7f963effd640 1 -- 172.21.15.1:0/1138751644 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630037070 con 0x7f96400c8210 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.977+0000 7f963effd640 1 -- 172.21.15.1:0/1138751644 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9630032440 con 0x7f96400c8210 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.977+0000 7f963effd640 1 -- 172.21.15.1:0/1138751644 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630032810 con 0x7f96400c8210 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.978+0000 7f964656c640 1 -- 172.21.15.1:0/1138751644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c8210 msgr2=0x7f96400c5600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.978+0000 7f964656c640 1 --2- 172.21.15.1:0/1138751644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c8210 0x7f96400c5600 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f9640159b00 tx=0x7f963002a1a0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.978+0000 7f964656c640 1 -- 172.21.15.1:0/1138751644 shutdown_connections 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.978+0000 7f964656c640 1 --2- 172.21.15.1:0/1138751644 >> v2:172.21.15.196:3300/0 conn(0x7f96400cc1f0 0x7f96400cc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.978+0000 7f964656c640 1 --2- 172.21.15.1:0/1138751644 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c8210 0x7f96400c5600 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.978+0000 7f964656c640 1 -- 172.21.15.1:0/1138751644 >> 172.21.15.1:0/1138751644 conn(0x7f96400ce180 msgr2=0x7f96400ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.978+0000 7f964656c640 1 -- 172.21.15.1:0/1138751644 shutdown_connections 2024-05-09T05:09:23.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.978+0000 7f964656c640 1 -- 172.21.15.1:0/1138751644 wait complete. 2024-05-09T05:09:23.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.980+0000 7f964656c640 1 Processor -- start 2024-05-09T05:09:23.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.980+0000 7f964656c640 1 -- start start 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f964656c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f96400c8210 0x7f96401f02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f964656c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96400cc1f0 0x7f9640155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f964656c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f964014ff20 0x7f9640145c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f964656c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9640150330 con 0x7f964014ff20 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f964656c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9640159610 con 0x7f96400cc1f0 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f964656c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f96400c5f50 con 0x7f96400c8210 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f9644b09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f964014ff20 0x7f9640145c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f9644b09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f964014ff20 0x7f9640145c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34410/0 (socket says 172.21.15.1:34410) 2024-05-09T05:09:23.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f9644b09640 1 -- 172.21.15.1:0/711057274 learned_addr learned my addr 172.21.15.1:0/711057274 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f963ffff640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f96400c8210 0x7f96401f02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.982+0000 7f963f7fe640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96400cc1f0 0x7f9640155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f9644b09640 1 -- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f96400c8210 msgr2=0x7f96401f02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f9644b09640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f96400c8210 0x7f96401f02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f9644b09640 1 -- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96400cc1f0 msgr2=0x7f9640155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f9644b09640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f96400cc1f0 0x7f9640155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f9644b09640 1 -- 172.21.15.1:0/711057274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9640156390 con 0x7f964014ff20 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f9644b09640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f964014ff20 0x7f9640145c00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9638009d30 tx=0x7f963800c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9638018020 con 0x7f964014ff20 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9638014480 con 0x7f964014ff20 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9640156bc0 con 0x7f964014ff20 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9638014870 con 0x7f964014ff20 2024-05-09T05:09:23.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.983+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96401568e0 con 0x7f964014ff20 2024-05-09T05:09:23.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.984+0000 7f964656c640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c5070 0x7f96401f3bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.984+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f96401f40f0 con 0x7f96400c5070 2024-05-09T05:09:23.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.985+0000 7f963ffff640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c5070 0x7f96401f3bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.985+0000 7f963ffff640 1 -- 172.21.15.1:0/711057274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9640156650 con 0x7f96400c5070 2024-05-09T05:09:23.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.985+0000 7f963ffff640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c5070 0x7f96401f3bb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f96400721e0 tx=0x7f96300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:23.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.985+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630042070 con 0x7f96400c5070 2024-05-09T05:09:23.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.985+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9638005d10 con 0x7f964014ff20 2024-05-09T05:09:23.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.986+0000 7f963cff9640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f962c0761d0 0x7f962c0786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.986+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9638097fb0 con 0x7f964014ff20 2024-05-09T05:09:23.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.986+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f963003e660 con 0x7f96400c5070 2024-05-09T05:09:23.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.986+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c5070 msgr2=0x7f96401f3bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.986+0000 7f963cff9640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c5070 0x7f96401f3bb0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f96400721e0 tx=0x7f96300046c0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.986+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c5070 msgr2=0x7f96401f3bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:23.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.986+0000 7f963cff9640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96400c5070 0x7f96401f3bb0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:23.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.986+0000 7f963f7fe640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f962c0761d0 0x7f962c0786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.987+0000 7f963f7fe640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f962c0761d0 0x7f962c0786e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96400c9a20 tx=0x7f9634007de0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:23.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.994+0000 7f964656c640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96401f4470 0x7f964013f780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:23.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.994+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9640050870 con 0x7f96401f4470 2024-05-09T05:09:23.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.994+0000 7f963ffff640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96401f4470 0x7f964013f780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:23.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.994+0000 7f963ffff640 1 -- 172.21.15.1:0/711057274 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f96400c6ab0 con 0x7f96401f4470 2024-05-09T05:09:23.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.994+0000 7f963ffff640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96401f4470 0x7f964013f780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9630000c00 tx=0x7f9630008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:23.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.994+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9630037040 con 0x7f96401f4470 2024-05-09T05:09:23.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4892 (secure 0 0 0) 0x7f963003edb0 con 0x7f96401f4470 2024-05-09T05:09:23.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96401f4470 msgr2=0x7f964013f780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:23.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f963cff9640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96401f4470 0x7f964013f780 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9630000c00 tx=0x7f9630008030 comp rx=0 tx=0).stop 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f963cff9640 1 -- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96401f4470 msgr2=0x7f964013f780 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f963cff9640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f96401f4470 0x7f964013f780 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9630000c00 tx=0x7f9630008030 comp rx=0 tx=0).stop 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 70073, 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:09:23.996 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:23.997 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:23.998 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:23.999 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:24.000 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:24.001 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:09:24.003 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:09:24.005 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:09:24.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.007 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:09:24.008 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:09:24.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f962c0761d0 msgr2=0x7f962c0786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f962c0761d0 0x7f962c0786e0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f96400c9a20 tx=0x7f9634007de0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f964014ff20 msgr2=0x7f9640145c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f964014ff20 0x7f9640145c00 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f9638009d30 tx=0x7f963800c490 comp rx=0 tx=0).stop 2024-05-09T05:09:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f9644b09640 1 -- 172.21.15.1:0/711057274 reap_dead start 2024-05-09T05:09:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 shutdown_connections 2024-05-09T05:09:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 --2- 172.21.15.1:0/711057274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f964014ff20 0x7f9640145c00 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 >> 172.21.15.1:0/711057274 conn(0x7f96400ce180 msgr2=0x7f964014d5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 shutdown_connections 2024-05-09T05:09:24.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:23.995+0000 7f964656c640 1 -- 172.21.15.1:0/711057274 wait complete. 2024-05-09T05:09:24.011 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:09:24.011 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:09:24.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.138+0000 7f99ba786640 1 Processor -- start 2024-05-09T05:09:24.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.138+0000 7f99ba786640 1 -- start start 2024-05-09T05:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.140+0000 7f99ba786640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f99b415afd0 0x7f99b4165ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.140+0000 7f99ba786640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99b40c5710 con 0x7f99b4154980 2024-05-09T05:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.140+0000 7f99ba786640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f99b40c4b00 con 0x7f99b40cbd10 2024-05-09T05:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.140+0000 7f99ba786640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f99b415e510 con 0x7f99b415afd0 2024-05-09T05:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.140+0000 7f99b8d23640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f99b415afd0 0x7f99b4165ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.140+0000 7f99b37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f99b4154980 0x7f99b40cc830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41186/0 (socket says 172.21.15.1:41186) 2024-05-09T05:09:24.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.140+0000 7f99b37fe640 1 -- 172.21.15.1:0/3673091714 learned_addr learned my addr 172.21.15.1:0/3673091714 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.141+0000 7f99b8d23640 1 -- 172.21.15.1:0/3673091714 >> v1:172.21.15.196:6789/0 conn(0x7f99b40cbd10 legacy=0x7f99b40c5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.141+0000 7f99b8d23640 1 -- 172.21.15.1:0/3673091714 >> v1:172.21.15.1:6789/0 conn(0x7f99b4154980 legacy=0x7f99b40cc830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.141+0000 7f99b8d23640 1 -- 172.21.15.1:0/3673091714 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b40cafd0 con 0x7f99b415afd0 2024-05-09T05:09:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.141+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3673091714 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1505703450 0 0) 0x7f99b40c5710 con 0x7f99b4154980 2024-05-09T05:09:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.142+0000 7f99b8d23640 1 --2- 172.21.15.1:0/3673091714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 0x7f99b4165ee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f99ac009d90 tx=0x7f99ac02a660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bef498ae5cdb113 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.142+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3673091714 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac007740 con 0x7f99b415afd0 2024-05-09T05:09:24.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.142+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3673091714 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99ac0078e0 con 0x7f99b415afd0 2024-05-09T05:09:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.142+0000 7f99b2ffd640 1 -- 172.21.15.1:0/3673091714 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac007cb0 con 0x7f99b415afd0 2024-05-09T05:09:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.143+0000 7f99ba786640 1 -- 172.21.15.1:0/3673091714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 msgr2=0x7f99b4165ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.143+0000 7f99ba786640 1 --2- 172.21.15.1:0/3673091714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 0x7f99b4165ee0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f99ac009d90 tx=0x7f99ac02a660 comp rx=0 tx=0).stop 2024-05-09T05:09:24.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.143+0000 7f99ba786640 1 -- 172.21.15.1:0/3673091714 shutdown_connections 2024-05-09T05:09:24.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.143+0000 7f99ba786640 1 --2- 172.21.15.1:0/3673091714 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 0x7f99b4165ee0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.143+0000 7f99ba786640 1 -- 172.21.15.1:0/3673091714 >> 172.21.15.1:0/3673091714 conn(0x7f99b4072e50 msgr2=0x7f99b4073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:24.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.143+0000 7f99ba786640 1 -- 172.21.15.1:0/3673091714 shutdown_connections 2024-05-09T05:09:24.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.143+0000 7f99ba786640 1 -- 172.21.15.1:0/3673091714 wait complete. 2024-05-09T05:09:24.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.145+0000 7f99ba786640 1 Processor -- start 2024-05-09T05:09:24.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.145+0000 7f99ba786640 1 -- start start 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99ba786640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40cbd10 0x7f99b40cdfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99ba786640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f99b4154980 0x7f99b4151390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99ba786640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 0x7f99b41518d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99ba786640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f99b415dbe0 con 0x7f99b415afd0 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99ba786640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f99b40cb290 con 0x7f99b40cbd10 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99ba786640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f99b415dec0 con 0x7f99b4154980 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99b8d23640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 0x7f99b41518d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99b8d23640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 0x7f99b41518d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34450/0 (socket says 172.21.15.1:34450) 2024-05-09T05:09:24.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99b8d23640 1 -- 172.21.15.1:0/3071765620 learned_addr learned my addr 172.21.15.1:0/3071765620 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40cbd10 0x7f99b40cdfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.146+0000 7f99b37fe640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f99b4154980 0x7f99b4151390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b8d23640 1 -- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f99b4154980 msgr2=0x7f99b4151390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b8d23640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f99b4154980 0x7f99b4151390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b8d23640 1 -- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40cbd10 msgr2=0x7f99b40cdfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b8d23640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40cbd10 0x7f99b40cdfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b8d23640 1 -- 172.21.15.1:0/3071765620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f99b40ce500 con 0x7f99b415afd0 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b8d23640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 0x7f99b41518d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f99ac009eb0 tx=0x7f99ac004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac040070 con 0x7f99b415afd0 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f99ac031c40 con 0x7f99b415afd0 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f99b406eae0 con 0x7f99b415afd0 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99ac003990 con 0x7f99b415afd0 2024-05-09T05:09:24.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.147+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99b406f0a0 con 0x7f99b415afd0 2024-05-09T05:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.148+0000 7f99ba786640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40ce6d0 0x7f99b41f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.148+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f99b406f270 con 0x7f99b40ce6d0 2024-05-09T05:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.149+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40ce6d0 0x7f99b41f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.149+0000 7f99b3fff640 1 -- 172.21.15.1:0/3071765620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f99b4050870 con 0x7f99b40ce6d0 2024-05-09T05:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.149+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40ce6d0 0x7f99b41f6230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f99a400f520 tx=0x7f99a400f9e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.149+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f99ac039040 con 0x7f99b415afd0 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b0ff9640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f99a00762d0 0x7f99a00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f99ac0b87d0 con 0x7f99b415afd0 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b37fe640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f99a00762d0 0x7f99a00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a40052e0 con 0x7f99b40ce6d0 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f99a4018070 con 0x7f99b40ce6d0 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40ce6d0 msgr2=0x7f99b41f6230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b0ff9640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40ce6d0 0x7f99b41f6230 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f99a400f520 tx=0x7f99a400f9e0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40ce6d0 msgr2=0x7f99b41f6230 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.150+0000 7f99b0ff9640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b40ce6d0 0x7f99b41f6230 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.151+0000 7f99b37fe640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f99a00762d0 0x7f99a00787e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f99b4072c40 tx=0x7f99a8007e40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.158+0000 7f99ba786640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b406f540 0x7f99b41f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.158+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f99b40c68f0 con 0x7f99b406f540 2024-05-09T05:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.158+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b406f540 0x7f99b41f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.159+0000 7f99b3fff640 1 -- 172.21.15.1:0/3071765620 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f99b4158800 con 0x7f99b406f540 2024-05-09T05:09:24.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.159+0000 7f99b3fff640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b406f540 0x7f99b41f6770 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f99a400f520 tx=0x7f99a4001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.159+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f99a4003b40 con 0x7f99b406f540 2024-05-09T05:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.159+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f99a4003b40 con 0x7f99b406f540 2024-05-09T05:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.159+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b406f540 msgr2=0x7f99b41f6770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.159+0000 7f99b0ff9640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b406f540 0x7f99b41f6770 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f99a400f520 tx=0x7f99a4001080 comp rx=0 tx=0).stop 2024-05-09T05:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.159+0000 7f99b0ff9640 1 -- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b406f540 msgr2=0x7f99b41f6770 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:24.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.159+0000 7f99b0ff9640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f99b406f540 0x7f99b41f6770 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 7731, 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:09:24.161 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:24.162 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:24.163 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:09:24.164 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:24.165 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.167 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:24.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.171 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:09:24.172 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f99a00762d0 msgr2=0x7f99a00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99ba786640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f99a00762d0 0x7f99a00787e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f99b4072c40 tx=0x7f99a8007e40 comp rx=0 tx=0).stop 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 msgr2=0x7f99b41518d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99ba786640 1 --2- 172.21.15.1:0/3071765620 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f99b415afd0 0x7f99b41518d0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f99ac009eb0 tx=0x7f99ac004a40 comp rx=0 tx=0).stop 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99b8d23640 1 -- 172.21.15.1:0/3071765620 reap_dead start 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99b8d23640 1 -- 172.21.15.1:0/3071765620 reap_dead start 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 shutdown_connections 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 >> 172.21.15.1:0/3071765620 conn(0x7f99b4072e50 msgr2=0x7f99b4152a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:24.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 shutdown_connections 2024-05-09T05:09:24.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.160+0000 7f99ba786640 1 -- 172.21.15.1:0/3071765620 wait complete. 2024-05-09T05:09:24.175 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:09:24.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:09:24.175 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.302+0000 7f3d2c36c640 1 Processor -- start 2024-05-09T05:09:24.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.302+0000 7f3d2c36c640 1 -- start start 2024-05-09T05:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.303+0000 7f3d2c36c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3d241546b0 0x7f3d2406c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.303+0000 7f3d2c36c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 0x7f3d240c8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.303+0000 7f3d2c36c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d2415dbf0 con 0x7f3d240c9630 2024-05-09T05:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.303+0000 7f3d2c36c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3d240c9c30 con 0x7f3d241546b0 2024-05-09T05:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.303+0000 7f3d2c36c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2415de90 con 0x7f3d240c8020 2024-05-09T05:09:24.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.303+0000 7f3d29907640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3d241546b0 0x7f3d2406c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.303+0000 7f3d29907640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3d241546b0 0x7f3d2406c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34464/0 (socket says 172.21.15.1:34464) 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.303+0000 7f3d29907640 1 -- 172.21.15.1:0/1688593706 learned_addr learned my addr 172.21.15.1:0/1688593706 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.304+0000 7f3d2a909640 1 --2- 172.21.15.1:0/1688593706 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 0x7f3d240c8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.304+0000 7f3d29907640 1 -- 172.21.15.1:0/1688593706 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 msgr2=0x7f3d240c8430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.304+0000 7f3d29907640 1 --2- 172.21.15.1:0/1688593706 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 0x7f3d240c8430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.304+0000 7f3d29907640 1 -- 172.21.15.1:0/1688593706 >> v1:172.21.15.196:6789/0 conn(0x7f3d240c9630 legacy=0x7f3d2415afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.304+0000 7f3d29907640 1 -- 172.21.15.1:0/1688593706 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d2415e170 con 0x7f3d241546b0 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.304+0000 7f3d2a909640 1 --2- 172.21.15.1:0/1688593706 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 0x7f3d240c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.305+0000 7f3d29907640 1 --2- 172.21.15.1:0/1688593706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 0x7f3d2406c6f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009870 tx=0x7f3d1402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f45d138c4400dcd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.305+0000 7f3d29106640 1 -- 172.21.15.1:0/1688593706 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14037070 con 0x7f3d241546b0 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.305+0000 7f3d29106640 1 -- 172.21.15.1:0/1688593706 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1402ae50 con 0x7f3d241546b0 2024-05-09T05:09:24.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.305+0000 7f3d29106640 1 -- 172.21.15.1:0/1688593706 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d14032860 con 0x7f3d241546b0 2024-05-09T05:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.306+0000 7f3d2c36c640 1 -- 172.21.15.1:0/1688593706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 msgr2=0x7f3d2406c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.306+0000 7f3d2c36c640 1 --2- 172.21.15.1:0/1688593706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 0x7f3d2406c6f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3d14009870 tx=0x7f3d1402a300 comp rx=0 tx=0).stop 2024-05-09T05:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.306+0000 7f3d2c36c640 1 -- 172.21.15.1:0/1688593706 shutdown_connections 2024-05-09T05:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.306+0000 7f3d2c36c640 1 --2- 172.21.15.1:0/1688593706 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 0x7f3d240c8430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.306+0000 7f3d2c36c640 1 --2- 172.21.15.1:0/1688593706 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 0x7f3d2406c6f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.306+0000 7f3d2c36c640 1 -- 172.21.15.1:0/1688593706 >> 172.21.15.1:0/1688593706 conn(0x7f3d240ce910 msgr2=0x7f3d240ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.306+0000 7f3d2c36c640 1 -- 172.21.15.1:0/1688593706 shutdown_connections 2024-05-09T05:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.306+0000 7f3d2c36c640 1 -- 172.21.15.1:0/1688593706 wait complete. 2024-05-09T05:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.308+0000 7f3d2c36c640 1 Processor -- start 2024-05-09T05:09:24.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.308+0000 7f3d2c36c640 1 -- start start 2024-05-09T05:09:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2c36c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 0x7f3d24145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2c36c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d240c9630 0x7f3d241f4420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2c36c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 0x7f3d24141090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2c36c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3d2415e970 con 0x7f3d241546b0 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2c36c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3d240c9f10 con 0x7f3d240c9630 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2c36c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2415eaf0 con 0x7f3d240c8020 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 0x7f3d24141090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 0x7f3d24141090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34470/0 (socket says 172.21.15.1:34470) 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 -- 172.21.15.1:0/3747366012 learned_addr learned my addr 172.21.15.1:0/3747366012 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a108640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 0x7f3d24145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d29907640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d240c9630 0x7f3d241f4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 -- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 msgr2=0x7f3d24145fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d240c8020 0x7f3d24145fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 -- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d240c9630 msgr2=0x7f3d241f4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d240c9630 0x7f3d241f4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 -- 172.21.15.1:0/3747366012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d24147a30 con 0x7f3d241546b0 2024-05-09T05:09:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.310+0000 7f3d2a909640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 0x7f3d24141090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c007e10 tx=0x7f3d1c00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.311+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c01a070 con 0x7f3d241546b0 2024-05-09T05:09:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.311+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1c003b90 con 0x7f3d241546b0 2024-05-09T05:09:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.311+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c00de20 con 0x7f3d241546b0 2024-05-09T05:09:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.311+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d241483a0 con 0x7f3d241546b0 2024-05-09T05:09:24.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.311+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d2415c370 con 0x7f3d241546b0 2024-05-09T05:09:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.311+0000 7f3d2c36c640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241f4960 0x7f3d241464e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.312+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d2415c1d0 con 0x7f3d241f4960 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.312+0000 7f3d2a108640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241f4960 0x7f3d241464e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.312+0000 7f3d2a108640 1 -- 172.21.15.1:0/3747366012 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d24148130 con 0x7f3d241f4960 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.312+0000 7f3d2a108640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241f4960 0x7f3d241464e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000f520 tx=0x7f3d2000f9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.312+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d200052e0 con 0x7f3d241f4960 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.312+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3d1c01d070 con 0x7f3d241546b0 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.313+0000 7f3d1affd640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d080761d0 0x7f3d080786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.313+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3d1c0978a0 con 0x7f3d241546b0 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.313+0000 7f3d29907640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d080761d0 0x7f3d080786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.313+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d20018070 con 0x7f3d241f4960 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.313+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241f4960 msgr2=0x7f3d241464e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.313+0000 7f3d1affd640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241f4960 0x7f3d241464e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f3d2000f520 tx=0x7f3d2000f9e0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.313+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241f4960 msgr2=0x7f3d241464e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.313+0000 7f3d1affd640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241f4960 0x7f3d241464e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.314+0000 7f3d29907640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d080761d0 0x7f3d080786e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3d14034000 tx=0x7f3d1402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.323+0000 7f3d2c36c640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241415d0 0x7f3d24148b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.323+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d240c5aa0 con 0x7f3d241415d0 2024-05-09T05:09:24.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.324+0000 7f3d2a108640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241415d0 0x7f3d24148b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.324+0000 7f3d2a108640 1 -- 172.21.15.1:0/3747366012 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d240ca190 con 0x7f3d241415d0 2024-05-09T05:09:24.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.324+0000 7f3d2a108640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241415d0 0x7f3d24148b10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3d20017000 tx=0x7f3d20001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.324+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d20005480 con 0x7f3d241415d0 2024-05-09T05:09:24.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5061 (secure 0 0 0) 0x7f3d20003dd0 con 0x7f3d241415d0 2024-05-09T05:09:24.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241415d0 msgr2=0x7f3d24148b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d1affd640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241415d0 0x7f3d24148b10 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3d20017000 tx=0x7f3d20001080 comp rx=0 tx=0).stop 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d1affd640 1 -- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241415d0 msgr2=0x7f3d24148b10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d1affd640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d241415d0 0x7f3d24148b10 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 122321, 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:09:24.326 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:24.327 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:24.328 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:24.329 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:24.330 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:24.331 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:09:24.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.334 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.335 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.336 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.337 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:24.338 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:09:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:24.339 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:09:24.339 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:09:24.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d080761d0 msgr2=0x7f3d080786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2c36c640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d080761d0 0x7f3d080786e0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f3d14034000 tx=0x7f3d1402e000 comp rx=0 tx=0).stop 2024-05-09T05:09:24.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 msgr2=0x7f3d24141090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2c36c640 1 --2- 172.21.15.1:0/3747366012 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d241546b0 0x7f3d24141090 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c007e10 tx=0x7f3d1c00d4a0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2a909640 1 -- 172.21.15.1:0/3747366012 reap_dead start 2024-05-09T05:09:24.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2a909640 1 -- 172.21.15.1:0/3747366012 reap_dead start 2024-05-09T05:09:24.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 shutdown_connections 2024-05-09T05:09:24.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 >> 172.21.15.1:0/3747366012 conn(0x7f3d240ce910 msgr2=0x7f3d2406fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:24.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 shutdown_connections 2024-05-09T05:09:24.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.325+0000 7f3d2c36c640 1 -- 172.21.15.1:0/3747366012 wait complete. 2024-05-09T05:09:24.341 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:09:24.341 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:09:24.342 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:09:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.466+0000 7f2a41070640 1 Processor -- start 2024-05-09T05:09:24.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.466+0000 7f2a41070640 1 -- start start 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a41070640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a3c06dd70 0x7f2a3c0cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a41070640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd2b0 0x7f2a3c167180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a41070640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2a3c0cd0a0 con 0x7f2a3c0cd7f0 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a41070640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2a3c0cc530 con 0x7f2a3c06dd70 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a41070640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2a3c0cce00 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a3ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a3c06dd70 0x7f2a3c0cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a3ad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a3c06dd70 0x7f2a3c0cbff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34486/0 (socket says 172.21.15.1:34486) 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a3ad9d640 1 -- 172.21.15.1:0/369592371 learned_addr learned my addr 172.21.15.1:0/369592371 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:24.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a3b59e640 1 --2- 172.21.15.1:0/369592371 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd2b0 0x7f2a3c167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a3ad9d640 1 -- 172.21.15.1:0/369592371 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd2b0 msgr2=0x7f2a3c167180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a3ad9d640 1 --2- 172.21.15.1:0/369592371 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd2b0 0x7f2a3c167180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a3ad9d640 1 -- 172.21.15.1:0/369592371 >> v1:172.21.15.196:6789/0 conn(0x7f2a3c0cd7f0 legacy=0x7f2a3c0c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.468+0000 7f2a3ad9d640 1 -- 172.21.15.1:0/369592371 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c0c56a0 con 0x7f2a3c06dd70 2024-05-09T05:09:24.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.469+0000 7f2a3ad9d640 1 --2- 172.21.15.1:0/369592371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c06dd70 0x7f2a3c0cbff0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009870 tx=0x7f2a2c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ddaa65404537dbe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.472+0000 7f2a39d9b640 1 -- 172.21.15.1:0/369592371 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c037070 con 0x7f2a3c06dd70 2024-05-09T05:09:24.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.472+0000 7f2a39d9b640 1 -- 172.21.15.1:0/369592371 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2c032440 con 0x7f2a3c06dd70 2024-05-09T05:09:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.472+0000 7f2a39d9b640 1 -- 172.21.15.1:0/369592371 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c032810 con 0x7f2a3c06dd70 2024-05-09T05:09:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.473+0000 7f2a41070640 1 -- 172.21.15.1:0/369592371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c06dd70 msgr2=0x7f2a3c0cbff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.473+0000 7f2a41070640 1 --2- 172.21.15.1:0/369592371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c06dd70 0x7f2a3c0cbff0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009870 tx=0x7f2a2c02a250 comp rx=0 tx=0).stop 2024-05-09T05:09:24.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.473+0000 7f2a41070640 1 -- 172.21.15.1:0/369592371 shutdown_connections 2024-05-09T05:09:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.473+0000 7f2a41070640 1 --2- 172.21.15.1:0/369592371 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd2b0 0x7f2a3c167180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.473+0000 7f2a41070640 1 --2- 172.21.15.1:0/369592371 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c06dd70 0x7f2a3c0cbff0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.473+0000 7f2a41070640 1 -- 172.21.15.1:0/369592371 >> 172.21.15.1:0/369592371 conn(0x7f2a3c0ce110 msgr2=0x7f2a3c0ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.473+0000 7f2a41070640 1 -- 172.21.15.1:0/369592371 shutdown_connections 2024-05-09T05:09:24.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.473+0000 7f2a41070640 1 -- 172.21.15.1:0/369592371 wait complete. 2024-05-09T05:09:24.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.476+0000 7f2a41070640 1 Processor -- start 2024-05-09T05:09:24.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.476+0000 7f2a41070640 1 -- start start 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a41070640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c0cd2b0 0x7f2a3c1f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a41070640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd7f0 0x7f2a3c1e9320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a41070640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a3c0ca5d0 0x7f2a3c146c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a41070640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0c5b00 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a41070640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0c5980 con 0x7f2a3c0ca5d0 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a41070640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2a3c074a20 con 0x7f2a3c0cd7f0 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c0cd2b0 0x7f2a3c1f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c0cd2b0 0x7f2a3c1f5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34490/0 (socket says 172.21.15.1:34490) 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3ad9d640 1 -- 172.21.15.1:0/1653204543 learned_addr learned my addr 172.21.15.1:0/1653204543 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3b59e640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a3c0ca5d0 0x7f2a3c146c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3a59c640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd7f0 0x7f2a3c1e9320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3ad9d640 1 -- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd7f0 msgr2=0x7f2a3c1e9320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3ad9d640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd7f0 0x7f2a3c1e9320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3ad9d640 1 -- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a3c0ca5d0 msgr2=0x7f2a3c146c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3ad9d640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a3c0ca5d0 0x7f2a3c146c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.478+0000 7f2a3ad9d640 1 -- 172.21.15.1:0/1653204543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c1f62b0 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.479+0000 7f2a3ad9d640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c0cd2b0 0x7f2a3c1f5d70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009870 tx=0x7f2a2c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.479+0000 7f2a237fe640 1 -- 172.21.15.1:0/1653204543 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c037040 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.479+0000 7f2a237fe640 1 -- 172.21.15.1:0/1653204543 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a2c042070 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.479+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3c14ff90 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.479+0000 7f2a237fe640 1 -- 172.21.15.1:0/1653204543 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2c030880 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.479+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3c150550 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.481 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.480+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a3c050870 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.481+0000 7f2a237fe640 1 -- 172.21.15.1:0/1653204543 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2a2c030a20 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.482+0000 7f2a237fe640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a28076300 0x7f2a28078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:24.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.482+0000 7f2a237fe640 1 -- 172.21.15.1:0/1653204543 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2a2c0b7d50 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.482+0000 7f2a3a59c640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a28076300 0x7f2a28078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:24.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.483+0000 7f2a3a59c640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a28076300 0x7f2a28078810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0c5220 tx=0x7f2a30009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:24.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.487+0000 7f2a237fe640 1 -- 172.21.15.1:0/1653204543 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2a2c0bb030 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.822+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f2a3c074770 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.823+0000 7f2a237fe640 1 -- 172.21.15.1:0/1653204543 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f2a2c080fe0 con 0x7f2a3c0cd2b0 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a28076300 msgr2=0x7f2a28078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a28076300 0x7f2a28078810 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f2a3c0c5220 tx=0x7f2a30009040 comp rx=0 tx=0).stop 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c0cd2b0 msgr2=0x7f2a3c1f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c0cd2b0 0x7f2a3c1f5d70 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2a2c009870 tx=0x7f2a2c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 shutdown_connections 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a28076300 0x7f2a28078810 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a3c0ca5d0 0x7f2a3c146c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a3c0cd7f0 0x7f2a3c1e9320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 --2- 172.21.15.1:0/1653204543 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a3c0cd2b0 0x7f2a3c1f5d70 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 >> 172.21.15.1:0/1653204543 conn(0x7f2a3c0ce110 msgr2=0x7f2a3c155ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 shutdown_connections 2024-05-09T05:09:24.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:24.826+0000 7f2a41070640 1 -- 172.21.15.1:0/1653204543 wait complete. 2024-05-09T05:09:24.840 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:09:32.459 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:09:32.462 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:09:34.841 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:09:34.842 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:09:34.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.966+0000 7fccf9727640 1 Processor -- start 2024-05-09T05:09:34.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.966+0000 7fccf9727640 1 -- start start 2024-05-09T05:09:34.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.967+0000 7fccf9727640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fccf40c5ca0 0x7fccf40ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:34.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.967+0000 7fccf9727640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fccf4156a80 0x7fccf406df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.967+0000 7fccf9727640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fccf40c86a0 con 0x7fccf4156e90 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.967+0000 7fccf9727640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fccf40cb910 con 0x7fccf40c5ca0 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.967+0000 7fccf9727640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fccf40c8980 con 0x7fccf4156a80 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.968+0000 7fccf2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fccf40c5ca0 0x7fccf40ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.968+0000 7fccf37fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fccf4156e90 0x7fccf40c7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42216/0 (socket says 172.21.15.1:42216) 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.968+0000 7fccf37fe640 1 -- 172.21.15.1:0/3891703970 learned_addr learned my addr 172.21.15.1:0/3891703970 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.968+0000 7fccf27fc640 1 --2- 172.21.15.1:0/3891703970 >> v2:172.21.15.196:3300/0 conn(0x7fccf4156a80 0x7fccf406df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.968+0000 7fccf2ffd640 1 -- 172.21.15.1:0/3891703970 >> v2:172.21.15.196:3300/0 conn(0x7fccf4156a80 msgr2=0x7fccf406df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.968+0000 7fccf2ffd640 1 --2- 172.21.15.1:0/3891703970 >> v2:172.21.15.196:3300/0 conn(0x7fccf4156a80 0x7fccf406df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:34.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.968+0000 7fccf2ffd640 1 -- 172.21.15.1:0/3891703970 >> v1:172.21.15.1:6789/0 conn(0x7fccf4156e90 legacy=0x7fccf40c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:34.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.968+0000 7fccf2ffd640 1 -- 172.21.15.1:0/3891703970 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf40c8c60 con 0x7fccf40c5ca0 2024-05-09T05:09:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.969+0000 7fccf2ffd640 1 --2- 172.21.15.1:0/3891703970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf40c5ca0 0x7fccf40ca4a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fccf40cafd0 tx=0x7fcce402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee14e60483c29126 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.969+0000 7fccf1ffb640 1 -- 172.21.15.1:0/3891703970 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4037070 con 0x7fccf40c5ca0 2024-05-09T05:09:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.969+0000 7fccf1ffb640 1 -- 172.21.15.1:0/3891703970 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcce4032440 con 0x7fccf40c5ca0 2024-05-09T05:09:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.969+0000 7fccf1ffb640 1 -- 172.21.15.1:0/3891703970 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcce4032810 con 0x7fccf40c5ca0 2024-05-09T05:09:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.970+0000 7fccf9727640 1 -- 172.21.15.1:0/3891703970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf40c5ca0 msgr2=0x7fccf40ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:34.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.970+0000 7fccf9727640 1 --2- 172.21.15.1:0/3891703970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf40c5ca0 0x7fccf40ca4a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fccf40cafd0 tx=0x7fcce402a230 comp rx=0 tx=0).stop 2024-05-09T05:09:34.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.970+0000 7fccf9727640 1 -- 172.21.15.1:0/3891703970 shutdown_connections 2024-05-09T05:09:34.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.970+0000 7fccf9727640 1 --2- 172.21.15.1:0/3891703970 >> v2:172.21.15.196:3300/0 conn(0x7fccf4156a80 0x7fccf406df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:34.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.970+0000 7fccf9727640 1 --2- 172.21.15.1:0/3891703970 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf40c5ca0 0x7fccf40ca4a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:34.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.970+0000 7fccf9727640 1 -- 172.21.15.1:0/3891703970 >> 172.21.15.1:0/3891703970 conn(0x7fccf40745a0 msgr2=0x7fccf4071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:34.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.970+0000 7fccf9727640 1 -- 172.21.15.1:0/3891703970 shutdown_connections 2024-05-09T05:09:34.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.970+0000 7fccf9727640 1 -- 172.21.15.1:0/3891703970 wait complete. 2024-05-09T05:09:34.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.972+0000 7fccf9727640 1 Processor -- start 2024-05-09T05:09:34.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.972+0000 7fccf9727640 1 -- start start 2024-05-09T05:09:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf9727640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fccf40c5ca0 0x7fccf41572a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf9727640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf4156a80 0x7fccf41f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf9727640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf4156e90 0x7fccf4157f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf9727640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fccf40cad60 con 0x7fccf4156e90 2024-05-09T05:09:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf9727640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fccf40cb5a0 con 0x7fccf4156a80 2024-05-09T05:09:34.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf9727640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fccf40ca9e0 con 0x7fccf40c5ca0 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf4156e90 0x7fccf4157f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf4156e90 0x7fccf4157f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53414/0 (socket says 172.21.15.1:53414) 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf37fe640 1 -- 172.21.15.1:0/2350844657 learned_addr learned my addr 172.21.15.1:0/2350844657 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf27fc640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf4156a80 0x7fccf41f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.974+0000 7fccf2ffd640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fccf40c5ca0 0x7fccf41572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccf37fe640 1 -- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fccf40c5ca0 msgr2=0x7fccf41572a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccf37fe640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fccf40c5ca0 0x7fccf41572a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccf37fe640 1 -- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf4156a80 msgr2=0x7fccf41f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccf37fe640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf4156a80 0x7fccf41f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccf37fe640 1 -- 172.21.15.1:0/2350844657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccf415bff0 con 0x7fccf4156e90 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccf37fe640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf4156e90 0x7fccf4157f40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fccec00cab0 tx=0x7fccec00cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccdb7fe640 1 -- 172.21.15.1:0/2350844657 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec017070 con 0x7fccf4156e90 2024-05-09T05:09:34.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccdb7fe640 1 -- 172.21.15.1:0/2350844657 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fccec00ac30 con 0x7fccf4156e90 2024-05-09T05:09:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccdb7fe640 1 -- 172.21.15.1:0/2350844657 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccec012400 con 0x7fccf4156e90 2024-05-09T05:09:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccf415c2d0 con 0x7fccf4156e90 2024-05-09T05:09:34.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.975+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccf4159af0 con 0x7fccf4156e90 2024-05-09T05:09:34.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.976+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccf4050870 con 0x7fccf4156e90 2024-05-09T05:09:34.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.977+0000 7fccdb7fe640 1 -- 172.21.15.1:0/2350844657 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fccec01a070 con 0x7fccf4156e90 2024-05-09T05:09:34.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.977+0000 7fccdb7fe640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcce00762d0 0x7fcce00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:34.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.978+0000 7fccdb7fe640 1 -- 172.21.15.1:0/2350844657 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fccec097280 con 0x7fccf4156e90 2024-05-09T05:09:34.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.978+0000 7fccf2ffd640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcce00762d0 0x7fcce00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:34.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.978+0000 7fccf2ffd640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcce00762d0 0x7fcce00787e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcce4034000 tx=0x7fcce402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:34.983+0000 7fccdb7fe640 1 -- 172.21.15.1:0/2350844657 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fccec060510 con 0x7fccf4156e90 2024-05-09T05:09:35.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.329+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fccf41f3620 con 0x7fccf4156e90 2024-05-09T05:09:35.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.330+0000 7fccdb7fe640 1 -- 172.21.15.1:0/2350844657 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fccec064810 con 0x7fccf4156e90 2024-05-09T05:09:35.331 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:09:35.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcce00762d0 msgr2=0x7fcce00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcce00762d0 0x7fcce00787e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fcce4034000 tx=0x7fcce402e000 comp rx=0 tx=0).stop 2024-05-09T05:09:35.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf4156e90 msgr2=0x7fccf4157f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf4156e90 0x7fccf4157f40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fccec00cab0 tx=0x7fccec00cf70 comp rx=0 tx=0).stop 2024-05-09T05:09:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 shutdown_connections 2024-05-09T05:09:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcce00762d0 0x7fcce00787e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fccf4156a80 0x7fccf41f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fccf4156e90 0x7fccf4157f40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.332+0000 7fccf9727640 1 --2- 172.21.15.1:0/2350844657 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fccf40c5ca0 0x7fccf41572a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.333+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 >> 172.21.15.1:0/2350844657 conn(0x7fccf40745a0 msgr2=0x7fccf40c5500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.333+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 shutdown_connections 2024-05-09T05:09:35.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.333+0000 7fccf9727640 1 -- 172.21.15.1:0/2350844657 wait complete. 2024-05-09T05:09:35.347 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:09:35.347 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:09:35.347 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:09:35.347 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:09:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.464+0000 7f336b449640 1 Processor -- start 2024-05-09T05:09:35.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.464+0000 7f336b449640 1 -- start start 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f336b449640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33640c7990 0x7f33640c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f336b449640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f336415c7b0 con 0x7f33640ce010 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f336b449640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33640cca50 con 0x7f33640b4ec0 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f336b449640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f336415d0c0 con 0x7f33640c7990 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f33699e6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33640c7990 0x7f33640c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f33689e4640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f33640ce010 0x7f33640c7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42232/0 (socket says 172.21.15.1:42232) 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f33689e4640 1 -- 172.21.15.1:0/1104446188 learned_addr learned my addr 172.21.15.1:0/1104446188 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f33699e6640 1 -- 172.21.15.1:0/1104446188 >> v1:172.21.15.196:6789/0 conn(0x7f33640b4ec0 legacy=0x7f33640cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f33699e6640 1 -- 172.21.15.1:0/1104446188 >> v1:172.21.15.1:6789/0 conn(0x7f33640ce010 legacy=0x7f33640c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.466+0000 7f33699e6640 1 -- 172.21.15.1:0/1104446188 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33640ccd30 con 0x7f33640c7990 2024-05-09T05:09:35.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.467+0000 7f33699e6640 1 --2- 172.21.15.1:0/1104446188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c7990 0x7f33640c7da0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f335c009d90 tx=0x7f335c02a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4352a48ec04ac34 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.467+0000 7f335bfff640 1 -- 172.21.15.1:0/1104446188 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c037070 con 0x7f33640c7990 2024-05-09T05:09:35.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.468+0000 7f335bfff640 1 -- 172.21.15.1:0/1104446188 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335c007740 con 0x7f33640c7990 2024-05-09T05:09:35.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.468+0000 7f335bfff640 1 -- 172.21.15.1:0/1104446188 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c007b10 con 0x7f33640c7990 2024-05-09T05:09:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.468+0000 7f336b449640 1 -- 172.21.15.1:0/1104446188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c7990 msgr2=0x7f33640c7da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.468+0000 7f336b449640 1 --2- 172.21.15.1:0/1104446188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c7990 0x7f33640c7da0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f335c009d90 tx=0x7f335c02a780 comp rx=0 tx=0).stop 2024-05-09T05:09:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.468+0000 7f336b449640 1 -- 172.21.15.1:0/1104446188 shutdown_connections 2024-05-09T05:09:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.468+0000 7f336b449640 1 --2- 172.21.15.1:0/1104446188 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c7990 0x7f33640c7da0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.469+0000 7f336b449640 1 -- 172.21.15.1:0/1104446188 >> 172.21.15.1:0/1104446188 conn(0x7f33640d2360 msgr2=0x7f33640ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.469+0000 7f336b449640 1 -- 172.21.15.1:0/1104446188 shutdown_connections 2024-05-09T05:09:35.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.469+0000 7f336b449640 1 -- 172.21.15.1:0/1104446188 wait complete. 2024-05-09T05:09:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.471+0000 7f336b449640 1 Processor -- start 2024-05-09T05:09:35.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.471+0000 7f336b449640 1 -- start start 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f336b449640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33640b4ec0 0x7f3364154d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f336b449640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33640c7990 0x7f336406e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f336b449640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640ce010 0x7f33641595c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f336b449640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f33640cc750 con 0x7f33640ce010 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f336b449640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f336415cd50 con 0x7f33640c7990 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f336b449640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3364156ae0 con 0x7f33640b4ec0 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f33699e6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640ce010 0x7f33641595c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f33699e6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640ce010 0x7f33641595c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53428/0 (socket says 172.21.15.1:53428) 2024-05-09T05:09:35.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f33699e6640 1 -- 172.21.15.1:0/423450088 learned_addr learned my addr 172.21.15.1:0/423450088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f33689e4640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33640c7990 0x7f336406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.472+0000 7f33691e5640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33640b4ec0 0x7f3364154d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f33699e6640 1 -- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33640b4ec0 msgr2=0x7f3364154d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f33699e6640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33640b4ec0 0x7f3364154d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f33699e6640 1 -- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33640c7990 msgr2=0x7f336406e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f33699e6640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33640c7990 0x7f336406e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f33699e6640 1 -- 172.21.15.1:0/423450088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33641542d0 con 0x7f33640ce010 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f33699e6640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640ce010 0x7f33641595c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f335c009eb0 tx=0x7f335c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c042020 con 0x7f33640ce010 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f335c032990 con 0x7f33640ce010 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f335c037040 con 0x7f33640ce010 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33641552b0 con 0x7f33640ce010 2024-05-09T05:09:35.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.473+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3364154570 con 0x7f33640ce010 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.474+0000 7f336b449640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c8f20 0x7f3364159b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.474+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3364050870 con 0x7f33640c8f20 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.475+0000 7f33691e5640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c8f20 0x7f3364159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.475+0000 7f33691e5640 1 -- 172.21.15.1:0/423450088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f336415a400 con 0x7f33640c8f20 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.475+0000 7f33691e5640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c8f20 0x7f3364159b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f336000f570 tx=0x7f336000fa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.475+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33600052e0 con 0x7f33640c8f20 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f335c045020 con 0x7f33640ce010 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f3359ffb640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33480761d0 0x7f33480786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3360018070 con 0x7f33640c8f20 2024-05-09T05:09:35.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c8f20 msgr2=0x7f3364159b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f3359ffb640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c8f20 0x7f3364159b00 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f336000f570 tx=0x7f336000fa30 comp rx=0 tx=0).stop 2024-05-09T05:09:35.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c8f20 msgr2=0x7f3364159b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:35.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f3359ffb640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640c8f20 0x7f3364159b00 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f335c0b8780 con 0x7f33640ce010 2024-05-09T05:09:35.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.476+0000 7f33689e4640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33480761d0 0x7f33480786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.477+0000 7f33689e4640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33480761d0 0x7f33480786e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3354000f60 tx=0x7f3354009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.484+0000 7f336b449640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3364155570 0x7f33641f63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.484+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f33640cd2b0 con 0x7f3364155570 2024-05-09T05:09:35.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.484+0000 7f33691e5640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3364155570 0x7f33641f63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.485+0000 7f33691e5640 1 -- 172.21.15.1:0/423450088 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f33640cc000 con 0x7f3364155570 2024-05-09T05:09:35.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.485+0000 7f33691e5640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3364155570 0x7f33641f63b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f336000f570 tx=0x7f3360000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.485+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3360003e50 con 0x7f3364155570 2024-05-09T05:09:35.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.485+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4893 (secure 0 0 0) 0x7f3360005480 con 0x7f3364155570 2024-05-09T05:09:35.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.485+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3364155570 msgr2=0x7f33641f63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.485+0000 7f3359ffb640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3364155570 0x7f33641f63b0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f336000f570 tx=0x7f3360000ed0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.485+0000 7f3359ffb640 1 -- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3364155570 msgr2=0x7f33641f63b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:35.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.485+0000 7f3359ffb640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3364155570 0x7f33641f63b0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 81564, 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:09:35.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 13, 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.488 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.489 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:09:35.490 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.491 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:35.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.497 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.498 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:09:35.499 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:09:35.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33480761d0 msgr2=0x7f33480786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f336b449640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f33480761d0 0x7f33480786e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f3354000f60 tx=0x7f3354009040 comp rx=0 tx=0).stop 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640ce010 msgr2=0x7f33641595c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f336b449640 1 --2- 172.21.15.1:0/423450088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33640ce010 0x7f33641595c0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f335c009eb0 tx=0x7f335c004a40 comp rx=0 tx=0).stop 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f33699e6640 1 -- 172.21.15.1:0/423450088 reap_dead start 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f33699e6640 1 -- 172.21.15.1:0/423450088 reap_dead start 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 shutdown_connections 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 >> 172.21.15.1:0/423450088 conn(0x7f33640d2360 msgr2=0x7f33640b6550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 shutdown_connections 2024-05-09T05:09:35.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.486+0000 7f336b449640 1 -- 172.21.15.1:0/423450088 wait complete. 2024-05-09T05:09:35.501 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:09:35.502 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:09:35.502 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:09:35.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.622+0000 7fb6096a8640 1 Processor -- start 2024-05-09T05:09:35.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.622+0000 7fb6096a8640 1 -- start start 2024-05-09T05:09:35.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.623+0000 7fb6096a8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb6040c5e10 0x7fb60406e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.623+0000 7fb6096a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60415c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.623+0000 7fb6096a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb60406e6b0 con 0x7fb60406db90 2024-05-09T05:09:35.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.623+0000 7fb6096a8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb6040c7960 con 0x7fb6040c5e10 2024-05-09T05:09:35.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.623+0000 7fb6096a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb6040c7c40 con 0x7fb604156910 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.623+0000 7fb6037fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60415c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.623+0000 7fb5fbfff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb6040c5e10 0x7fb60406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.623+0000 7fb6037fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60415c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:33414/0 (socket says 172.21.15.1:33414) 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.624+0000 7fb6037fe640 1 -- 172.21.15.1:0/2528220710 learned_addr learned my addr 172.21.15.1:0/2528220710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.624+0000 7fb602ffd640 1 --1- 172.21.15.1:0/2528220710 >> v1:172.21.15.196:6789/0 conn(0x7fb60406db90 0x7fb60406fc30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:37948/0 (socket says 172.21.15.1:37948) 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.624+0000 7fb5fbfff640 1 -- 172.21.15.1:0/2528220710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 msgr2=0x7fb60415c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.624+0000 7fb5fbfff640 1 --2- 172.21.15.1:0/2528220710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60415c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.624+0000 7fb5fbfff640 1 -- 172.21.15.1:0/2528220710 >> v1:172.21.15.196:6789/0 conn(0x7fb60406db90 legacy=0x7fb60406fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.624+0000 7fb5fbfff640 1 -- 172.21.15.1:0/2528220710 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6040c7f20 con 0x7fb6040c5e10 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.624+0000 7fb6027fc640 1 -- 172.21.15.1:0/2528220710 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1351365159 0 0) 0x7fb60406e6b0 con 0x7fb60406db90 2024-05-09T05:09:35.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.624+0000 7fb6037fe640 1 --2- 172.21.15.1:0/2528220710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60415c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:35.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.625+0000 7fb5fbfff640 1 --2- 172.21.15.1:0/2528220710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6040c5e10 0x7fb60406e090 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0097a0 tx=0x7fb5ec02a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e93a4eddc139e5e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.625+0000 7fb6027fc640 1 -- 172.21.15.1:0/2528220710 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec037070 con 0x7fb6040c5e10 2024-05-09T05:09:35.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.625+0000 7fb6027fc640 1 -- 172.21.15.1:0/2528220710 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5ec032440 con 0x7fb6040c5e10 2024-05-09T05:09:35.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.625+0000 7fb6027fc640 1 -- 172.21.15.1:0/2528220710 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec032810 con 0x7fb6040c5e10 2024-05-09T05:09:35.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.626+0000 7fb6096a8640 1 -- 172.21.15.1:0/2528220710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6040c5e10 msgr2=0x7fb60406e090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.626+0000 7fb6096a8640 1 --2- 172.21.15.1:0/2528220710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6040c5e10 0x7fb60406e090 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec0097a0 tx=0x7fb5ec02a210 comp rx=0 tx=0).stop 2024-05-09T05:09:35.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.626+0000 7fb6096a8640 1 -- 172.21.15.1:0/2528220710 shutdown_connections 2024-05-09T05:09:35.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.626+0000 7fb6096a8640 1 --2- 172.21.15.1:0/2528220710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60415c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.626+0000 7fb6096a8640 1 --2- 172.21.15.1:0/2528220710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6040c5e10 0x7fb60406e090 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.626+0000 7fb6096a8640 1 -- 172.21.15.1:0/2528220710 >> 172.21.15.1:0/2528220710 conn(0x7fb6040ce910 msgr2=0x7fb6040ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:35.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.626+0000 7fb6096a8640 1 -- 172.21.15.1:0/2528220710 shutdown_connections 2024-05-09T05:09:35.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.626+0000 7fb6096a8640 1 -- 172.21.15.1:0/2528220710 wait complete. 2024-05-09T05:09:35.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.628+0000 7fb6096a8640 1 Processor -- start 2024-05-09T05:09:35.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.628+0000 7fb6096a8640 1 -- start start 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb6096a8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60406db90 0x7fb6041f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb6096a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6040c5e10 0x7fb6040cda40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb6096a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60414e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb6096a8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb604070340 con 0x7fb6040c5e10 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb6096a8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb604156d20 con 0x7fb60406db90 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb6096a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb6040704c0 con 0x7fb604156910 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb5fbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6040c5e10 0x7fb6040cda40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb5fbfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6040c5e10 0x7fb6040cda40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53454/0 (socket says 172.21.15.1:53454) 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb5fbfff640 1 -- 172.21.15.1:0/1367155357 learned_addr learned my addr 172.21.15.1:0/1367155357 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb602ffd640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60406db90 0x7fb6041f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.630+0000 7fb6037fe640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60414e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb602ffd640 1 -- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 msgr2=0x7fb60414e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb602ffd640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb604156910 0x7fb60414e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb602ffd640 1 -- 172.21.15.1:0/1367155357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6040c5e10 msgr2=0x7fb6040cda40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb602ffd640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6040c5e10 0x7fb6040cda40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb602ffd640 1 -- 172.21.15.1:0/1367155357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6041f6dd0 con 0x7fb60406db90 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb5fbfff640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb6040c5e10 0x7fb6040cda40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb602ffd640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60406db90 0x7fb6041f2c50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c390 tx=0x7fb5f400c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f40074e0 con 0x7fb60406db90 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb5f40047e0 con 0x7fb60406db90 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5f4004bd0 con 0x7fb60406db90 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb6096a8640 1 -- 172.21.15.1:0/1367155357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6041f3190 con 0x7fb60406db90 2024-05-09T05:09:35.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.631+0000 7fb6096a8640 1 -- 172.21.15.1:0/1367155357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6041f61c0 con 0x7fb60406db90 2024-05-09T05:09:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.632+0000 7fb6096a8640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60415de50 0x7fb60414e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.632+0000 7fb6096a8640 1 -- 172.21.15.1:0/1367155357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb6041f5f70 con 0x7fb60415de50 2024-05-09T05:09:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.632+0000 7fb5fbfff640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60415de50 0x7fb60414e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.633+0000 7fb5fbfff640 1 -- 172.21.15.1:0/1367155357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb60406e6b0 con 0x7fb60415de50 2024-05-09T05:09:35.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.633+0000 7fb5fbfff640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60415de50 0x7fb60414e6a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec02a250 tx=0x7fb5ec0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.633+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec037040 con 0x7fb60415de50 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.633+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb5f401a020 con 0x7fb60406db90 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.634+0000 7fb5fb7fe640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb5e40761d0 0x7fb5e40786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.634+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb5f4096da0 con 0x7fb60406db90 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.634+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb5ec0093c0 con 0x7fb60415de50 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.634+0000 7fb6037fe640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb5e40761d0 0x7fb5e40786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.634+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60415de50 msgr2=0x7fb60414e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.634+0000 7fb5fb7fe640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60415de50 0x7fb60414e6a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb5ec02a250 tx=0x7fb5ec0046c0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.634+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60415de50 msgr2=0x7fb60414e6a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.634+0000 7fb5fb7fe640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60415de50 0x7fb60414e6a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.635+0000 7fb6037fe640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb5e40761d0 0x7fb5e40786e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb60415c4b0 tx=0x7fb5fc00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.641+0000 7fb6096a8640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6041f3360 0x7fb6041f64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.641+0000 7fb6096a8640 1 -- 172.21.15.1:0/1367155357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb604050870 con 0x7fb6041f3360 2024-05-09T05:09:35.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.641+0000 7fb5fbfff640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6041f3360 0x7fb6041f64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.641+0000 7fb5fbfff640 1 -- 172.21.15.1:0/1367155357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb6041f29b0 con 0x7fb6041f3360 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.642+0000 7fb5fbfff640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6041f3360 0x7fb6041f64a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb604153e40 tx=0x7fb5ec009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.642+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb5ec0473f0 con 0x7fb6041f3360 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.642+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7fb5ec03d3f0 con 0x7fb6041f3360 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.642+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6041f3360 msgr2=0x7fb6041f64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.642+0000 7fb5fb7fe640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6041f3360 0x7fb6041f64a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fb604153e40 tx=0x7fb5ec009190 comp rx=0 tx=0).stop 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.642+0000 7fb5fb7fe640 1 -- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6041f3360 msgr2=0x7fb6041f64a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.642+0000 7fb5fb7fe640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb6041f3360 0x7fb6041f64a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:35.643 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19215, 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 13, 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:09:35.646 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:35.647 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:09:35.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:35.650 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.653 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.655 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:09:35.655 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb5f97fa640 1 -- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb5e40761d0 msgr2=0x7fb5e40786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb5f97fa640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb5e40761d0 0x7fb5e40786e0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fb60415c4b0 tx=0x7fb5fc00b040 comp rx=0 tx=0).stop 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb5f97fa640 1 -- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60406db90 msgr2=0x7fb6041f2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb5f97fa640 1 --2- 172.21.15.1:0/1367155357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb60406db90 0x7fb6041f2c50 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb5f400c390 tx=0x7fb5f400c850 comp rx=0 tx=0).stop 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb6037fe640 1 -- 172.21.15.1:0/1367155357 reap_dead start 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb6037fe640 1 -- 172.21.15.1:0/1367155357 reap_dead start 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb5f97fa640 1 -- 172.21.15.1:0/1367155357 shutdown_connections 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb5f97fa640 1 -- 172.21.15.1:0/1367155357 >> 172.21.15.1:0/1367155357 conn(0x7fb6040ce910 msgr2=0x7fb60406fc30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb5f97fa640 1 -- 172.21.15.1:0/1367155357 shutdown_connections 2024-05-09T05:09:35.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.643+0000 7fb5f97fa640 1 -- 172.21.15.1:0/1367155357 wait complete. 2024-05-09T05:09:35.657 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:09:35.657 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:09:35.657 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:09:35.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.783+0000 7f9f1eaa8640 1 Processor -- start 2024-05-09T05:09:35.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.783+0000 7f9f1eaa8640 1 -- start start 2024-05-09T05:09:35.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1eaa8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9f180c9f40 0x7f9f18154ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1eaa8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 0x7f9f180c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1eaa8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9f18070350 con 0x7f9f18070cb0 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1eaa8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9f180705f0 con 0x7f9f180c9f40 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1eaa8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9f18070890 con 0x7f9f180c9490 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1c844640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9f180c9f40 0x7f9f18154ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1c844640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f9f180c9f40 0x7f9f18154ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53460/0 (socket says 172.21.15.1:53460) 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1c844640 1 -- 172.21.15.1:0/3932815482 learned_addr learned my addr 172.21.15.1:0/3932815482 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f17fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9f18070cb0 0x7f9f180c7890 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42234/0 (socket says 172.21.15.1:42234) 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.785+0000 7f9f1d045640 1 --2- 172.21.15.1:0/3932815482 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 0x7f9f180c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.786+0000 7f9f1c844640 1 -- 172.21.15.1:0/3932815482 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 msgr2=0x7f9f180c98a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:35.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.786+0000 7f9f1c844640 1 --2- 172.21.15.1:0/3932815482 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 0x7f9f180c98a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.786+0000 7f9f1c844640 1 -- 172.21.15.1:0/3932815482 >> v1:172.21.15.1:6789/0 conn(0x7f9f18070cb0 legacy=0x7f9f180c7890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.786+0000 7f9f1c844640 1 -- 172.21.15.1:0/3932815482 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f1806c820 con 0x7f9f180c9f40 2024-05-09T05:09:35.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.786+0000 7f9f177fe640 1 -- 172.21.15.1:0/3932815482 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4130682535 0 0) 0x7f9f18070350 con 0x7f9f18070cb0 2024-05-09T05:09:35.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.786+0000 7f9f1c844640 1 --2- 172.21.15.1:0/3932815482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f180c9f40 0x7f9f18154ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9f18070a60 tx=0x7f9f0c02a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c2144386f86cf197 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.787+0000 7f9f177fe640 1 -- 172.21.15.1:0/3932815482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c037070 con 0x7f9f180c9f40 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.787+0000 7f9f177fe640 1 -- 172.21.15.1:0/3932815482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0c032440 con 0x7f9f180c9f40 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.787+0000 7f9f177fe640 1 -- 172.21.15.1:0/3932815482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c032810 con 0x7f9f180c9f40 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.787+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/3932815482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f180c9f40 msgr2=0x7f9f18154ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.787+0000 7f9f1eaa8640 1 --2- 172.21.15.1:0/3932815482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f180c9f40 0x7f9f18154ac0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f9f18070a60 tx=0x7f9f0c02a250 comp rx=0 tx=0).stop 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.787+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/3932815482 shutdown_connections 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.787+0000 7f9f1eaa8640 1 --2- 172.21.15.1:0/3932815482 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 0x7f9f180c98a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.787+0000 7f9f1eaa8640 1 --2- 172.21.15.1:0/3932815482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f180c9f40 0x7f9f18154ac0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.788+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/3932815482 >> 172.21.15.1:0/3932815482 conn(0x7f9f180cb590 msgr2=0x7f9f180cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.788+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/3932815482 shutdown_connections 2024-05-09T05:09:35.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.788+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/3932815482 wait complete. 2024-05-09T05:09:35.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.790+0000 7f9f1eaa8640 1 Processor -- start 2024-05-09T05:09:35.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.790+0000 7f9f1eaa8640 1 -- start start 2024-05-09T05:09:35.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1eaa8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f18070cb0 0x7f9f180c5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1eaa8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 0x7f9f180b3960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1eaa8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f180c9f40 0x7f9f18153380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1eaa8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9f180c7fa0 con 0x7f9f18070cb0 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1eaa8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9f1806cde0 con 0x7f9f180c9f40 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1eaa8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9f180c8120 con 0x7f9f180c9490 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1c844640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f18070cb0 0x7f9f180c5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1c844640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f18070cb0 0x7f9f180c5570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53464/0 (socket says 172.21.15.1:53464) 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1c844640 1 -- 172.21.15.1:0/2407302637 learned_addr learned my addr 172.21.15.1:0/2407302637 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f17fff640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 0x7f9f180b3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1d045640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f180c9f40 0x7f9f18153380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1c844640 1 -- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 msgr2=0x7f9f180b3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1c844640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 0x7f9f180b3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1c844640 1 -- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f180c9f40 msgr2=0x7f9f18153380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1c844640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f180c9f40 0x7f9f18153380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1c844640 1 -- 172.21.15.1:0/2407302637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9f180c3b40 con 0x7f9f18070cb0 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f17fff640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180c9490 0x7f9f180b3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.792+0000 7f9f1d045640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9f180c9f40 0x7f9f18153380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.793+0000 7f9f1c844640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f18070cb0 0x7f9f180c5570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0090b0 tx=0x7f9f0c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.793+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c037040 con 0x7f9f18070cb0 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.793+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9f0c004230 con 0x7f9f18070cb0 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.793+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f0c003610 con 0x7f9f18070cb0 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.793+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9f180b42a0 con 0x7f9f18070cb0 2024-05-09T05:09:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.793+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9f180c4d70 con 0x7f9f18070cb0 2024-05-09T05:09:35.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.794+0000 7f9f1eaa8640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180cc950 0x7f9f1815b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.794+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9f18153030 con 0x7f9f180cc950 2024-05-09T05:09:35.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.794+0000 7f9f17fff640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180cc950 0x7f9f1815b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.794+0000 7f9f17fff640 1 -- 172.21.15.1:0/2407302637 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9f180ce3e0 con 0x7f9f180cc950 2024-05-09T05:09:35.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.795+0000 7f9f17fff640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180cc950 0x7f9f1815b950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f18158610 tx=0x7f9f0400d950 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.795+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04018070 con 0x7f9f180cc950 2024-05-09T05:09:35.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.795+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9f0c0037b0 con 0x7f9f18070cb0 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.795+0000 7f9f157fa640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9efc076200 0x7f9efc078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.796+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9f0c0b8400 con 0x7f9f18070cb0 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.796+0000 7f9f1d045640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9efc076200 0x7f9efc078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.796+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9f040074e0 con 0x7f9f180cc950 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.796+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180cc950 msgr2=0x7f9f1815b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.796+0000 7f9f157fa640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180cc950 0x7f9f1815b950 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9f18158610 tx=0x7f9f0400d950 comp rx=0 tx=0).stop 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.796+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180cc950 msgr2=0x7f9f1815b950 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.796+0000 7f9f157fa640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f180cc950 0x7f9f1815b950 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.796+0000 7f9f1d045640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9efc076200 0x7f9efc078710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9f18154710 tx=0x7f9f1000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.804+0000 7f9f1eaa8640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f181538c0 0x7f9f18153cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:35.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.804+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9f18154210 con 0x7f9f181538c0 2024-05-09T05:09:35.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.804+0000 7f9f17fff640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f181538c0 0x7f9f18153cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:35.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.804+0000 7f9f17fff640 1 -- 172.21.15.1:0/2407302637 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9f180c4b80 con 0x7f9f181538c0 2024-05-09T05:09:35.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.804+0000 7f9f17fff640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f181538c0 0x7f9f18153cd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9f04007fd0 tx=0x7f9f04001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:35.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9f04007690 con 0x7f9f181538c0 2024-05-09T05:09:35.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f9f04013470 con 0x7f9f181538c0 2024-05-09T05:09:35.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f181538c0 msgr2=0x7f9f18153cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f157fa640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f181538c0 0x7f9f18153cd0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9f04007fd0 tx=0x7f9f04001080 comp rx=0 tx=0).stop 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f157fa640 1 -- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f181538c0 msgr2=0x7f9f18153cd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f157fa640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9f181538c0 0x7f9f18153cd0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 30, 2024-05-09T05:09:35.806 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 133803, 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 13, 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.807 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.808 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:09:35.809 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:35.810 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.811 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:09:35.814 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.815 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.816 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.817 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.818 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:09:35.819 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9efc076200 msgr2=0x7f9efc078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f1eaa8640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9efc076200 0x7f9efc078710 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f9f18154710 tx=0x7f9f1000b040 comp rx=0 tx=0).stop 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f18070cb0 msgr2=0x7f9f180c5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f1eaa8640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f18070cb0 0x7f9f180c5570 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9f0c0090b0 tx=0x7f9f0c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.805+0000 7f9f1d045640 1 -- 172.21.15.1:0/2407302637 reap_dead start 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.806+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 shutdown_connections 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.806+0000 7f9f1eaa8640 1 --2- 172.21.15.1:0/2407302637 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9f18070cb0 0x7f9f180c5570 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.806+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 >> 172.21.15.1:0/2407302637 conn(0x7f9f180cb590 msgr2=0x7f9f180c82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.806+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 shutdown_connections 2024-05-09T05:09:35.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:35.806+0000 7f9f1eaa8640 1 -- 172.21.15.1:0/2407302637 wait complete. 2024-05-09T05:09:35.822 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:09:35.822 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-05-09T05:09:35.822 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:09:35.822 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-05-09T05:09:35.822 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-05-09T05:09:35.822 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T05:09:35.822 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:09:35.838 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T05:09:35.838 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:09:35.838 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:09:36.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.005+0000 7f6187bb7640 1 Processor -- start 2024-05-09T05:09:36.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.005+0000 7f6187bb7640 1 -- start start 2024-05-09T05:09:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6187bb7640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f61800c49f0 0x7f61800cbd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6187bb7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c5060 0x7f61801512d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6187bb7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61800cafb0 0x7f61800cb3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:36.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6187bb7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f618015eb40 con 0x7f61800cafb0 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6187bb7640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f618015dcb0 con 0x7f61800c49f0 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6187bb7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f61800c5470 con 0x7f61800c5060 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6186154640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7f61800cafb0 msgr2=0x7f61800cb3c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6186154640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f61800cafb0 0x7f61800cb3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6185152640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c5060 0x7f61801512d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6185152640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c5060 0x7f61801512d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:33538/0 (socket says 172.21.15.1:33538) 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6185953640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f61800c49f0 0x7f61800cbd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.008+0000 7f6185152640 1 -- 172.21.15.1:0/210647115 learned_addr learned my addr 172.21.15.1:0/210647115 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.009+0000 7f6185152640 1 -- 172.21.15.1:0/210647115 >> v2:172.21.15.196:3300/0 conn(0x7f61800c49f0 msgr2=0x7f61800cbd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.009+0000 7f6185152640 1 --2- 172.21.15.1:0/210647115 >> v2:172.21.15.196:3300/0 conn(0x7f61800c49f0 0x7f61800cbd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.009+0000 7f6185152640 1 -- 172.21.15.1:0/210647115 >> v2:172.21.15.1:3300/0 conn(0x7f61800cafb0 msgr2=0x7f61800cb3c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.009+0000 7f6185152640 1 --2- 172.21.15.1:0/210647115 >> v2:172.21.15.1:3300/0 conn(0x7f61800cafb0 0x7f61800cb3c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.009+0000 7f6185152640 1 -- 172.21.15.1:0/210647115 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f618015df30 con 0x7f61800c5060 2024-05-09T05:09:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.009+0000 7f6185953640 1 --2- 172.21.15.1:0/210647115 >> v2:172.21.15.196:3300/0 conn(0x7f61800c49f0 0x7f61800cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.010+0000 7f6185152640 1 --2- 172.21.15.1:0/210647115 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c5060 0x7f61801512d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f61800ca7f0 tx=0x7f617002a200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e8b451b78a16a35 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.010+0000 7f6184951640 1 -- 172.21.15.1:0/210647115 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170037070 con 0x7f61800c5060 2024-05-09T05:09:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.010+0000 7f6184951640 1 -- 172.21.15.1:0/210647115 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6170032440 con 0x7f61800c5060 2024-05-09T05:09:36.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.010+0000 7f6184951640 1 -- 172.21.15.1:0/210647115 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170032810 con 0x7f61800c5060 2024-05-09T05:09:36.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.011+0000 7f6187bb7640 1 -- 172.21.15.1:0/210647115 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c5060 msgr2=0x7f61801512d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:36.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.011+0000 7f6187bb7640 1 --2- 172.21.15.1:0/210647115 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c5060 0x7f61801512d0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f61800ca7f0 tx=0x7f617002a200 comp rx=0 tx=0).stop 2024-05-09T05:09:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.011+0000 7f6187bb7640 1 -- 172.21.15.1:0/210647115 shutdown_connections 2024-05-09T05:09:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.011+0000 7f6187bb7640 1 --2- 172.21.15.1:0/210647115 >> v2:172.21.15.1:3300/0 conn(0x7f61800cafb0 0x7f61800cb3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.011+0000 7f6187bb7640 1 --2- 172.21.15.1:0/210647115 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c5060 0x7f61801512d0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.011+0000 7f6187bb7640 1 --2- 172.21.15.1:0/210647115 >> v2:172.21.15.196:3300/0 conn(0x7f61800c49f0 0x7f61800cbd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.011+0000 7f6187bb7640 1 -- 172.21.15.1:0/210647115 >> 172.21.15.1:0/210647115 conn(0x7f6180068830 msgr2=0x7f6180068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.012+0000 7f6187bb7640 1 -- 172.21.15.1:0/210647115 shutdown_connections 2024-05-09T05:09:36.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.012+0000 7f6187bb7640 1 -- 172.21.15.1:0/210647115 wait complete. 2024-05-09T05:09:36.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.014+0000 7f6187bb7640 1 Processor -- start 2024-05-09T05:09:36.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.014+0000 7f6187bb7640 1 -- start start 2024-05-09T05:09:36.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.015+0000 7f6187bb7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c49f0 0x7f61801f1c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:36.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6187bb7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61800c5060 0x7f618014fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:36.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6187bb7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800cafb0 0x7f6180147170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:36.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6187bb7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f618015e7d0 con 0x7f61800cafb0 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6187bb7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f618015e4f0 con 0x7f61800c5060 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6187bb7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f618015a900 con 0x7f61800c49f0 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6186154640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800cafb0 msgr2=0x7f6180147170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6186154640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800cafb0 0x7f6180147170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185953640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c49f0 0x7f61801f1c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185152640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61800c5060 0x7f618014fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185953640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c49f0 0x7f61801f1c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:33542/0 (socket says 172.21.15.1:33542) 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185953640 1 -- 172.21.15.1:0/904096222 learned_addr learned my addr 172.21.15.1:0/904096222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185152640 1 -- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c49f0 msgr2=0x7f61801f1c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:36.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185152640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c49f0 0x7f61801f1c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185152640 1 -- 172.21.15.1:0/904096222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800cafb0 msgr2=0x7f6180147170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185152640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800cafb0 0x7f6180147170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185152640 1 -- 172.21.15.1:0/904096222 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f61801f2190 con 0x7f61800c5060 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.016+0000 7f6185953640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c49f0 0x7f61801f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.017+0000 7f6185152640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61800c5060 0x7f618014fe20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6170031000 tx=0x7f61700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.017+0000 7f61767fc640 1 -- 172.21.15.1:0/904096222 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170041020 con 0x7f61800c5060 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.017+0000 7f61767fc640 1 -- 172.21.15.1:0/904096222 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6170030640 con 0x7f61800c5060 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.017+0000 7f61767fc640 1 -- 172.21.15.1:0/904096222 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6170037040 con 0x7f61800c5060 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.017+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f61801e52a0 con 0x7f61800c5060 2024-05-09T05:09:36.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.017+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f61801e5820 con 0x7f61800c5060 2024-05-09T05:09:36.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.018+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6180050870 con 0x7f61800c5060 2024-05-09T05:09:36.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.019+0000 7f61767fc640 1 -- 172.21.15.1:0/904096222 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6170044020 con 0x7f61800c5060 2024-05-09T05:09:36.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.019+0000 7f61767fc640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f61640762b0 0x7f61640787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:36.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.019+0000 7f61767fc640 1 -- 172.21.15.1:0/904096222 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f61700b7990 con 0x7f61800c5060 2024-05-09T05:09:36.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.020+0000 7f6185953640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f61640762b0 0x7f61640787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:36.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.020+0000 7f6185953640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f61640762b0 0x7f61640787c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f617c0095e0 tx=0x7f617c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:36.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.025+0000 7f61767fc640 1 -- 172.21.15.1:0/904096222 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f617004b070 con 0x7f61800c5060 2024-05-09T05:09:36.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.311+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f61800cbaf0 con 0x7f61800c5060 2024-05-09T05:09:36.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.311+0000 7f61767fc640 1 -- 172.21.15.1:0/904096222 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f6170081300 con 0x7f61800c5060 2024-05-09T05:09:36.312 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":14,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:09:36.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f61640762b0 msgr2=0x7f61640787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f61640762b0 0x7f61640787c0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f617c0095e0 tx=0x7f617c009190 comp rx=0 tx=0).stop 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61800c5060 msgr2=0x7f618014fe20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61800c5060 0x7f618014fe20 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f6170031000 tx=0x7f61700046c0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 shutdown_connections 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f61640762b0 0x7f61640787c0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f61800c5060 0x7f618014fe20 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f61800cafb0 0x7f6180147170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 --2- 172.21.15.1:0/904096222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f61800c49f0 0x7f61801f1c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 >> 172.21.15.1:0/904096222 conn(0x7f6180068830 msgr2=0x7f61800b4530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 shutdown_connections 2024-05-09T05:09:36.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:36.314+0000 7f6187bb7640 1 -- 172.21.15.1:0/904096222 wait complete. 2024-05-09T05:09:39.326 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:09:39.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.444+0000 7f0b99784640 1 Processor -- start 2024-05-09T05:09:39.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.445+0000 7f0b99784640 1 -- start start 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.446+0000 7f0b99784640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0b940c78e0 0x7f0b94156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.446+0000 7f0b99784640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b940c53c0 0x7f0b9415d230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b99784640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0b94073580 0x7f0b94073990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b99784640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f0b940749b0 con 0x7f0b940c78e0 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b99784640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0b940c6b00 con 0x7f0b94073580 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b99784640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9415ca70 con 0x7f0b940c53c0 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b92ffd640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7f0b940c78e0 msgr2=0x7f0b94156bc0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b92ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0b940c78e0 0x7f0b94156bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b8bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b940c53c0 0x7f0b9415d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:39.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b937fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0b94073580 0x7f0b94073990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b8bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b940c53c0 0x7f0b9415d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:33556/0 (socket says 172.21.15.1:33556) 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b937fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0b94073580 0x7f0b94073990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:42576/0 (socket says 172.21.15.1:42576) 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b8bfff640 1 -- 172.21.15.1:0/3655411995 learned_addr learned my addr 172.21.15.1:0/3655411995 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b937fe640 1 -- 172.21.15.1:0/3655411995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b940c53c0 msgr2=0x7f0b9415d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b937fe640 1 --2- 172.21.15.1:0/3655411995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b940c53c0 0x7f0b9415d230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b937fe640 1 -- 172.21.15.1:0/3655411995 >> v2:172.21.15.1:3300/0 conn(0x7f0b940c78e0 msgr2=0x7f0b94156bc0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b937fe640 1 --2- 172.21.15.1:0/3655411995 >> v2:172.21.15.1:3300/0 conn(0x7f0b940c78e0 0x7f0b94156bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b937fe640 1 -- 172.21.15.1:0/3655411995 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b940c8d60 con 0x7f0b94073580 2024-05-09T05:09:39.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.447+0000 7f0b8bfff640 1 --2- 172.21.15.1:0/3655411995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b940c53c0 0x7f0b9415d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:39.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.448+0000 7f0b937fe640 1 --2- 172.21.15.1:0/3655411995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b94073580 0x7f0b94073990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0b94160ef0 tx=0x7f0b8c02a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bc258e4c1aaf15cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:39.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b927fc640 1 -- 172.21.15.1:0/3655411995 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c007740 con 0x7f0b94073580 2024-05-09T05:09:39.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b927fc640 1 -- 172.21.15.1:0/3655411995 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8c0078e0 con 0x7f0b94073580 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b927fc640 1 -- 172.21.15.1:0/3655411995 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c007cb0 con 0x7f0b94073580 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b99784640 1 -- 172.21.15.1:0/3655411995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b94073580 msgr2=0x7f0b94073990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b99784640 1 --2- 172.21.15.1:0/3655411995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b94073580 0x7f0b94073990 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f0b94160ef0 tx=0x7f0b8c02a5f0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b99784640 1 -- 172.21.15.1:0/3655411995 shutdown_connections 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b99784640 1 --2- 172.21.15.1:0/3655411995 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b94073580 0x7f0b94073990 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b99784640 1 --2- 172.21.15.1:0/3655411995 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b940c53c0 0x7f0b9415d230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b99784640 1 --2- 172.21.15.1:0/3655411995 >> v2:172.21.15.1:3300/0 conn(0x7f0b940c78e0 0x7f0b94156bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.449+0000 7f0b99784640 1 -- 172.21.15.1:0/3655411995 >> 172.21.15.1:0/3655411995 conn(0x7f0b940caa40 msgr2=0x7f0b940cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.450+0000 7f0b99784640 1 -- 172.21.15.1:0/3655411995 shutdown_connections 2024-05-09T05:09:39.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.450+0000 7f0b99784640 1 -- 172.21.15.1:0/3655411995 wait complete. 2024-05-09T05:09:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.452+0000 7f0b99784640 1 Processor -- start 2024-05-09T05:09:39.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.452+0000 7f0b99784640 1 -- start start 2024-05-09T05:09:39.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.453+0000 7f0b99784640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b94073580 0x7f0b941f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:39.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.453+0000 7f0b99784640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b940c53c0 0x7f0b9415aeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:39.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.453+0000 7f0b99784640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b940c78e0 0x7f0b94146950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:39.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.453+0000 7f0b99784640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0b9406df00 con 0x7f0b940c53c0 2024-05-09T05:09:39.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.453+0000 7f0b99784640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0b940746d0 con 0x7f0b940c78e0 2024-05-09T05:09:39.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.453+0000 7f0b99784640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0b9415d770 con 0x7f0b94073580 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.453+0000 7f0b8bfff640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b940c53c0 msgr2=0x7f0b9415aeb0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.453+0000 7f0b8bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b940c53c0 0x7f0b9415aeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b940c78e0 0x7f0b94146950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b940c78e0 0x7f0b94146950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:42586/0 (socket says 172.21.15.1:42586) 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 -- 172.21.15.1:0/4087701536 learned_addr learned my addr 172.21.15.1:0/4087701536 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b92ffd640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b94073580 0x7f0b941f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 -- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b94073580 msgr2=0x7f0b941f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b94073580 0x7f0b941f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 -- 172.21.15.1:0/4087701536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b940c53c0 msgr2=0x7f0b9415aeb0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b940c53c0 0x7f0b9415aeb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 -- 172.21.15.1:0/4087701536 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b9415b3f0 con 0x7f0b940c78e0 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.454+0000 7f0b937fe640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b940c78e0 0x7f0b94146950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c02abe0 tx=0x7f0b8c004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.455+0000 7f0b8b7fe640 1 -- 172.21.15.1:0/4087701536 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c03f070 con 0x7f0b940c78e0 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.455+0000 7f0b8b7fe640 1 -- 172.21.15.1:0/4087701536 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0b8c031bc0 con 0x7f0b940c78e0 2024-05-09T05:09:39.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.455+0000 7f0b8b7fe640 1 -- 172.21.15.1:0/4087701536 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b8c009410 con 0x7f0b940c78e0 2024-05-09T05:09:39.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.455+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b940cc430 con 0x7f0b940c78e0 2024-05-09T05:09:39.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.455+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b940cc9f0 con 0x7f0b940c78e0 2024-05-09T05:09:39.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.456+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b94050870 con 0x7f0b940c78e0 2024-05-09T05:09:39.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.456+0000 7f0b8b7fe640 1 -- 172.21.15.1:0/4087701536 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0b8c044460 con 0x7f0b940c78e0 2024-05-09T05:09:39.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.457+0000 7f0b8b7fe640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0b740762b0 0x7f0b740787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:39.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.457+0000 7f0b8b7fe640 1 -- 172.21.15.1:0/4087701536 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0b8c0b8d40 con 0x7f0b940c78e0 2024-05-09T05:09:39.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.457+0000 7f0b92ffd640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0b740762b0 0x7f0b740787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:39.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.458+0000 7f0b92ffd640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0b740762b0 0x7f0b740787c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0b84003980 tx=0x7f0b84009110 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:39.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.462+0000 7f0b8b7fe640 1 -- 172.21.15.1:0/4087701536 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0b8c081fd0 con 0x7f0b940c78e0 2024-05-09T05:09:39.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.813+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0b9406dce0 con 0x7f0b940c78e0 2024-05-09T05:09:39.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.814+0000 7f0b8b7fe640 1 -- 172.21.15.1:0/4087701536 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0b8c0862d0 con 0x7f0b940c78e0 2024-05-09T05:09:39.814 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0b740762b0 msgr2=0x7f0b740787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0b740762b0 0x7f0b740787c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0b84003980 tx=0x7f0b84009110 comp rx=0 tx=0).stop 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b940c78e0 msgr2=0x7f0b94146950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b940c78e0 0x7f0b94146950 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0b8c02abe0 tx=0x7f0b8c004a40 comp rx=0 tx=0).stop 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 shutdown_connections 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0b740762b0 0x7f0b740787c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0b940c78e0 0x7f0b94146950 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0b940c53c0 0x7f0b9415aeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 --2- 172.21.15.1:0/4087701536 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0b94073580 0x7f0b941f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 >> 172.21.15.1:0/4087701536 conn(0x7f0b940caa40 msgr2=0x7f0b94075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 shutdown_connections 2024-05-09T05:09:39.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:39.816+0000 7f0b99784640 1 -- 172.21.15.1:0/4087701536 wait complete. 2024-05-09T05:09:42.830 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:09:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.952+0000 7fb785fa8640 1 Processor -- start 2024-05-09T05:09:42.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.952+0000 7fb785fa8640 1 -- start start 2024-05-09T05:09:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb785fa8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7800ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb785fa8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb780156e90 0x7fb7800c7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb785fa8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb7800c86a0 con 0x7fb780156a80 2024-05-09T05:09:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb785fa8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb7800cb910 con 0x7fb780156e90 2024-05-09T05:09:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb785fa8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb7800c8980 con 0x7fb7800c5ca0 2024-05-09T05:09:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb77ffff640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7fb780156e90 msgr2=0x7fb7800c7750 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb77ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb780156e90 0x7fb7800c7750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:42.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb77effd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fb780156a80 0x7fb78006df20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:59768/0 (socket says 172.21.15.1:59768) 2024-05-09T05:09:42.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb77effd640 1 -- 172.21.15.1:0/2126721864 learned_addr learned my addr 172.21.15.1:0/2126721864 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:42.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.954+0000 7fb77f7fe640 1 --2- 172.21.15.1:0/2126721864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7800ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:42.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.955+0000 7fb77f7fe640 1 -- 172.21.15.1:0/2126721864 >> v2:172.21.15.1:3300/0 conn(0x7fb780156e90 msgr2=0x7fb7800c7750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:42.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.955+0000 7fb77f7fe640 1 --2- 172.21.15.1:0/2126721864 >> v2:172.21.15.1:3300/0 conn(0x7fb780156e90 0x7fb7800c7750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:42.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.955+0000 7fb77f7fe640 1 -- 172.21.15.1:0/2126721864 >> v1:172.21.15.196:6789/0 conn(0x7fb780156a80 legacy=0x7fb78006df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:42.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.955+0000 7fb77f7fe640 1 -- 172.21.15.1:0/2126721864 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7800c8c60 con 0x7fb7800c5ca0 2024-05-09T05:09:42.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.956+0000 7fb77f7fe640 1 --2- 172.21.15.1:0/2126721864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7800ca4a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb7700097a0 tx=0x7fb77002a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fdffb5db45fd376 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:42.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.956+0000 7fb77e7fc640 1 -- 172.21.15.1:0/2126721864 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770037070 con 0x7fb7800c5ca0 2024-05-09T05:09:42.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.956+0000 7fb77e7fc640 1 -- 172.21.15.1:0/2126721864 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb770032440 con 0x7fb7800c5ca0 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.956+0000 7fb77e7fc640 1 -- 172.21.15.1:0/2126721864 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770032810 con 0x7fb7800c5ca0 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.957+0000 7fb785fa8640 1 -- 172.21.15.1:0/2126721864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 msgr2=0x7fb7800ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.957+0000 7fb785fa8640 1 --2- 172.21.15.1:0/2126721864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7800ca4a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fb7700097a0 tx=0x7fb77002a230 comp rx=0 tx=0).stop 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.957+0000 7fb785fa8640 1 -- 172.21.15.1:0/2126721864 shutdown_connections 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.957+0000 7fb785fa8640 1 --2- 172.21.15.1:0/2126721864 >> v2:172.21.15.1:3300/0 conn(0x7fb780156e90 0x7fb7800c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.957+0000 7fb785fa8640 1 --2- 172.21.15.1:0/2126721864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7800ca4a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.957+0000 7fb785fa8640 1 -- 172.21.15.1:0/2126721864 >> 172.21.15.1:0/2126721864 conn(0x7fb7800745a0 msgr2=0x7fb780071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.957+0000 7fb785fa8640 1 -- 172.21.15.1:0/2126721864 shutdown_connections 2024-05-09T05:09:42.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.957+0000 7fb785fa8640 1 -- 172.21.15.1:0/2126721864 wait complete. 2024-05-09T05:09:42.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.959+0000 7fb785fa8640 1 Processor -- start 2024-05-09T05:09:42.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.959+0000 7fb785fa8640 1 -- start start 2024-05-09T05:09:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb785fa8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7801547a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb785fa8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb780156a80 0x7fb780155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb785fa8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb780156e90 0x7fb78014b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb785fa8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb7800cad60 con 0x7fb780156a80 2024-05-09T05:09:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb785fa8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb7800cb5a0 con 0x7fb780156e90 2024-05-09T05:09:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb785fa8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb7800ca9e0 con 0x7fb7800c5ca0 2024-05-09T05:09:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb77effd640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb780156a80 msgr2=0x7fb780155170 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb77effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb780156a80 0x7fb780155170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb77f7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7801547a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.960+0000 7fb77ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb780156e90 0x7fb78014b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb780156e90 0x7fb78014b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33358/0 (socket says 172.21.15.1:33358) 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77f7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7801547a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:41744/0 (socket says 172.21.15.1:41744) 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77ffff640 1 -- 172.21.15.1:0/3770403609 learned_addr learned my addr 172.21.15.1:0/3770403609 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77f7fe640 1 -- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb780156e90 msgr2=0x7fb78014b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77f7fe640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb780156e90 0x7fb78014b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77f7fe640 1 -- 172.21.15.1:0/3770403609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb780156a80 msgr2=0x7fb780155170 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77f7fe640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb780156a80 0x7fb780155170 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77f7fe640 1 -- 172.21.15.1:0/3770403609 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7800ccbd0 con 0x7fb7800c5ca0 2024-05-09T05:09:42.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77ffff640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb780156e90 0x7fb78014b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb77f7fe640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7801547a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb770031000 tx=0x7fb7700046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb767fff640 1 -- 172.21.15.1:0/3770403609 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770041070 con 0x7fb7800c5ca0 2024-05-09T05:09:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.961+0000 7fb767fff640 1 -- 172.21.15.1:0/3770403609 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb77003c930 con 0x7fb7800c5ca0 2024-05-09T05:09:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.962+0000 7fb767fff640 1 -- 172.21.15.1:0/3770403609 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb770037040 con 0x7fb7800c5ca0 2024-05-09T05:09:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.962+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7800cd300 con 0x7fb7800c5ca0 2024-05-09T05:09:42.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.962+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7800cd8c0 con 0x7fb7800c5ca0 2024-05-09T05:09:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.963+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb780050870 con 0x7fb7800c5ca0 2024-05-09T05:09:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.963+0000 7fb767fff640 1 -- 172.21.15.1:0/3770403609 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb770044070 con 0x7fb7800c5ca0 2024-05-09T05:09:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.964+0000 7fb767fff640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb76c076280 0x7fb76c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.964+0000 7fb767fff640 1 -- 172.21.15.1:0/3770403609 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb7700b8160 con 0x7fb7800c5ca0 2024-05-09T05:09:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.964+0000 7fb77effd640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb76c076280 0x7fb76c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:42.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.965+0000 7fb77effd640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb76c076280 0x7fb76c078790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb7801f0690 tx=0x7fb774007ae0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:42.969+0000 7fb767fff640 1 -- 172.21.15.1:0/3770403609 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb770081360 con 0x7fb7800c5ca0 2024-05-09T05:09:43.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.267+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb7800cb000 con 0x7fb7800c5ca0 2024-05-09T05:09:43.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.268+0000 7fb767fff640 1 -- 172.21.15.1:0/3770403609 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb770085660 con 0x7fb7800c5ca0 2024-05-09T05:09:43.268 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":30,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:09:43.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb76c076280 msgr2=0x7fb76c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:43.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb76c076280 0x7fb76c078790 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fb7801f0690 tx=0x7fb774007ae0 comp rx=0 tx=0).stop 2024-05-09T05:09:43.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 msgr2=0x7fb7801547a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:43.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7801547a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb770031000 tx=0x7fb7700046c0 comp rx=0 tx=0).stop 2024-05-09T05:09:43.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 shutdown_connections 2024-05-09T05:09:43.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb76c076280 0x7fb76c078790 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:43.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb780156e90 0x7fb78014b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:43.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb780156a80 0x7fb780155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:43.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 --2- 172.21.15.1:0/3770403609 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7800c5ca0 0x7fb7801547a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:43.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 >> 172.21.15.1:0/3770403609 conn(0x7fb7800745a0 msgr2=0x7fb7800c9610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:43.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 shutdown_connections 2024-05-09T05:09:43.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:43.270+0000 7fb785fa8640 1 -- 172.21.15.1:0/3770403609 wait complete. 2024-05-09T05:09:46.283 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:09:46.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.408+0000 7fb116c05640 1 Processor -- start 2024-05-09T05:09:46.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.408+0000 7fb116c05640 1 -- start start 2024-05-09T05:09:46.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.409+0000 7fb116c05640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.409+0000 7fb116c05640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.409+0000 7fb116c05640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1101594f0 con 0x7fb1100c9810 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.409+0000 7fb116c05640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb11015a380 con 0x7fb1100c87d0 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.409+0000 7fb116c05640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb110159770 con 0x7fb1101538b0 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.409+0000 7fb1151a2640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fb1100c9810 legacy=0x7fb11015cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb1149a1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb10ffff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb1149a1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33368/0 (socket says 172.21.15.1:33368) 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 learned_addr learned my addr 172.21.15.1:0/3295388987 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:46.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:46.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:09:46.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:46.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:46.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:46.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.410+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:46.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.610+0000 7fb1151a2640 1 -- 172.21.15.1:0/3295388987 >> v1:172.21.15.1:6789/0 conn(0x7fb1100c9810 legacy=0x7fb11015cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:09:46.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:46.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:46.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:09:46.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:46.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:46.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:09:46.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:46.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:46.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:09:46.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:46.611+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:09:47.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.011+0000 7fb1151a2640 1 -- 172.21.15.1:0/3295388987 >> v1:172.21.15.1:6789/0 conn(0x7fb1100c9810 legacy=0x7fb11015cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:09:47.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:47.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.012+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:09:47.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.812+0000 7fb1151a2640 1 -- 172.21.15.1:0/3295388987 >> v1:172.21.15.1:6789/0 conn(0x7fb1100c9810 legacy=0x7fb11015cc90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:09:47.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:47.813+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:09:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10e7fc640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 msgr2=0x7fb1100c92d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10e7fc640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1101538b0 0x7fb1100c92d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10e7fc640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 msgr2=0x7fb1100c5ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10e7fc640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.196:3300/0 conn(0x7fb1100c87d0 0x7fb1100c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:49.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10e7fc640 1 -- 172.21.15.1:0/3295388987 >> v1:172.21.15.1:6789/0 conn(0x7fb1100c9810 legacy=0x7fb11015cc90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:49.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10e7fc640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 0x7fb0f4009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:49.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10e7fc640 1 -- 172.21.15.1:0/3295388987 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb110159a50 con 0x7fb0f4003680 2024-05-09T05:09:49.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10e7fc640 1 -- 172.21.15.1:0/3295388987 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb11015a380 con 0x7fb0f4007380 2024-05-09T05:09:49.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 msgr2=0x7fb0f4009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:49.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.409+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 0x7fb0f4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.610+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 msgr2=0x7fb0f4009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:49.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:49.610+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 0x7fb0f4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:09:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.011+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 msgr2=0x7fb0f4009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:50.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.011+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 0x7fb0f4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:09:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.680+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3144285995 0 0) 0x7fb110159a50 con 0x7fb0f4003680 2024-05-09T05:09:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.680+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb100003520 con 0x7fb0f4003680 2024-05-09T05:09:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1794964722 0 0) 0x7fb100003520 con 0x7fb0f4003680 2024-05-09T05:09:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 msgr2=0x7fb0f4009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 --2- 172.21.15.1:0/3295388987 >> v2:172.21.15.1:3300/0 conn(0x7fb0f4007380 0x7fb0f4009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f400afb0 con 0x7fb0f4003680 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fb1040107a0 con 0x7fb0f4003680 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 >> v1:172.21.15.196:6789/0 conn(0x7fb0f4003680 legacy=0x7fb0f4005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb1151a2640 1 -- 172.21.15.1:0/3295388987 reap_dead start 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1000039b0 0x7fb100003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100004800 0x7fb100004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000099b0 0x7fb10000bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb11015a380 con 0x7fb1000039b0 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb10000c3c0 con 0x7fb100004800 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb10000c5c0 con 0x7fb1000099b0 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb1149a1640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1000039b0 msgr2=0x7fb100003e00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb1149a1640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1000039b0 0x7fb100003e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100004800 0x7fb100004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.685+0000 7fb1151a2640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000099b0 0x7fb10000bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:50.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.686+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000099b0 msgr2=0x7fb10000bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.686+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000099b0 0x7fb10000bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.686+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1000039b0 msgr2=0x7fb100003e00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.686+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1000039b0 0x7fb100003e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.686+0000 7fb10ffff640 1 -- 172.21.15.1:0/3295388987 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb100003520 con 0x7fb100004800 2024-05-09T05:09:50.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.687+0000 7fb10ffff640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100004800 0x7fb100004c10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb1100cc770 tx=0x7fb100005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=133b129d499a4f68 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.690+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc031070 con 0x7fb100004800 2024-05-09T05:09:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.690+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb0fc02b800 con 0x7fb100004800 2024-05-09T05:09:50.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.690+0000 7fb10f7fe640 1 -- 172.21.15.1:0/3295388987 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0fc003610 con 0x7fb100004800 2024-05-09T05:09:50.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 -- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100004800 msgr2=0x7fb100004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100004800 0x7fb100004c10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb1100cc770 tx=0x7fb100005830 comp rx=0 tx=0).stop 2024-05-09T05:09:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 -- 172.21.15.1:0/3295388987 shutdown_connections 2024-05-09T05:09:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb100004800 0x7fb100004c10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000099b0 0x7fb10000bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 --2- 172.21.15.1:0/3295388987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1000039b0 0x7fb100003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 -- 172.21.15.1:0/3295388987 >> 172.21.15.1:0/3295388987 conn(0x7fb1100745a0 msgr2=0x7fb110071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 -- 172.21.15.1:0/3295388987 shutdown_connections 2024-05-09T05:09:50.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.691+0000 7fb116c05640 1 -- 172.21.15.1:0/3295388987 wait complete. 2024-05-09T05:09:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.693+0000 7fb116c05640 1 Processor -- start 2024-05-09T05:09:50.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.693+0000 7fb116c05640 1 -- start start 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb116c05640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000039b0 0x7fb11015a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb116c05640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb100004800 0x7fb1101e38f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb116c05640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1000099b0 0x7fb11006d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb116c05640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1100c70c0 con 0x7fb100004800 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb116c05640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb11015a010 con 0x7fb1000099b0 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb116c05640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb1100c7240 con 0x7fb1000039b0 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb10ffff640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb100004800 msgr2=0x7fb1101e38f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb10ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb100004800 0x7fb1101e38f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb1151a2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1000099b0 0x7fb11006d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb1149a1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000039b0 0x7fb11015a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb1151a2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1000099b0 0x7fb11006d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:36930/0 (socket says 172.21.15.1:36930) 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.695+0000 7fb1151a2640 1 -- 172.21.15.1:0/2556394622 learned_addr learned my addr 172.21.15.1:0/2556394622 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:50.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb1151a2640 1 -- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000039b0 msgr2=0x7fb11015a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb1151a2640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000039b0 0x7fb11015a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb1151a2640 1 -- 172.21.15.1:0/2556394622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb100004800 msgr2=0x7fb1101e38f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb1151a2640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb100004800 0x7fb1101e38f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb1151a2640 1 -- 172.21.15.1:0/2556394622 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1101589c0 con 0x7fb1000099b0 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb1149a1640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000039b0 0x7fb11015a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb1151a2640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1000099b0 0x7fb11006d920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb108007fd0 tx=0x7fb10800fb80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb10d7fa640 1 -- 172.21.15.1:0/2556394622 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb10800dae0 con 0x7fb1000099b0 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb10d7fa640 1 -- 172.21.15.1:0/2556394622 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb108008040 con 0x7fb1000099b0 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb11015c330 con 0x7fb1000099b0 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb10d7fa640 1 -- 172.21.15.1:0/2556394622 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1080103f0 con 0x7fb1000099b0 2024-05-09T05:09:50.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.696+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb11015ad60 con 0x7fb1000099b0 2024-05-09T05:09:50.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.697+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb110159250 con 0x7fb1000099b0 2024-05-09T05:09:50.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.698+0000 7fb10d7fa640 1 -- 172.21.15.1:0/2556394622 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb108010590 con 0x7fb1000099b0 2024-05-09T05:09:50.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.699+0000 7fb10d7fa640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb0f4076b10 0x7fb0f4079020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:50.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.699+0000 7fb10d7fa640 1 -- 172.21.15.1:0/2556394622 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb1080979e0 con 0x7fb1000099b0 2024-05-09T05:09:50.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.699+0000 7fb1149a1640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb0f4076b10 0x7fb0f4079020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:50.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.700+0000 7fb1149a1640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb0f4076b10 0x7fb0f4079020 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb104001eb0 tx=0x7fb104013040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:50.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.704+0000 7fb10d7fa640 1 -- 172.21.15.1:0/2556394622 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb108060be0 con 0x7fb1000099b0 2024-05-09T05:09:50.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.977+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb110152870 con 0x7fb1000099b0 2024-05-09T05:09:50.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.978+0000 7fb10d7fa640 1 -- 172.21.15.1:0/2556394622 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fb110152870 con 0x7fb1000099b0 2024-05-09T05:09:50.978 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:09:50.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb0f4076b10 msgr2=0x7fb0f4079020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb0f4076b10 0x7fb0f4079020 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fb104001eb0 tx=0x7fb104013040 comp rx=0 tx=0).stop 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1000099b0 msgr2=0x7fb11006d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1000099b0 0x7fb11006d920 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fb108007fd0 tx=0x7fb10800fb80 comp rx=0 tx=0).stop 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 shutdown_connections 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb0f4076b10 0x7fb0f4079020 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1000099b0 0x7fb11006d920 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb100004800 0x7fb1101e38f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 --2- 172.21.15.1:0/2556394622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1000039b0 0x7fb11015a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 >> 172.21.15.1:0/2556394622 conn(0x7fb1100745a0 msgr2=0x7fb110075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 shutdown_connections 2024-05-09T05:09:50.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:50.980+0000 7fb116c05640 1 -- 172.21.15.1:0/2556394622 wait complete. 2024-05-09T05:09:50.991 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:09:50.991 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:09:50.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:09:51.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.106+0000 7fc34e23b640 1 Processor -- start 2024-05-09T05:09:51.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.106+0000 7fc34e23b640 1 -- start start 2024-05-09T05:09:51.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc34e23b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348072400 0x7fc348154530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc34e23b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc348071310 0x7fc348164ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc34e23b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3480cbf90 con 0x7fc348070f00 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc34e23b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc3480ca3c0 con 0x7fc348071310 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc34e23b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc34815a780 con 0x7fc348072400 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc346ffd640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fc348070f00 legacy=0x7fc3480cae10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc347fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc348071310 0x7fc348164ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc3477fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348072400 0x7fc348154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc347fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc348071310 0x7fc348164ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:36944/0 (socket says 172.21.15.1:36944) 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc3477fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348072400 0x7fc348154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:36728/0 (socket says 172.21.15.1:36728) 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.107+0000 7fc347fff640 1 -- 172.21.15.1:0/278716985 learned_addr learned my addr 172.21.15.1:0/278716985 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.108+0000 7fc3477fe640 1 -- 172.21.15.1:0/278716985 >> v2:172.21.15.196:3300/0 conn(0x7fc348071310 msgr2=0x7fc348164ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.108+0000 7fc3477fe640 1 --2- 172.21.15.1:0/278716985 >> v2:172.21.15.196:3300/0 conn(0x7fc348071310 0x7fc348164ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.108+0000 7fc3477fe640 1 -- 172.21.15.1:0/278716985 >> v1:172.21.15.1:6789/0 conn(0x7fc348070f00 legacy=0x7fc3480cae10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.108+0000 7fc3477fe640 1 -- 172.21.15.1:0/278716985 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3480ca610 con 0x7fc348072400 2024-05-09T05:09:51.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.108+0000 7fc347fff640 1 --2- 172.21.15.1:0/278716985 >> v2:172.21.15.196:3300/0 conn(0x7fc348071310 0x7fc348164ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:51.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.109+0000 7fc3477fe640 1 --2- 172.21.15.1:0/278716985 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348072400 0x7fc348154530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3380097a0 tx=0x7fc33802a0e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1f813de33e07b03b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.109+0000 7fc3467fc640 1 -- 172.21.15.1:0/278716985 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338037070 con 0x7fc348072400 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.109+0000 7fc3467fc640 1 -- 172.21.15.1:0/278716985 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc33802ae30 con 0x7fc348072400 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.109+0000 7fc3467fc640 1 -- 172.21.15.1:0/278716985 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338032640 con 0x7fc348072400 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.109+0000 7fc34e23b640 1 -- 172.21.15.1:0/278716985 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348072400 msgr2=0x7fc348154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.109+0000 7fc34e23b640 1 --2- 172.21.15.1:0/278716985 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348072400 0x7fc348154530 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc3380097a0 tx=0x7fc33802a0e0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.110+0000 7fc34e23b640 1 -- 172.21.15.1:0/278716985 shutdown_connections 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.110+0000 7fc34e23b640 1 --2- 172.21.15.1:0/278716985 >> v2:172.21.15.196:3300/0 conn(0x7fc348071310 0x7fc348164ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.110+0000 7fc34e23b640 1 --2- 172.21.15.1:0/278716985 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348072400 0x7fc348154530 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.110+0000 7fc34e23b640 1 -- 172.21.15.1:0/278716985 >> 172.21.15.1:0/278716985 conn(0x7fc3480d0470 msgr2=0x7fc3480cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.110+0000 7fc34e23b640 1 -- 172.21.15.1:0/278716985 shutdown_connections 2024-05-09T05:09:51.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.110+0000 7fc34e23b640 1 -- 172.21.15.1:0/278716985 wait complete. 2024-05-09T05:09:51.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.111+0000 7fc34e23b640 1 Processor -- start 2024-05-09T05:09:51.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.112+0000 7fc34e23b640 1 -- start start 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc34e23b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc348070f00 0x7fc34814f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc34e23b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348071310 0x7fc34806e2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc34e23b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc348072400 0x7fc3480b40d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc34e23b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3480b2630 con 0x7fc348072400 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc34e23b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc348071720 con 0x7fc348070f00 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc34e23b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc3480b27b0 con 0x7fc348071310 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc347fff640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc348072400 msgr2=0x7fc3480b40d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc347fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc348072400 0x7fc3480b40d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc3477fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc348070f00 0x7fc34814f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc346ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348071310 0x7fc34806e2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc346ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348071310 0x7fc34806e2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:36738/0 (socket says 172.21.15.1:36738) 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc3477fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc348070f00 0x7fc34814f1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:36958/0 (socket says 172.21.15.1:36958) 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.113+0000 7fc346ffd640 1 -- 172.21.15.1:0/921880489 learned_addr learned my addr 172.21.15.1:0/921880489 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc346ffd640 1 -- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc348070f00 msgr2=0x7fc34814f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc346ffd640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc348070f00 0x7fc34814f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc346ffd640 1 -- 172.21.15.1:0/921880489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc348072400 msgr2=0x7fc3480b40d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc346ffd640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc348072400 0x7fc3480b40d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc346ffd640 1 -- 172.21.15.1:0/921880489 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc34806e820 con 0x7fc348071310 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc3477fe640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc348070f00 0x7fc34814f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc346ffd640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348071310 0x7fc34806e2e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc33c00ba10 tx=0x7fc33c00bed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c00cbd0 con 0x7fc348071310 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc33c010070 con 0x7fc348071310 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3480c69b0 con 0x7fc348071310 2024-05-09T05:09:51.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc33c015470 con 0x7fc348071310 2024-05-09T05:09:51.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.114+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc348152c40 con 0x7fc348071310 2024-05-09T05:09:51.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.115+0000 7fc34e23b640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc34814f970 0x7fc34806d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.115+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc34806ecf0 con 0x7fc34814f970 2024-05-09T05:09:51.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.115+0000 7fc3477fe640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc34814f970 0x7fc34806d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.115+0000 7fc3477fe640 1 -- 172.21.15.1:0/921880489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc348159d70 con 0x7fc34814f970 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.116+0000 7fc3477fe640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc34814f970 0x7fc34806d800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc3480696b0 tx=0x7fc3380046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.116+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338037040 con 0x7fc34814f970 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.116+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc33c015610 con 0x7fc348071310 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc32ffff640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3340761d0 0x7fc3340786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc33c097800 con 0x7fc348071310 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc347fff640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3340761d0 0x7fc3340786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc338032e50 con 0x7fc34814f970 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc34814f970 msgr2=0x7fc34806d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc32ffff640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc34814f970 0x7fc34806d800 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fc3480696b0 tx=0x7fc3380046c0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc34814f970 msgr2=0x7fc34806d800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc32ffff640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc34814f970 0x7fc34806d800 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.117+0000 7fc347fff640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3340761d0 0x7fc3340786e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc3480c56d0 tx=0x7fc340009240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.123+0000 7fc34e23b640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3481e6c30 0x7fc3481e7040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.123+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc348050870 con 0x7fc3481e6c30 2024-05-09T05:09:51.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.124+0000 7fc3477fe640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3481e6c30 0x7fc3481e7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.124+0000 7fc3477fe640 1 -- 172.21.15.1:0/921880489 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc3480cbcd0 con 0x7fc3481e6c30 2024-05-09T05:09:51.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.124+0000 7fc3477fe640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3481e6c30 0x7fc3481e7040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc33802a120 tx=0x7fc338008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.124+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc338003de0 con 0x7fc3481e6c30 2024-05-09T05:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7fc338003de0 con 0x7fc3481e6c30 2024-05-09T05:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3481e6c30 msgr2=0x7fc3481e7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc32ffff640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3481e6c30 0x7fc3481e7040 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc33802a120 tx=0x7fc338008030 comp rx=0 tx=0).stop 2024-05-09T05:09:51.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc32ffff640 1 -- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3481e6c30 msgr2=0x7fc3481e7040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc32ffff640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3481e6c30 0x7fc3481e7040 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 32, 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 34697, 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:09:51.126 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.127 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:09:51.128 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:09:51.129 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.130 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.135 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:09:51.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:09:51.137 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:09:51.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3340761d0 msgr2=0x7fc3340786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc34e23b640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3340761d0 0x7fc3340786e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc3480c56d0 tx=0x7fc340009240 comp rx=0 tx=0).stop 2024-05-09T05:09:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348071310 msgr2=0x7fc34806e2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc34e23b640 1 --2- 172.21.15.1:0/921880489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc348071310 0x7fc34806e2e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fc33c00ba10 tx=0x7fc33c00bed0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc347fff640 1 -- 172.21.15.1:0/921880489 reap_dead start 2024-05-09T05:09:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc347fff640 1 -- 172.21.15.1:0/921880489 reap_dead start 2024-05-09T05:09:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 shutdown_connections 2024-05-09T05:09:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 >> 172.21.15.1:0/921880489 conn(0x7fc3480d0470 msgr2=0x7fc3480cda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 shutdown_connections 2024-05-09T05:09:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.125+0000 7fc34e23b640 1 -- 172.21.15.1:0/921880489 wait complete. 2024-05-09T05:09:51.139 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:09:51.140 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:09:51.140 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:09:51.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.254+0000 7f5f08ac8640 1 Processor -- start 2024-05-09T05:09:51.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.254+0000 7f5f08ac8640 1 -- start start 2024-05-09T05:09:51.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.255+0000 7f5f08ac8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f04156c20 0x7f5f040cea20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f08ac8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f04160160 con 0x7f5f040cdaf0 2024-05-09T05:09:51.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f08ac8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f04160440 con 0x7f5f040cd610 2024-05-09T05:09:51.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f08ac8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5f04160ce0 con 0x7f5f04156c20 2024-05-09T05:09:51.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f02d9d640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f5f040cdaf0 legacy=0x7f5f041670d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:09:51.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f01d9b640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f5f040cd610 0x7f5f0415ce10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:55320/0 (socket says 172.21.15.1:55320) 2024-05-09T05:09:51.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f0259c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f04156c20 0x7f5f040cea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f01d9b640 1 -- 172.21.15.1:0/3994608489 learned_addr learned my addr 172.21.15.1:0/3994608489 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:09:51.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f0159a640 1 -- 172.21.15.1:0/3994608489 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2326186178 0 0) 0x7f5f04160440 con 0x7f5f040cd610 2024-05-09T05:09:51.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f0159a640 1 -- 172.21.15.1:0/3994608489 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5ef0003520 con 0x7f5f040cd610 2024-05-09T05:09:51.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f0259c640 1 -- 172.21.15.1:0/3994608489 >> v1:172.21.15.196:6789/0 conn(0x7f5f040cd610 legacy=0x7f5f0415ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f0259c640 1 -- 172.21.15.1:0/3994608489 >> v1:172.21.15.1:6789/0 conn(0x7f5f040cdaf0 legacy=0x7f5f041670d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:09:51.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.256+0000 7f5f0259c640 1 -- 172.21.15.1:0/3994608489 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f04160720 con 0x7f5f04156c20 2024-05-09T05:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.257+0000 7f5f0259c640 1 --2- 172.21.15.1:0/3994608489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f04156c20 0x7f5f040cea20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009dc0 tx=0x7f5efc02a6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=500199cff22d3eca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.257+0000 7f5f0159a640 1 -- 172.21.15.1:0/3994608489 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc007740 con 0x7f5f04156c20 2024-05-09T05:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f0159a640 1 -- 172.21.15.1:0/3994608489 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5efc0078e0 con 0x7f5f04156c20 2024-05-09T05:09:51.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f0159a640 1 -- 172.21.15.1:0/3994608489 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc007c90 con 0x7f5f04156c20 2024-05-09T05:09:51.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3994608489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f04156c20 msgr2=0x7f5f040cea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f08ac8640 1 --2- 172.21.15.1:0/3994608489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f04156c20 0x7f5f040cea20 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009dc0 tx=0x7f5efc02a6c0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3994608489 shutdown_connections 2024-05-09T05:09:51.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f08ac8640 1 --2- 172.21.15.1:0/3994608489 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f04156c20 0x7f5f040cea20 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3994608489 >> 172.21.15.1:0/3994608489 conn(0x7f5f04068830 msgr2=0x7f5f04068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:51.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3994608489 shutdown_connections 2024-05-09T05:09:51.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.258+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3994608489 wait complete. 2024-05-09T05:09:51.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.260+0000 7f5f08ac8640 1 Processor -- start 2024-05-09T05:09:51.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.260+0000 7f5f08ac8640 1 -- start start 2024-05-09T05:09:51.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.261+0000 7f5f08ac8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f040cd610 0x7f5f041f60d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.261+0000 7f5f08ac8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040cdaf0 0x7f5f04150bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.261+0000 7f5f08ac8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d1f60 0x7f5f0414c550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.261+0000 7f5f08ac8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5f040ce2a0 con 0x7f5f040cdaf0 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.261+0000 7f5f08ac8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5f0415c8f0 con 0x7f5f040cd610 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.261+0000 7f5f08ac8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5f0415d520 con 0x7f5f040d1f60 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.261+0000 7f5f01d9b640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040cdaf0 msgr2=0x7f5f04150bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.261+0000 7f5f01d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040cdaf0 0x7f5f04150bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f0259c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f040cd610 0x7f5f041f60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f02d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d1f60 0x7f5f0414c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f0259c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f040cd610 0x7f5f041f60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:36982/0 (socket says 172.21.15.1:36982) 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f02d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d1f60 0x7f5f0414c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:36758/0 (socket says 172.21.15.1:36758) 2024-05-09T05:09:51.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f0259c640 1 -- 172.21.15.1:0/3903709999 learned_addr learned my addr 172.21.15.1:0/3903709999 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f02d9d640 1 -- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f040cd610 msgr2=0x7f5f041f60d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f040cd610 0x7f5f041f60d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f02d9d640 1 -- 172.21.15.1:0/3903709999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040cdaf0 msgr2=0x7f5f04150bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f040cdaf0 0x7f5f04150bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f02d9d640 1 -- 172.21.15.1:0/3903709999 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f041f6610 con 0x7f5f040d1f60 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.262+0000 7f5f02d9d640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d1f60 0x7f5f0414c550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400b360 tx=0x7f5ef400b820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.263+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4017070 con 0x7f5f040d1f60 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.263+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5ef4003db0 con 0x7f5f040d1f60 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.263+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f040cfb90 con 0x7f5f040d1f60 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.263+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ef4006310 con 0x7f5f040d1f60 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.263+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f040d0110 con 0x7f5f040d1f60 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.263+0000 7f5f08ac8640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d2370 0x7f5f0414ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.263+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5f041f6ae0 con 0x7f5f040d2370 2024-05-09T05:09:51.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.263+0000 7f5f0259c640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d2370 0x7f5f0414ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.264+0000 7f5f0259c640 1 -- 172.21.15.1:0/3903709999 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f040d2960 con 0x7f5f040d2370 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.264+0000 7f5f0259c640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d2370 0x7f5f0414ca90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5efc02a700 tx=0x7f5efc004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.264+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc007540 con 0x7f5f040d2370 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5ef40064b0 con 0x7f5f040d1f60 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5efaffd640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ee4076200 0x7f5ee4078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5ef4096640 con 0x7f5f040d1f60 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5f01d9b640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ee4076200 0x7f5ee4078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5efc0076e0 con 0x7f5f040d2370 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d2370 msgr2=0x7f5f0414ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5efaffd640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d2370 0x7f5f0414ca90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5efc02a700 tx=0x7f5efc004a40 comp rx=0 tx=0).stop 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d2370 msgr2=0x7f5f0414ca90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:51.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.265+0000 7f5efaffd640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d2370 0x7f5f0414ca90 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5efc02a700 tx=0x7f5efc004a40 comp rx=0 tx=0).stop 2024-05-09T05:09:51.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.266+0000 7f5f01d9b640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ee4076200 0x7f5ee4078710 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5f040b5ef0 tx=0x7f5eec0079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.272+0000 7f5f08ac8640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f0415a5c0 0x7f5f0415a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:09:51.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.272+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5f040c8870 con 0x7f5f0415a5c0 2024-05-09T05:09:51.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.272+0000 7f5f0259c640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f0415a5c0 0x7f5f0415a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:09:51.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.273+0000 7f5f0259c640 1 -- 172.21.15.1:0/3903709999 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f040ce420 con 0x7f5f0415a5c0 2024-05-09T05:09:51.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.273+0000 7f5f0259c640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f0415a5c0 0x7f5f0415a9d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5efc000f80 tx=0x7f5efc030000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:09:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.273+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5efc046c20 con 0x7f5f0415a5c0 2024-05-09T05:09:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5050 (secure 0 0 0) 0x7f5efc03c420 con 0x7f5f0415a5c0 2024-05-09T05:09:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f0415a5c0 msgr2=0x7f5f0415a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5efaffd640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f0415a5c0 0x7f5f0415a9d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5efc000f80 tx=0x7f5efc030000 comp rx=0 tx=0).stop 2024-05-09T05:09:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5efaffd640 1 -- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f0415a5c0 msgr2=0x7f5f0415a9d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:09:51.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5efaffd640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f0415a5c0 0x7f5f0415a9d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 32, 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 149270, 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:51.275 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:09:51.276 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:09:51.277 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:09:51.278 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:09:51.279 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.280 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.281 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:09:51.282 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:09:51.283 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.284 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:09:51.285 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 7 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:09:51.286 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:09:51.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ee4076200 msgr2=0x7f5ee4078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ee4076200 0x7f5ee4078710 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f5f040b5ef0 tx=0x7f5eec0079b0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d1f60 msgr2=0x7f5f0414c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f02d9d640 1 -- 172.21.15.1:0/3903709999 reap_dead start 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d1f60 0x7f5f0414c550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5ef400b360 tx=0x7f5ef400b820 comp rx=0 tx=0).stop 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 shutdown_connections 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 --2- 172.21.15.1:0/3903709999 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f040d1f60 0x7f5f0414c550 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 >> 172.21.15.1:0/3903709999 conn(0x7f5f04068830 msgr2=0x7f5f040c7420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 shutdown_connections 2024-05-09T05:09:51.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:09:51.274+0000 7f5f08ac8640 1 -- 172.21.15.1:0/3903709999 wait complete. 2024-05-09T05:09:51.288 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:09:51.288 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:10:02.518 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:10:02.520 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:10:11.289 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-05-09T05:10:11.289 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-05-09T05:10:11.289 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T05:10:11.289 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T05:10:11.292 INFO:tasks.ceph.mon.a:Started 2024-05-09T05:10:11.293 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:10:11.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:10:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.418+0000 7fa88c670640 1 Processor -- start 2024-05-09T05:10:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.419+0000 7fa88c670640 1 -- start start 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88c670640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa8840c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88c670640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 0x7fa8841547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88c670640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa884154cf0 con 0x7fa8840c62e0 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88c670640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa8840c53f0 con 0x7fa884072120 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88c670640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa8840c55b0 con 0x7fa8840ca340 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa889c0b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 0x7fa8841547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa889c0b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 0x7fa8841547b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54586/0 (socket says 172.21.15.1:54586) 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa889c0b640 1 -- 172.21.15.1:0/1431561423 learned_addr learned my addr 172.21.15.1:0/1431561423 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88a40c640 1 --2- 172.21.15.1:0/1431561423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa8840c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:11.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa889c0b640 1 -- 172.21.15.1:0/1431561423 >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 msgr2=0x7fa8841547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa889c0b640 1 -- 172.21.15.1:0/1431561423 >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 msgr2=0x7fa8841547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa889c0b640 1 --2- 172.21.15.1:0/1431561423 >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 0x7fa8841547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa889c0b640 1 --2- 172.21.15.1:0/1431561423 >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 0x7fa8841547b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88940a640 1 -- 172.21.15.1:0/1431561423 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3026475915 0 0) 0x7fa87c001490 con 0x7fa8840c62e0 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88a40c640 1 -- 172.21.15.1:0/1431561423 >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 msgr2=0x7fa8841547b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88a40c640 1 --2- 172.21.15.1:0/1431561423 >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 0x7fa8841547b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88a40c640 1 -- 172.21.15.1:0/1431561423 >> v1:172.21.15.196:6789/0 conn(0x7fa8840c62e0 legacy=0x7fa884070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.421+0000 7fa88a40c640 1 -- 172.21.15.1:0/1431561423 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa88415b110 con 0x7fa8840ca340 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.422+0000 7fa88a40c640 1 --2- 172.21.15.1:0/1431561423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa8840c4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa8800097a0 tx=0x7fa88002a100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=919e7018ab2b517e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.422+0000 7fa88940a640 1 -- 172.21.15.1:0/1431561423 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880037070 con 0x7fa8840ca340 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.422+0000 7fa88940a640 1 -- 172.21.15.1:0/1431561423 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa88002ae50 con 0x7fa8840ca340 2024-05-09T05:10:11.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.422+0000 7fa88940a640 1 -- 172.21.15.1:0/1431561423 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa880032660 con 0x7fa8840ca340 2024-05-09T05:10:11.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.423+0000 7fa88c670640 1 -- 172.21.15.1:0/1431561423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 msgr2=0x7fa8840c4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:11.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.423+0000 7fa88c670640 1 --2- 172.21.15.1:0/1431561423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa8840c4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fa8800097a0 tx=0x7fa88002a100 comp rx=0 tx=0).stop 2024-05-09T05:10:11.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.423+0000 7fa88c670640 1 -- 172.21.15.1:0/1431561423 shutdown_connections 2024-05-09T05:10:11.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.423+0000 7fa88c670640 1 --2- 172.21.15.1:0/1431561423 >> v2:172.21.15.1:3300/0 conn(0x7fa884072120 0x7fa8841547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.423+0000 7fa88c670640 1 --2- 172.21.15.1:0/1431561423 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa8840c4eb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.423+0000 7fa88c670640 1 -- 172.21.15.1:0/1431561423 >> 172.21.15.1:0/1431561423 conn(0x7fa8840731d0 msgr2=0x7fa8840cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:11.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.423+0000 7fa88c670640 1 -- 172.21.15.1:0/1431561423 shutdown_connections 2024-05-09T05:10:11.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.423+0000 7fa88c670640 1 -- 172.21.15.1:0/1431561423 wait complete. 2024-05-09T05:10:11.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.425+0000 7fa88c670640 1 Processor -- start 2024-05-09T05:10:11.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.425+0000 7fa88c670640 1 -- start start 2024-05-09T05:10:11.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.426+0000 7fa88c670640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 0x7fa8841f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:11.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.426+0000 7fa88c670640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8840c62e0 0x7fa8841523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.426+0000 7fa88c670640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa884151170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.426+0000 7fa88c670640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa8840c5ae0 con 0x7fa884072120 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.426+0000 7fa88c670640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa8840c5800 con 0x7fa8840c62e0 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.426+0000 7fa88c670640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa8840c5dc0 con 0x7fa8840ca340 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.426+0000 7fa88a40c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 0x7fa8841f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa88a40c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 0x7fa8841f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:54592/0 (socket says 172.21.15.1:54592) 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa88a40c640 1 -- 172.21.15.1:0/1480519769 learned_addr learned my addr 172.21.15.1:0/1480519769 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa889c0b640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8840c62e0 0x7fa8841523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa88ac0d640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa884151170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa88a40c640 1 -- 172.21.15.1:0/1480519769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 msgr2=0x7fa8841f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa88a40c640 1 -- 172.21.15.1:0/1480519769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 msgr2=0x7fa8841f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa88a40c640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 0x7fa8841f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa88a40c640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 0x7fa8841f31a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:11.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa889c0b640 1 -- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 msgr2=0x7fa884151170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa889c0b640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa884151170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa889c0b640 1 -- 172.21.15.1:0/1480519769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 msgr2=0x7fa8841f31a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa889c0b640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 0x7fa8841f31a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa889c0b640 1 -- 172.21.15.1:0/1480519769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8841f36e0 con 0x7fa8840c62e0 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa889c0b640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8840c62e0 0x7fa8841523a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa87400e7e0 tx=0x7fa87400eca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa87affd640 1 -- 172.21.15.1:0/1480519769 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa874017070 con 0x7fa8840c62e0 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa87affd640 1 -- 172.21.15.1:0/1480519769 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa874004020 con 0x7fa8840c62e0 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.427+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa88406e080 con 0x7fa8840c62e0 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.428+0000 7fa87affd640 1 -- 172.21.15.1:0/1480519769 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa87400f040 con 0x7fa8840c62e0 2024-05-09T05:10:11.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.428+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa88406dda0 con 0x7fa8840c62e0 2024-05-09T05:10:11.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.428+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa884050870 con 0x7fa8840c62e0 2024-05-09T05:10:11.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.429+0000 7fa87affd640 1 -- 172.21.15.1:0/1480519769 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa87400c6d0 con 0x7fa8840c62e0 2024-05-09T05:10:11.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.430+0000 7fa87affd640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa8680762d0 0x7fa8680787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:11.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.430+0000 7fa87affd640 1 -- 172.21.15.1:0/1480519769 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa8740970d0 con 0x7fa8840c62e0 2024-05-09T05:10:11.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.430+0000 7fa88a40c640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa8680762d0 0x7fa8680787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:11.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.431+0000 7fa88a40c640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa8680762d0 0x7fa8680787e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa880009560 tx=0x7fa88002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:11.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.435+0000 7fa87affd640 1 -- 172.21.15.1:0/1480519769 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa8740602d0 con 0x7fa8840c62e0 2024-05-09T05:10:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.757+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa8840cba10 con 0x7fa8840c62e0 2024-05-09T05:10:11.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.757+0000 7fa87affd640 1 -- 172.21.15.1:0/1480519769 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fa8740645d0 con 0x7fa8840c62e0 2024-05-09T05:10:11.758 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":32,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:10:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa8680762d0 msgr2=0x7fa8680787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa8680762d0 0x7fa8680787e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa880009560 tx=0x7fa88002e000 comp rx=0 tx=0).stop 2024-05-09T05:10:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8840c62e0 msgr2=0x7fa8841523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8840c62e0 0x7fa8841523a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa87400e7e0 tx=0x7fa87400eca0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 shutdown_connections 2024-05-09T05:10:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa8680762d0 0x7fa8680787e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa8840c62e0 0x7fa8841523a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa8840ca340 0x7fa884151170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 --2- 172.21.15.1:0/1480519769 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa884072120 0x7fa8841f31a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 >> 172.21.15.1:0/1480519769 conn(0x7fa8840731d0 msgr2=0x7fa8840cf890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.759+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 shutdown_connections 2024-05-09T05:10:11.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:11.760+0000 7fa88c670640 1 -- 172.21.15.1:0/1480519769 wait complete. 2024-05-09T05:10:14.773 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:10:14.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.901+0000 7fe2aa99f640 1 Processor -- start 2024-05-09T05:10:14.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.901+0000 7fe2aa99f640 1 -- start start 2024-05-09T05:10:14.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.903+0000 7fe2aa99f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:14.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.903+0000 7fe2aa99f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2a4073f90 con 0x7fe2a40af530 2024-05-09T05:10:14.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.903+0000 7fe2aa99f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2a40ca280 con 0x7fe2a415c860 2024-05-09T05:10:14.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.903+0000 7fe2aa99f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe2a4156c70 con 0x7fe2a4156860 2024-05-09T05:10:14.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.903+0000 7fe2aa19e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe2a40af530 0x7fe2a40af940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46506/0 (socket says 172.21.15.1:46506) 2024-05-09T05:10:14.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.903+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 learned_addr learned my addr 172.21.15.1:0/1639105044 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:14.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.903+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:14.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.904+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:14.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.904+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:14.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.904+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:14.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:14.904+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.105+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.105+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:15.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.506+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.506+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:15.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:15.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:16.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:16.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:16.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:16.307+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:16.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:16.307+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:16.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:16.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:16.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:16.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 msgr2=0x7fe2a40c9d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe2a4156860 0x7fe2a40c9d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 >> v1:172.21.15.196:6789/0 conn(0x7fe2a415c860 legacy=0x7fe2a415cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 >> v1:172.21.15.1:6789/0 conn(0x7fe2a40af530 legacy=0x7fe2a40af940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe288008d90 con 0x7fe288006220 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe288008f40 con 0x7fe288003710 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:17.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:17.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.903+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:17.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.904+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:17.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.904+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:17.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.904+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:17.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.904+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:17.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:17.904+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:18.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.104+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:18.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.104+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:18.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.104+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:18.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.104+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:18.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.104+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:18.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:18.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.105+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:18.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.105+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:18.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:18.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.505+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.505+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.505+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.505+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:18.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.505+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.506+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.506+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:18.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:18.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.306+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.306+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.306+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.306+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:19.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.306+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.307+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.307+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:19.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:19.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 msgr2=0x7fe288005bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003710 0x7fe288005bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 msgr2=0x7fe288008710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:20.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288006220 0x7fe288008710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 reap_dead start 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe288008d90 con 0x7fe288007640 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe288008f40 con 0x7fe288006b70 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe288008320 con 0x7fe288003b80 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:20.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:20.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:20.904+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:21.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:21.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:21.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:21.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:21.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:21.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:21.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.105+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:21.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:10:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:21.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:21.506+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:22.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a999d640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:22.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:22.307+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:23.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe288006630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:23.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe288006630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:23.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:23.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 msgr2=0x7fe288006f80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:23.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.196:3300/0 conn(0x7fe288006b70 0x7fe288006f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:23.903+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 msgr2=0x7fe288007b40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:23.903+0000 7fe29b7fe640 1 --2- 172.21.15.1:0/1639105044 >> v2:172.21.15.1:3300/0 conn(0x7fe288007640 0x7fe288007b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:23.904+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe2a40733c0 con 0x7fe288010ad0 2024-05-09T05:10:23.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:23.904+0000 7fe29b7fe640 1 -- 172.21.15.1:0/1639105044 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe288015820 con 0x7fe288008740 2024-05-09T05:10:24.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.441+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2297717566 0 0) 0x7fe288015820 con 0x7fe288008740 2024-05-09T05:10:24.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.441+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe290003520 con 0x7fe288008740 2024-05-09T05:10:24.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2961972171 0 0) 0x7fe290003520 con 0x7fe288008740 2024-05-09T05:10:24.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 >> v1:172.21.15.1:6789/0 conn(0x7fe288010ad0 legacy=0x7fe288012fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:24.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe288016a20 con 0x7fe288008740 2024-05-09T05:10:24.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fe2a00176e0 con 0x7fe288008740 2024-05-09T05:10:24.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 >> v1:172.21.15.196:6789/0 conn(0x7fe288008740 legacy=0x7fe28800f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2aa19e640 1 -- 172.21.15.1:0/1639105044 reap_dead start 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe290003ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004200 0x7fe290004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004fc0 0x7fe29000b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe29000bf00 con 0x7fe290004200 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe29000c0b0 con 0x7fe290004fc0 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.445+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe29000c2b0 con 0x7fe288003b80 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.446+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004200 0x7fe290004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.446+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004fc0 0x7fe29000b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.446+0000 7fe2a999d640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe290003ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.446+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe290003ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.446+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe290003ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.446+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004fc0 msgr2=0x7fe29000b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:24.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.446+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004fc0 0x7fe29000b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.446+0000 7fe2a919c640 1 -- 172.21.15.1:0/1639105044 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe290003520 con 0x7fe290004200 2024-05-09T05:10:24.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.447+0000 7fe2a919c640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004200 0x7fe290004800 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe294005560 tx=0x7fe290005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d43f780e54cb89e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:24.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.447+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294039070 con 0x7fe290004200 2024-05-09T05:10:24.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.447+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe294034410 con 0x7fe290004200 2024-05-09T05:10:24.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.447+0000 7fe2a899b640 1 -- 172.21.15.1:0/1639105044 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe294005230 con 0x7fe290004200 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 -- 172.21.15.1:0/1639105044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004200 msgr2=0x7fe290004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004200 0x7fe290004800 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fe294005560 tx=0x7fe290005830 comp rx=0 tx=0).stop 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 -- 172.21.15.1:0/1639105044 shutdown_connections 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004fc0 0x7fe29000b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004200 0x7fe290004800 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/1639105044 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe290003ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 -- 172.21.15.1:0/1639105044 >> 172.21.15.1:0/1639105044 conn(0x7fe2a40ca860 msgr2=0x7fe2a40cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 -- 172.21.15.1:0/1639105044 shutdown_connections 2024-05-09T05:10:24.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.448+0000 7fe2aa99f640 1 -- 172.21.15.1:0/1639105044 wait complete. 2024-05-09T05:10:24.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.451+0000 7fe2aa99f640 1 Processor -- start 2024-05-09T05:10:24.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.451+0000 7fe2aa99f640 1 -- start start 2024-05-09T05:10:24.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.452+0000 7fe2aa99f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe2a41f33f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:24.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.452+0000 7fe2aa99f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004200 0x7fe2a40cd530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:24.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa99f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 0x7fe2a415a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa99f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a4073c20 con 0x7fe290004fc0 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa99f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe2a4073940 con 0x7fe290004200 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa99f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe2a40ca470 con 0x7fe288003b80 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe2a41f33f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa19e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 0x7fe2a415a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe2a41f33f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:52170/0 (socket says 172.21.15.1:52170) 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 -- 172.21.15.1:0/2705383723 learned_addr learned my addr 172.21.15.1:0/2705383723 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a919c640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004200 0x7fe2a40cd530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa19e640 1 -- 172.21.15.1:0/2705383723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 msgr2=0x7fe2a415a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa19e640 1 -- 172.21.15.1:0/2705383723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 msgr2=0x7fe2a415a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 0x7fe2a415a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2aa19e640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 0x7fe2a415a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 -- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004200 msgr2=0x7fe2a40cd530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:24.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004200 0x7fe2a40cd530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 -- 172.21.15.1:0/2705383723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 msgr2=0x7fe2a415a590 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 0x7fe2a415a590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 -- 172.21.15.1:0/2705383723 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe2a40cc800 con 0x7fe288003b80 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a919c640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004200 0x7fe2a40cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe2a999d640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe2a41f33f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0001eb0 tx=0x7fe2a00064d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.453+0000 7fe29a7fc640 1 -- 172.21.15.1:0/2705383723 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0004260 con 0x7fe288003b80 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.454+0000 7fe29a7fc640 1 -- 172.21.15.1:0/2705383723 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe2a0004420 con 0x7fe288003b80 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.454+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2a415bee0 con 0x7fe288003b80 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.454+0000 7fe29a7fc640 1 -- 172.21.15.1:0/2705383723 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2a0003850 con 0x7fe288003b80 2024-05-09T05:10:24.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.454+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2a41f2910 con 0x7fe288003b80 2024-05-09T05:10:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.455+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe2a40766d0 con 0x7fe288003b80 2024-05-09T05:10:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.455+0000 7fe29a7fc640 1 -- 172.21.15.1:0/2705383723 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe2a00039f0 con 0x7fe288003b80 2024-05-09T05:10:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.456+0000 7fe29a7fc640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe288077e70 0x7fe28807a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.456+0000 7fe29a7fc640 1 -- 172.21.15.1:0/2705383723 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe2a0099af0 con 0x7fe288003b80 2024-05-09T05:10:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.456+0000 7fe2a919c640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe288077e70 0x7fe28807a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:24.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.457+0000 7fe2a919c640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe288077e70 0x7fe28807a380 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe294036000 tx=0x7fe29402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:24.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.461+0000 7fe29a7fc640 1 -- 172.21.15.1:0/2705383723 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe2a40766d0 con 0x7fe288003b80 2024-05-09T05:10:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.769+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe2a4076f80 con 0x7fe288003b80 2024-05-09T05:10:24.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.770+0000 7fe29a7fc640 1 -- 172.21.15.1:0/2705383723 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fe2a001f070 con 0x7fe288003b80 2024-05-09T05:10:24.770 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:10:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe288077e70 msgr2=0x7fe28807a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe288077e70 0x7fe28807a380 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe294036000 tx=0x7fe29402f000 comp rx=0 tx=0).stop 2024-05-09T05:10:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 msgr2=0x7fe2a41f33f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe2a41f33f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe2a0001eb0 tx=0x7fe2a00064d0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 shutdown_connections 2024-05-09T05:10:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe288077e70 0x7fe28807a380 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe290004200 0x7fe2a40cd530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe290004fc0 0x7fe2a415a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 --2- 172.21.15.1:0/2705383723 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe288003b80 0x7fe2a41f33f0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 >> 172.21.15.1:0/2705383723 conn(0x7fe2a40ca860 msgr2=0x7fe2a4074160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 shutdown_connections 2024-05-09T05:10:24.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:24.772+0000 7fe2aa99f640 1 -- 172.21.15.1:0/2705383723 wait complete. 2024-05-09T05:10:27.785 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:10:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.910+0000 7fe001be3640 1 Processor -- start 2024-05-09T05:10:27.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.910+0000 7fe001be3640 1 -- start start 2024-05-09T05:10:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fe001be3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc0ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fe001be3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 0x7fdffc0c7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:27.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fe001be3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdffc0c86a0 con 0x7fdffc156a80 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fe001be3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdffc0cb910 con 0x7fdffc156e90 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fe001be3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc0c8980 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fdffbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 0x7fdffc0c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fdffbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 0x7fdffc0c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50838/0 (socket says 172.21.15.1:50838) 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fdffbfff640 1 -- 172.21.15.1:0/1540208601 learned_addr learned my addr 172.21.15.1:0/1540208601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.911+0000 7fdffb7fe640 1 --2- 172.21.15.1:0/1540208601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc0ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffbfff640 1 -- 172.21.15.1:0/1540208601 >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 msgr2=0x7fdffc0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffbfff640 1 -- 172.21.15.1:0/1540208601 >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 msgr2=0x7fdffc0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffbfff640 1 --2- 172.21.15.1:0/1540208601 >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 0x7fdffc0c7750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffbfff640 1 --2- 172.21.15.1:0/1540208601 >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 0x7fdffc0c7750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffb7fe640 1 -- 172.21.15.1:0/1540208601 >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 msgr2=0x7fdffc0c7750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffb7fe640 1 --2- 172.21.15.1:0/1540208601 >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 0x7fdffc0c7750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffa7fc640 1 -- 172.21.15.1:0/1540208601 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 457780891 0 0) 0x7fdffc0c86a0 con 0x7fdffc156a80 2024-05-09T05:10:27.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffb7fe640 1 -- 172.21.15.1:0/1540208601 >> v1:172.21.15.196:6789/0 conn(0x7fdffc156a80 legacy=0x7fdffc06df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:27.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.912+0000 7fdffb7fe640 1 -- 172.21.15.1:0/1540208601 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc0c8c60 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.913+0000 7fdffb7fe640 1 --2- 172.21.15.1:0/1540208601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc0ca4a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0097a0 tx=0x7fdfec02a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b3a63b6f7e7a0232 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.913+0000 7fdffa7fc640 1 -- 172.21.15.1:0/1540208601 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec037070 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.913+0000 7fdffa7fc640 1 -- 172.21.15.1:0/1540208601 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfec032440 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.913+0000 7fdffa7fc640 1 -- 172.21.15.1:0/1540208601 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec032810 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.914+0000 7fe001be3640 1 -- 172.21.15.1:0/1540208601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 msgr2=0x7fdffc0ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.914+0000 7fe001be3640 1 --2- 172.21.15.1:0/1540208601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc0ca4a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdfec0097a0 tx=0x7fdfec02a230 comp rx=0 tx=0).stop 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.914+0000 7fe001be3640 1 -- 172.21.15.1:0/1540208601 shutdown_connections 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.914+0000 7fe001be3640 1 --2- 172.21.15.1:0/1540208601 >> v2:172.21.15.1:3300/0 conn(0x7fdffc156e90 0x7fdffc0c7750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.914+0000 7fe001be3640 1 --2- 172.21.15.1:0/1540208601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc0ca4a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.914+0000 7fe001be3640 1 -- 172.21.15.1:0/1540208601 >> 172.21.15.1:0/1540208601 conn(0x7fdffc0745a0 msgr2=0x7fdffc071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.914+0000 7fe001be3640 1 -- 172.21.15.1:0/1540208601 shutdown_connections 2024-05-09T05:10:27.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.914+0000 7fe001be3640 1 -- 172.21.15.1:0/1540208601 wait complete. 2024-05-09T05:10:27.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.916+0000 7fe001be3640 1 Processor -- start 2024-05-09T05:10:27.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.916+0000 7fe001be3640 1 -- start start 2024-05-09T05:10:27.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.917+0000 7fe001be3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc1f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.917+0000 7fe001be3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 0x7fdffc14d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.917+0000 7fe001be3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdffc156e90 0x7fdffc155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.917+0000 7fe001be3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc0cad60 con 0x7fdffc156a80 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.917+0000 7fe001be3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdffc0cb5a0 con 0x7fdffc156e90 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.917+0000 7fe001be3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdffc0ca9e0 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 0x7fdffc14d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffaffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 0x7fdffc14d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50854/0 (socket says 172.21.15.1:50854) 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffaffd640 1 -- 172.21.15.1:0/3543591025 learned_addr learned my addr 172.21.15.1:0/3543591025 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffaffd640 1 -- 172.21.15.1:0/3543591025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 msgr2=0x7fdffc14d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffaffd640 1 -- 172.21.15.1:0/3543591025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 msgr2=0x7fdffc14d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:10:27.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffaffd640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 0x7fdffc14d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffaffd640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 0x7fdffc14d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffb7fe640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc1f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffbfff640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdffc156e90 0x7fdffc155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffb7fe640 1 -- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdffc156e90 msgr2=0x7fdffc155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffb7fe640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdffc156e90 0x7fdffc155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffb7fe640 1 -- 172.21.15.1:0/3543591025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 msgr2=0x7fdffc14d800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffb7fe640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 0x7fdffc14d800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffb7fe640 1 -- 172.21.15.1:0/3543591025 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdffc153c00 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffbfff640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdffc156e90 0x7fdffc155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.918+0000 7fdffb7fe640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc1f6bb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdfec031000 tx=0x7fdfec0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.919+0000 7fdfe3fff640 1 -- 172.21.15.1:0/3543591025 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec042070 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.919+0000 7fdfe3fff640 1 -- 172.21.15.1:0/3543591025 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdfec004240 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.919+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdffc153ee0 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.919+0000 7fdfe3fff640 1 -- 172.21.15.1:0/3543591025 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfec037040 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.919+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fdffc1544a0 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.920+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdffc050870 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.921+0000 7fdfe3fff640 1 -- 172.21.15.1:0/3543591025 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdfec030770 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.921+0000 7fdfe3fff640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdfe8076300 0x7fdfe8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:27.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.921+0000 7fdfe3fff640 1 -- 172.21.15.1:0/3543591025 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdfec0b8dd0 con 0x7fdffc0c5ca0 2024-05-09T05:10:27.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.922+0000 7fdffaffd640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdfe8076300 0x7fdfe8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:27.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.922+0000 7fdffaffd640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdfe8076300 0x7fdfe8078810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdff0007fd0 tx=0x7fdff0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:27.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:27.926+0000 7fdfe3fff640 1 -- 172.21.15.1:0/3543591025 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdfec04c020 con 0x7fdffc0c5ca0 2024-05-09T05:10:28.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.210+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fdffc0cb000 con 0x7fdffc0c5ca0 2024-05-09T05:10:28.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.211+0000 7fdfe3fff640 1 -- 172.21.15.1:0/3543591025 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fdfec0828b0 con 0x7fdffc0c5ca0 2024-05-09T05:10:28.212 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":38,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdfe8076300 msgr2=0x7fdfe8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdfe8076300 0x7fdfe8078810 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fdff0007fd0 tx=0x7fdff0009040 comp rx=0 tx=0).stop 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 msgr2=0x7fdffc1f6bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc1f6bb0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdfec031000 tx=0x7fdfec0046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 shutdown_connections 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdfe8076300 0x7fdfe8078810 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdffc156e90 0x7fdffc155940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdffc156a80 0x7fdffc14d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 --2- 172.21.15.1:0/3543591025 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdffc0c5ca0 0x7fdffc1f6bb0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:28.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 >> 172.21.15.1:0/3543591025 conn(0x7fdffc0745a0 msgr2=0x7fdffc15bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 shutdown_connections 2024-05-09T05:10:28.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:28.214+0000 7fe001be3640 1 -- 172.21.15.1:0/3543591025 wait complete. 2024-05-09T05:10:31.228 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:10:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.352+0000 7fc50a6b8640 1 Processor -- start 2024-05-09T05:10:31.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.352+0000 7fc50a6b8640 1 -- start start 2024-05-09T05:10:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.353+0000 7fc50a6b8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.353+0000 7fc50a6b8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5040c9000 con 0x7fc50406c6f0 2024-05-09T05:10:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.353+0000 7fc50a6b8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc5040c8a60 con 0x7fc5040c9d80 2024-05-09T05:10:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.353+0000 7fc50a6b8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc50406dc90 con 0x7fc5040689a0 2024-05-09T05:10:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.353+0000 7fc508c55640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc50406c6f0 0x7fc50415e1e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53774/0 (socket says 172.21.15.1:53774) 2024-05-09T05:10:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.353+0000 7fc508c55640 1 -- 172.21.15.1:0/3417672712 learned_addr learned my addr 172.21.15.1:0/3417672712 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.353+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.354+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.354+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.354+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.354+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.555+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.555+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.555+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.555+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:31.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.555+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:31.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.956+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:31.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.956+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:31.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.956+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:31.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.956+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:31.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:31.956+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:32.638 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:10:32.640 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:10:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:32.757+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:32.757+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:32.757+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:32.757+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:32.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:32.757+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 msgr2=0x7fc5041548f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc5027fc640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc5040689a0 0x7fc5041548f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 >> v1:172.21.15.196:6789/0 conn(0x7fc5040c9d80 legacy=0x7fc5040c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 >> v1:172.21.15.1:6789/0 conn(0x7fc50406c6f0 legacy=0x7fc50415e1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc5027fc640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc5027fc640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc50406dc90 con 0x7fc4e40036f0 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc4e4008d90 con 0x7fc4e40061d0 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.353+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:34.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:34.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.354+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:34.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:34.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:34.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.555+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:10:34.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:34.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:34.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:34.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:34.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:34.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:34.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:34.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:34.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:34.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:34.956+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:10:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:10:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:10:35.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc4fbfff640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:10:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:10:35.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:35.757+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 msgr2=0x7fc4e40086e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc5027fc640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.196:3300/0 conn(0x7fc4e40061d0 0x7fc4e40086e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 msgr2=0x7fc4e4005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc5027fc640 1 --2- 172.21.15.1:0/3417672712 >> v2:172.21.15.1:3300/0 conn(0x7fc4e40036f0 0x7fc4e4005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc508c55640 1 -- 172.21.15.1:0/3417672712 reap_dead start 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc5027fc640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e40066d0 0x7fc4e4003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc50406df70 con 0x7fc4e40040d0 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc4e400fbf0 con 0x7fc4e4005cd0 2024-05-09T05:10:37.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.352+0000 7fc5027fc640 1 -- 172.21.15.1:0/3417672712 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc50406dc90 con 0x7fc4e40066d0 2024-05-09T05:10:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.353+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e40066d0 0x7fc4e4003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.353+0000 7fc5037fe640 1 -- 172.21.15.1:0/3417672712 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1581761784 0 0) 0x7fc50406df70 con 0x7fc4e40040d0 2024-05-09T05:10:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.353+0000 7fc5037fe640 1 -- 172.21.15.1:0/3417672712 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc4f0003520 con 0x7fc4e40040d0 2024-05-09T05:10:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.353+0000 7fc5037fe640 1 -- 172.21.15.1:0/3417672712 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3098722385 0 0) 0x7fc4e400fbf0 con 0x7fc4e4005cd0 2024-05-09T05:10:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.353+0000 7fc5037fe640 1 -- 172.21.15.1:0/3417672712 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc50406df70 con 0x7fc4e4005cd0 2024-05-09T05:10:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.353+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v1:172.21.15.196:6789/0 conn(0x7fc4e4005cd0 legacy=0x7fc4e400d230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.353+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 >> v1:172.21.15.1:6789/0 conn(0x7fc4e40040d0 legacy=0x7fc4e40044e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.353+0000 7fc503fff640 1 -- 172.21.15.1:0/3417672712 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e4010ee0 con 0x7fc4e40066d0 2024-05-09T05:10:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.354+0000 7fc503fff640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e40066d0 0x7fc4e4003b90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40052d0 tx=0x7fc4f402aae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32ebdfcf6bb69803 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.354+0000 7fc5037fe640 1 -- 172.21.15.1:0/3417672712 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4034020 con 0x7fc4e40066d0 2024-05-09T05:10:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.354+0000 7fc5037fe640 1 -- 172.21.15.1:0/3417672712 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4f4030750 con 0x7fc4e40066d0 2024-05-09T05:10:37.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.354+0000 7fc5037fe640 1 -- 172.21.15.1:0/3417672712 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4f4030b00 con 0x7fc4e40066d0 2024-05-09T05:10:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.355+0000 7fc50a6b8640 1 -- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e40066d0 msgr2=0x7fc4e4003b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.355+0000 7fc50a6b8640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e40066d0 0x7fc4e4003b90 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc4f40052d0 tx=0x7fc4f402aae0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.356+0000 7fc50a6b8640 1 -- 172.21.15.1:0/3417672712 shutdown_connections 2024-05-09T05:10:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.356+0000 7fc50a6b8640 1 --2- 172.21.15.1:0/3417672712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e40066d0 0x7fc4e4003b90 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.356+0000 7fc50a6b8640 1 -- 172.21.15.1:0/3417672712 >> 172.21.15.1:0/3417672712 conn(0x7fc5040ce0e0 msgr2=0x7fc5040ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.356+0000 7fc50a6b8640 1 -- 172.21.15.1:0/3417672712 shutdown_connections 2024-05-09T05:10:37.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.356+0000 7fc50a6b8640 1 -- 172.21.15.1:0/3417672712 wait complete. 2024-05-09T05:10:37.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.358+0000 7fc50a6b8640 1 Processor -- start 2024-05-09T05:10:37.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.358+0000 7fc50a6b8640 1 -- start start 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc50a6b8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4e40040d0 0x7fc5040cbec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc50a6b8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e4005cd0 0x7fc5040cd370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc50a6b8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e40066d0 0x7fc5041e4720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc50a6b8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc50406cb00 con 0x7fc4e40066d0 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc50a6b8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc50415dcc0 con 0x7fc4e40040d0 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc50a6b8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc5041548f0 con 0x7fc4e4005cd0 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc508c55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e40066d0 0x7fc5041e4720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc508c55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e40066d0 0x7fc5041e4720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58098/0 (socket says 172.21.15.1:58098) 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc508c55640 1 -- 172.21.15.1:0/422458052 learned_addr learned my addr 172.21.15.1:0/422458052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e4005cd0 0x7fc5040cd370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.360+0000 7fc503fff640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4e40040d0 0x7fc5040cbec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc508c55640 1 -- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e4005cd0 msgr2=0x7fc5040cd370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc508c55640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e4005cd0 0x7fc5040cd370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc508c55640 1 -- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4e40040d0 msgr2=0x7fc5040cbec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc508c55640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4e40040d0 0x7fc5040cbec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc508c55640 1 -- 172.21.15.1:0/422458052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc5040cd8b0 con 0x7fc4e40066d0 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc4fbfff640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e4005cd0 0x7fc5040cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc508c55640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e40066d0 0x7fc5041e4720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc5040734d0 tx=0x7fc4fc00cb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc5017fa640 1 -- 172.21.15.1:0/422458052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc004d60 con 0x7fc4e40066d0 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc5017fa640 1 -- 172.21.15.1:0/422458052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc4fc00a610 con 0x7fc4e40066d0 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc5041550e0 con 0x7fc4e40066d0 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc5017fa640 1 -- 172.21.15.1:0/422458052 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4fc00a9e0 con 0x7fc4e40066d0 2024-05-09T05:10:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.361+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc5041556a0 con 0x7fc4e40066d0 2024-05-09T05:10:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.362+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc504050870 con 0x7fc4e40066d0 2024-05-09T05:10:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.363+0000 7fc5017fa640 1 -- 172.21.15.1:0/422458052 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc4fc011070 con 0x7fc4e40066d0 2024-05-09T05:10:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.364+0000 7fc5017fa640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4e4077fa0 0x7fc4e407a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.364+0000 7fc5017fa640 1 -- 172.21.15.1:0/422458052 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc4fc096e00 con 0x7fc4e40066d0 2024-05-09T05:10:37.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.364+0000 7fc503fff640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4e4077fa0 0x7fc4e407a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.364+0000 7fc503fff640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4e4077fa0 0x7fc4e407a4b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4006170 tx=0x7fc4f4030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:37.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.371+0000 7fc5017fa640 1 -- 172.21.15.1:0/422458052 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc4fc01a060 con 0x7fc4e40066d0 2024-05-09T05:10:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.691+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc50406cd40 con 0x7fc4e40066d0 2024-05-09T05:10:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.691+0000 7fc5017fa640 1 -- 172.21.15.1:0/422458052 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fc4fc060770 con 0x7fc4e40066d0 2024-05-09T05:10:37.693 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4e4077fa0 msgr2=0x7fc4e407a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4e4077fa0 0x7fc4e407a4b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fc4e4006170 tx=0x7fc4f4030000 comp rx=0 tx=0).stop 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e40066d0 msgr2=0x7fc5041e4720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e40066d0 0x7fc5041e4720 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7fc5040734d0 tx=0x7fc4fc00cb90 comp rx=0 tx=0).stop 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 shutdown_connections 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc4e4077fa0 0x7fc4e407a4b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc4e40066d0 0x7fc5041e4720 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc4e4005cd0 0x7fc5040cd370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 --2- 172.21.15.1:0/422458052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc4e40040d0 0x7fc5040cbec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 >> 172.21.15.1:0/422458052 conn(0x7fc5040ce0e0 msgr2=0x7fc5040c91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:37.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 shutdown_connections 2024-05-09T05:10:37.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.694+0000 7fc50a6b8640 1 -- 172.21.15.1:0/422458052 wait complete. 2024-05-09T05:10:37.709 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:10:37.709 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:10:37.709 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:10:37.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.835+0000 7fa596661640 1 Processor -- start 2024-05-09T05:10:37.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.836+0000 7fa596661640 1 -- start start 2024-05-09T05:10:37.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.837+0000 7fa596661640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5900c89a0 0x7fa5900ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.837+0000 7fa596661640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa5900c46e0 0x7fa590159650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.837+0000 7fa596661640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa5900c3be0 con 0x7fa5900c4af0 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.837+0000 7fa596661640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa5900c9290 con 0x7fa5900c89a0 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.837+0000 7fa596661640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa59006e890 con 0x7fa5900c46e0 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.837+0000 7fa58ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5900c89a0 0x7fa5900ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.837+0000 7fa58ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa5900c89a0 0x7fa5900ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58102/0 (socket says 172.21.15.1:58102) 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.837+0000 7fa58ffff640 1 -- 172.21.15.1:0/2110133068 learned_addr learned my addr 172.21.15.1:0/2110133068 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.838+0000 7fa58ffff640 1 -- 172.21.15.1:0/2110133068 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa5900c46e0 msgr2=0x7fa590159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.838+0000 7fa58ffff640 1 --2- 172.21.15.1:0/2110133068 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa5900c46e0 0x7fa590159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.838+0000 7fa58ffff640 1 -- 172.21.15.1:0/2110133068 >> v1:172.21.15.196:6789/0 conn(0x7fa5900c4af0 legacy=0x7fa590153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:37.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.838+0000 7fa58ffff640 1 -- 172.21.15.1:0/2110133068 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5900c3e60 con 0x7fa5900c89a0 2024-05-09T05:10:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.839+0000 7fa58ffff640 1 --2- 172.21.15.1:0/2110133068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 0x7fa5900ca930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa580009870 tx=0x7fa58002a130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f84c7e54eacfd4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.839+0000 7fa58effd640 1 -- 172.21.15.1:0/2110133068 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa580037070 con 0x7fa5900c89a0 2024-05-09T05:10:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.839+0000 7fa58effd640 1 -- 172.21.15.1:0/2110133068 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa580032440 con 0x7fa5900c89a0 2024-05-09T05:10:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.839+0000 7fa58effd640 1 -- 172.21.15.1:0/2110133068 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa580032810 con 0x7fa5900c89a0 2024-05-09T05:10:37.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.840+0000 7fa596661640 1 -- 172.21.15.1:0/2110133068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 msgr2=0x7fa5900ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.840+0000 7fa596661640 1 --2- 172.21.15.1:0/2110133068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 0x7fa5900ca930 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fa580009870 tx=0x7fa58002a130 comp rx=0 tx=0).stop 2024-05-09T05:10:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.840+0000 7fa596661640 1 -- 172.21.15.1:0/2110133068 shutdown_connections 2024-05-09T05:10:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.840+0000 7fa596661640 1 --2- 172.21.15.1:0/2110133068 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa5900c46e0 0x7fa590159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.840+0000 7fa596661640 1 --2- 172.21.15.1:0/2110133068 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 0x7fa5900ca930 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.840+0000 7fa596661640 1 -- 172.21.15.1:0/2110133068 >> 172.21.15.1:0/2110133068 conn(0x7fa5900747b0 msgr2=0x7fa5900ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.840+0000 7fa596661640 1 -- 172.21.15.1:0/2110133068 shutdown_connections 2024-05-09T05:10:37.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.840+0000 7fa596661640 1 -- 172.21.15.1:0/2110133068 wait complete. 2024-05-09T05:10:37.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.842+0000 7fa596661640 1 Processor -- start 2024-05-09T05:10:37.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.842+0000 7fa596661640 1 -- start start 2024-05-09T05:10:37.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.843+0000 7fa596661640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5900c46e0 0x7fa59015a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa596661640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa5900c4af0 0x7fa59014f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa596661640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 0x7fa590156780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa596661640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa5900c97c0 con 0x7fa5900c89a0 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa596661640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa5900c94e0 con 0x7fa5900c46e0 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa596661640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa5900cae70 con 0x7fa5900c4af0 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa594bfe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 0x7fa590156780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa594bfe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 0x7fa590156780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58112/0 (socket says 172.21.15.1:58112) 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa594bfe640 1 -- 172.21.15.1:0/401369052 learned_addr learned my addr 172.21.15.1:0/401369052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa58f7fe640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa5900c4af0 0x7fa59014f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa58ffff640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5900c46e0 0x7fa59015a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa594bfe640 1 -- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa5900c4af0 msgr2=0x7fa59014f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa594bfe640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa5900c4af0 0x7fa59014f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa594bfe640 1 -- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5900c46e0 msgr2=0x7fa59015a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa594bfe640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa5900c46e0 0x7fa59015a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.844+0000 7fa594bfe640 1 -- 172.21.15.1:0/401369052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5901f3230 con 0x7fa5900c89a0 2024-05-09T05:10:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.845+0000 7fa594bfe640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 0x7fa590156780 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa58800b930 tx=0x7fa58800bdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.845+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa588019070 con 0x7fa5900c89a0 2024-05-09T05:10:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.845+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa58800cb60 con 0x7fa5900c89a0 2024-05-09T05:10:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.845+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa588014440 con 0x7fa5900c89a0 2024-05-09T05:10:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.845+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5901f34d0 con 0x7fa5900c89a0 2024-05-09T05:10:37.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.845+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5901ef780 con 0x7fa5900c89a0 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.846+0000 7fa596661640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5901eff80 0x7fa590156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.846+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa59014fa60 con 0x7fa5901eff80 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.846+0000 7fa58ffff640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5901eff80 0x7fa590156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.847+0000 7fa58ffff640 1 -- 172.21.15.1:0/401369052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa5901f37b0 con 0x7fa5901eff80 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.847+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa58801c070 con 0x7fa5900c89a0 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.847+0000 7fa58ffff640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5901eff80 0x7fa590156cc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa5900cf430 tx=0x7fa5800046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.847+0000 7fa58cff9640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa57c0762d0 0x7fa57c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.847+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa5880971f0 con 0x7fa5900c89a0 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.847+0000 7fa58f7fe640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa57c0762d0 0x7fa57c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.848+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa580043070 con 0x7fa5901eff80 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.848+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa580032e10 con 0x7fa5901eff80 2024-05-09T05:10:37.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.848+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5901eff80 msgr2=0x7fa590156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.848+0000 7fa58cff9640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5901eff80 0x7fa590156cc0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fa5900cf430 tx=0x7fa5800046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.848+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5901eff80 msgr2=0x7fa590156cc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:37.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.848+0000 7fa58cff9640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5901eff80 0x7fa590156cc0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.848+0000 7fa58f7fe640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa57c0762d0 0x7fa57c0787e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa5900c7c80 tx=0x7fa584009060 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:37.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.855+0000 7fa596661640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa590157eb0 0x7fa59014cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:37.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.855+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa59014d3e0 con 0x7fa590157eb0 2024-05-09T05:10:37.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.855+0000 7fa58ffff640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa590157eb0 0x7fa59014cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:37.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.856+0000 7fa58ffff640 1 -- 172.21.15.1:0/401369052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa590050870 con 0x7fa590157eb0 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.856+0000 7fa58ffff640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa590157eb0 0x7fa59014cea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa5900780c0 tx=0x7fa580009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.856+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa580037040 con 0x7fa590157eb0 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.856+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fa58003b7b0 con 0x7fa590157eb0 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.856+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa590157eb0 msgr2=0x7fa59014cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.856+0000 7fa58cff9640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa590157eb0 0x7fa59014cea0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fa5900780c0 tx=0x7fa580009240 comp rx=0 tx=0).stop 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.856+0000 7fa58cff9640 1 -- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa590157eb0 msgr2=0x7fa59014cea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.856+0000 7fa58cff9640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa590157eb0 0x7fa59014cea0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:10:37.858 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 26451, 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:37.859 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:37.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:10:37.861 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:10:37.862 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.863 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:37.864 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:10:37.865 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:37.866 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:10:37.867 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:10:37.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:37.869 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:10:37.870 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:10:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa57c0762d0 msgr2=0x7fa57c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa596661640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa57c0762d0 0x7fa57c0787e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fa5900c7c80 tx=0x7fa584009060 comp rx=0 tx=0).stop 2024-05-09T05:10:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 msgr2=0x7fa590156780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa596661640 1 --2- 172.21.15.1:0/401369052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa5900c89a0 0x7fa590156780 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fa58800b930 tx=0x7fa58800bdf0 comp rx=0 tx=0).stop 2024-05-09T05:10:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa594bfe640 1 -- 172.21.15.1:0/401369052 reap_dead start 2024-05-09T05:10:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa594bfe640 1 -- 172.21.15.1:0/401369052 reap_dead start 2024-05-09T05:10:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 shutdown_connections 2024-05-09T05:10:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 >> 172.21.15.1:0/401369052 conn(0x7fa5900747b0 msgr2=0x7fa590152730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 shutdown_connections 2024-05-09T05:10:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.857+0000 7fa596661640 1 -- 172.21.15.1:0/401369052 wait complete. 2024-05-09T05:10:37.872 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:10:37.872 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:10:37.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:10:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.998+0000 7efef8300640 1 Processor -- start 2024-05-09T05:10:37.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.998+0000 7efef8300640 1 -- start start 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.999+0000 7efef8300640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 0x7efef00717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.999+0000 7efef8300640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef00cb450 con 0x7efef00c8b70 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.999+0000 7efef8300640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efef00c9e50 con 0x7efef0071d30 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:37.999+0000 7efef8300640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efef00caed0 con 0x7efef00c9520 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef609c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7efef00c8b70 0x7efef0156b90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53792/0 (socket says 172.21.15.1:53792) 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef609c640 1 -- 172.21.15.1:0/17630529 learned_addr learned my addr 172.21.15.1:0/17630529 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef589b640 1 --2- 172.21.15.1:0/17630529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 0x7efef00717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef509a640 1 -- 172.21.15.1:0/17630529 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3302552728 0 0) 0x7efef00cb450 con 0x7efef00c8b70 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef509a640 1 -- 172.21.15.1:0/17630529 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efedc003520 con 0x7efef00c8b70 2024-05-09T05:10:38.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef509a640 1 -- 172.21.15.1:0/17630529 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3320118283 0 0) 0x7efef00c9e50 con 0x7efef0071d30 2024-05-09T05:10:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef509a640 1 -- 172.21.15.1:0/17630529 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7efef00cb450 con 0x7efef0071d30 2024-05-09T05:10:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef589b640 1 -- 172.21.15.1:0/17630529 >> v1:172.21.15.196:6789/0 conn(0x7efef0071d30 legacy=0x7efef0159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef589b640 1 -- 172.21.15.1:0/17630529 >> v1:172.21.15.1:6789/0 conn(0x7efef00c8b70 legacy=0x7efef0156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.000+0000 7efef589b640 1 -- 172.21.15.1:0/17630529 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef00cb170 con 0x7efef00c9520 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.001+0000 7efef589b640 1 --2- 172.21.15.1:0/17630529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 0x7efef00717f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efee00097d0 tx=0x7efee002c420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=405982e2e7db0593 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.001+0000 7efef509a640 1 -- 172.21.15.1:0/17630529 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0038070 con 0x7efef00c9520 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.001+0000 7efef509a640 1 -- 172.21.15.1:0/17630529 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efee0031710 con 0x7efef00c9520 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.001+0000 7efef509a640 1 -- 172.21.15.1:0/17630529 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0031ae0 con 0x7efef00c9520 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.002+0000 7efef8300640 1 -- 172.21.15.1:0/17630529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 msgr2=0x7efef00717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.002+0000 7efef8300640 1 --2- 172.21.15.1:0/17630529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 0x7efef00717f0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efee00097d0 tx=0x7efee002c420 comp rx=0 tx=0).stop 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.002+0000 7efef8300640 1 -- 172.21.15.1:0/17630529 shutdown_connections 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.002+0000 7efef8300640 1 --2- 172.21.15.1:0/17630529 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 0x7efef00717f0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.002+0000 7efef8300640 1 -- 172.21.15.1:0/17630529 >> 172.21.15.1:0/17630529 conn(0x7efef006d800 msgr2=0x7efef006dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.002+0000 7efef8300640 1 -- 172.21.15.1:0/17630529 shutdown_connections 2024-05-09T05:10:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.003+0000 7efef8300640 1 -- 172.21.15.1:0/17630529 wait complete. 2024-05-09T05:10:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.004+0000 7efef8300640 1 Processor -- start 2024-05-09T05:10:38.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.004+0000 7efef8300640 1 -- start start 2024-05-09T05:10:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.005+0000 7efef8300640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef0071d30 0x7efef0155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.005+0000 7efef8300640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c8b70 0x7efef00c7660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.005+0000 7efef8300640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 0x7efef01435f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.005+0000 7efef8300640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efef0071210 con 0x7efef0071d30 2024-05-09T05:10:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.005+0000 7efef8300640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efef00c3f30 con 0x7efef00c8b70 2024-05-09T05:10:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.005+0000 7efef8300640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efef00c9a30 con 0x7efef00c9520 2024-05-09T05:10:38.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef0071d30 0x7efef0155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef0071d30 0x7efef0155f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58140/0 (socket says 172.21.15.1:58140) 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 -- 172.21.15.1:0/505174683 learned_addr learned my addr 172.21.15.1:0/505174683 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef589b640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c8b70 0x7efef00c7660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef689d640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 0x7efef01435f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 -- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 msgr2=0x7efef01435f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efef00c9520 0x7efef01435f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 -- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c8b70 msgr2=0x7efef00c7660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c8b70 0x7efef00c7660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 -- 172.21.15.1:0/505174683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efef00c7ba0 con 0x7efef0071d30 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efef609c640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef0071d30 0x7efef0155f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efeec00c400 tx=0x7efeec00c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec017070 con 0x7efef0071d30 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efeec005ce0 con 0x7efef0071d30 2024-05-09T05:10:38.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.006+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efeec0072d0 con 0x7efef0071d30 2024-05-09T05:10:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.007+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efef00c68e0 con 0x7efef0071d30 2024-05-09T05:10:38.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.007+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efef00c6e60 con 0x7efef0071d30 2024-05-09T05:10:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.007+0000 7efef8300640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c7d70 0x7efef0143b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.007+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efef01564a0 con 0x7efef00c7d70 2024-05-09T05:10:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.008+0000 7efef589b640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c7d70 0x7efef0143b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.008+0000 7efef589b640 1 -- 172.21.15.1:0/505174683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efef0144070 con 0x7efef00c7d70 2024-05-09T05:10:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.008+0000 7efef589b640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c7d70 0x7efef0143b30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efef0073740 tx=0x7efee00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.008+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0038050 con 0x7efef00c7d70 2024-05-09T05:10:38.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.009+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efeec01a070 con 0x7efef0071d30 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.009+0000 7efee6ffd640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efed4076200 0x7efed4078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.009+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7efeec096880 con 0x7efef0071d30 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.010+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efee0039040 con 0x7efef00c7d70 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.010+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c7d70 msgr2=0x7efef0143b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.010+0000 7efef689d640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efed4076200 0x7efed4078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.010+0000 7efee6ffd640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c7d70 0x7efef0143b30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efef0073740 tx=0x7efee00046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.010+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c7d70 msgr2=0x7efef0143b30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.010+0000 7efee6ffd640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef00c7d70 0x7efef0143b30 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7efef0073740 tx=0x7efee00046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.010+0000 7efef689d640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efed4076200 0x7efed4078710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efef015a0c0 tx=0x7efee8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.018+0000 7efef8300640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef0154890 0x7efef0154ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.018+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efef0050870 con 0x7efef0154890 2024-05-09T05:10:38.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.018+0000 7efef589b640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef0154890 0x7efef0154ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.018+0000 7efef589b640 1 -- 172.21.15.1:0/505174683 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efef00ca040 con 0x7efef0154890 2024-05-09T05:10:38.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.019+0000 7efef589b640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef0154890 0x7efef0154ca0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efee0009720 tx=0x7efee0031000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.019+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efee0006e10 con 0x7efef0154890 2024-05-09T05:10:38.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.019+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4890 (secure 0 0 0) 0x7efee00093c0 con 0x7efef0154890 2024-05-09T05:10:38.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.019+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef0154890 msgr2=0x7efef0154ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.019+0000 7efee6ffd640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef0154890 0x7efef0154ca0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7efee0009720 tx=0x7efee0031000 comp rx=0 tx=0).stop 2024-05-09T05:10:38.021 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:10:38.021 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:10:38.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:10:38.021 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:10:38.021 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 81593, 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:38.022 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:38.023 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:10:38.024 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:38.025 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.026 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.027 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:38.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:10:38.029 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.030 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.031 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.032 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.033 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.034 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.034 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:10:38.034 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:10:38.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.019+0000 7efee6ffd640 1 -- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef0154890 msgr2=0x7efef0154ca0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:38.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.019+0000 7efee6ffd640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efef0154890 0x7efef0154ca0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efed4076200 msgr2=0x7efed4078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef8300640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efed4076200 0x7efed4078710 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7efef015a0c0 tx=0x7efee8009040 comp rx=0 tx=0).stop 2024-05-09T05:10:38.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef0071d30 msgr2=0x7efef0155f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef8300640 1 --2- 172.21.15.1:0/505174683 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efef0071d30 0x7efef0155f60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7efeec00c400 tx=0x7efeec00c8c0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef689d640 1 -- 172.21.15.1:0/505174683 reap_dead start 2024-05-09T05:10:38.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef689d640 1 -- 172.21.15.1:0/505174683 reap_dead start 2024-05-09T05:10:38.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 shutdown_connections 2024-05-09T05:10:38.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 >> 172.21.15.1:0/505174683 conn(0x7efef006d800 msgr2=0x7efef014fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:38.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 shutdown_connections 2024-05-09T05:10:38.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.020+0000 7efef8300640 1 -- 172.21.15.1:0/505174683 wait complete. 2024-05-09T05:10:38.036 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:10:38.036 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:10:38.037 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:10:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.158+0000 7fc6fb1ce640 1 Processor -- start 2024-05-09T05:10:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.158+0000 7fc6fb1ce640 1 -- start start 2024-05-09T05:10:38.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.159+0000 7fc6fb1ce640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc6f40cbf60 0x7fc6f40cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.159+0000 7fc6fb1ce640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6f40c7970 con 0x7fc6f40cc370 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.159+0000 7fc6fb1ce640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc6f40c7be0 con 0x7fc6f40747f0 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6fb1ce640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc6f40c5600 con 0x7fc6f40cbf60 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f976b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc6f40cc370 0x7fc6f40c5a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53804/0 (socket says 172.21.15.1:53804) 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f976b640 1 -- 172.21.15.1:0/4247355050 learned_addr learned my addr 172.21.15.1:0/4247355050 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f3fff640 1 --2- 172.21.15.1:0/4247355050 >> v2:172.21.15.196:3300/0 conn(0x7fc6f40cbf60 0x7fc6f40cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f37fe640 1 -- 172.21.15.1:0/4247355050 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 398255792 0 0) 0x7fc6f40c7970 con 0x7fc6f40cc370 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f37fe640 1 -- 172.21.15.1:0/4247355050 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc6e4003520 con 0x7fc6f40cc370 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f3fff640 1 -- 172.21.15.1:0/4247355050 >> v1:172.21.15.196:6789/0 conn(0x7fc6f40747f0 legacy=0x7fc6f4152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f37fe640 1 -- 172.21.15.1:0/4247355050 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 702517443 0 0) 0x7fc6f40c7be0 con 0x7fc6f40747f0 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f3fff640 1 -- 172.21.15.1:0/4247355050 >> v1:172.21.15.1:6789/0 conn(0x7fc6f40cc370 legacy=0x7fc6f40c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.160+0000 7fc6f3fff640 1 -- 172.21.15.1:0/4247355050 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f40c7ec0 con 0x7fc6f40cbf60 2024-05-09T05:10:38.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.161+0000 7fc6f3fff640 1 --2- 172.21.15.1:0/4247355050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6f40cbf60 0x7fc6f40cd4c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00097a0 tx=0x7fc6e002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbfcace2a385f73b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.162+0000 7fc6f37fe640 1 -- 172.21.15.1:0/4247355050 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0037070 con 0x7fc6f40cbf60 2024-05-09T05:10:38.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.162+0000 7fc6f37fe640 1 -- 172.21.15.1:0/4247355050 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e0032440 con 0x7fc6f40cbf60 2024-05-09T05:10:38.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.162+0000 7fc6f37fe640 1 -- 172.21.15.1:0/4247355050 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0032810 con 0x7fc6f40cbf60 2024-05-09T05:10:38.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.162+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/4247355050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6f40cbf60 msgr2=0x7fc6f40cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.162+0000 7fc6fb1ce640 1 --2- 172.21.15.1:0/4247355050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6f40cbf60 0x7fc6f40cd4c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fc6e00097a0 tx=0x7fc6e002a1a0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.163+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/4247355050 shutdown_connections 2024-05-09T05:10:38.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.163+0000 7fc6fb1ce640 1 --2- 172.21.15.1:0/4247355050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6f40cbf60 0x7fc6f40cd4c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.163+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/4247355050 >> 172.21.15.1:0/4247355050 conn(0x7fc6f40ce180 msgr2=0x7fc6f40ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:38.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.163+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/4247355050 shutdown_connections 2024-05-09T05:10:38.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.163+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/4247355050 wait complete. 2024-05-09T05:10:38.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.164+0000 7fc6fb1ce640 1 Processor -- start 2024-05-09T05:10:38.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.165+0000 7fc6fb1ce640 1 -- start start 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6fb1ce640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f40747f0 0x7fc6f41f5c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6fb1ce640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6f40cbf60 0x7fc6f41438c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6fb1ce640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6f40cc370 0x7fc6f4143e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6fb1ce640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f40740c0 con 0x7fc6f40cbf60 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6fb1ce640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc6f40c8480 con 0x7fc6f40cc370 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6fb1ce640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f40c61a0 con 0x7fc6f40747f0 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6f40cbf60 0x7fc6f41438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6f3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6f40cbf60 0x7fc6f41438c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58144/0 (socket says 172.21.15.1:58144) 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6f3fff640 1 -- 172.21.15.1:0/2119800194 learned_addr learned my addr 172.21.15.1:0/2119800194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:38.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6f976b640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6f40cc370 0x7fc6f4143e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.166+0000 7fc6f8f6a640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f40747f0 0x7fc6f41f5c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f3fff640 1 -- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f40747f0 msgr2=0x7fc6f41f5c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f3fff640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f40747f0 0x7fc6f41f5c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f3fff640 1 -- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6f40cc370 msgr2=0x7fc6f4143e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f3fff640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc6f40cc370 0x7fc6f4143e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f3fff640 1 -- 172.21.15.1:0/2119800194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6f4144340 con 0x7fc6f40cbf60 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f8f6a640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f40747f0 0x7fc6f41f5c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f3fff640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6f40cbf60 0x7fc6f41438c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0035040 tx=0x7fc6e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0041020 con 0x7fc6f40cbf60 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc6e003d6e0 con 0x7fc6f40cbf60 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6f41f6970 con 0x7fc6f40cbf60 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e0037040 con 0x7fc6f40cbf60 2024-05-09T05:10:38.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.167+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6f41f6690 con 0x7fc6f40cbf60 2024-05-09T05:10:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.168+0000 7fc6fb1ce640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41460b0 0x7fc6f414cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.168+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f4145740 con 0x7fc6f41460b0 2024-05-09T05:10:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.168+0000 7fc6f8f6a640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41460b0 0x7fc6f414cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.169+0000 7fc6f8f6a640 1 -- 172.21.15.1:0/2119800194 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc6f4050870 con 0x7fc6f41460b0 2024-05-09T05:10:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.169+0000 7fc6f8f6a640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41460b0 0x7fc6f414cf30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8000f60 tx=0x7fc6e800e960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.169+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc6e0044020 con 0x7fc6f40cbf60 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f17fa640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6d80762d0 0x7fc6d80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc6e00b7df0 con 0x7fc6f40cbf60 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f976b640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6d80762d0 0x7fc6d80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e80052e0 con 0x7fc6f41460b0 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc6e8018070 con 0x7fc6f41460b0 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41460b0 msgr2=0x7fc6f414cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f17fa640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41460b0 0x7fc6f414cf30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8000f60 tx=0x7fc6e800e960 comp rx=0 tx=0).stop 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41460b0 msgr2=0x7fc6f414cf30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.170+0000 7fc6f17fa640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41460b0 0x7fc6f414cf30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.171+0000 7fc6f976b640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6d80762d0 0x7fc6d80787e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc6f406e200 tx=0x7fc6ec009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.177+0000 7fc6fb1ce640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41f61a0 0x7fc6f41f2560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.177+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc6f414de10 con 0x7fc6f41f61a0 2024-05-09T05:10:38.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.177+0000 7fc6f8f6a640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41f61a0 0x7fc6f41f2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.178+0000 7fc6f8f6a640 1 -- 172.21.15.1:0/2119800194 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc6f414eed0 con 0x7fc6f41f61a0 2024-05-09T05:10:38.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.178+0000 7fc6f8f6a640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41f61a0 0x7fc6f41f2560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8017000 tx=0x7fc6e8000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.178+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc6e8003a00 con 0x7fc6f41f61a0 2024-05-09T05:10:38.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.178+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5061 (secure 0 0 0) 0x7fc6e8003a00 con 0x7fc6f41f61a0 2024-05-09T05:10:38.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.178+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41f61a0 msgr2=0x7fc6f41f2560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.178+0000 7fc6f17fa640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41f61a0 0x7fc6f41f2560 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fc6e8017000 tx=0x7fc6e8000ed0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.178+0000 7fc6f17fa640 1 -- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41f61a0 msgr2=0x7fc6f41f2560 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:38.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.178+0000 7fc6f17fa640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6f41f61a0 0x7fc6f41f2560 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.180 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:10:38.180 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 196176, 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:38.181 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:10:38.182 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:10:38.183 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:10:38.184 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.185 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:38.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:10:38.187 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:10:38.188 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.191 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.192 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:10:38.193 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6d80762d0 msgr2=0x7fc6d80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc6d80762d0 0x7fc6d80787e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc6f406e200 tx=0x7fc6ec009040 comp rx=0 tx=0).stop 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6f40cbf60 msgr2=0x7fc6f41438c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6f40cbf60 0x7fc6f41438c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fc6e0035040 tx=0x7fc6e00046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6f976b640 1 -- 172.21.15.1:0/2119800194 reap_dead start 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 shutdown_connections 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 --2- 172.21.15.1:0/2119800194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6f40cbf60 0x7fc6f41438c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 >> 172.21.15.1:0/2119800194 conn(0x7fc6f40ce180 msgr2=0x7fc6f40cabb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 shutdown_connections 2024-05-09T05:10:38.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.179+0000 7fc6fb1ce640 1 -- 172.21.15.1:0/2119800194 wait complete. 2024-05-09T05:10:38.196 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:10:38.196 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:10:38.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:10:38.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.321+0000 7f9cb513f640 1 Processor -- start 2024-05-09T05:10:38.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.321+0000 7f9cb513f640 1 -- start start 2024-05-09T05:10:38.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.323+0000 7f9cb513f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156a80 0x7f9cb006df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.323+0000 7f9cb513f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f9cb0156e90 0x7f9cb006e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.323+0000 7f9cb513f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9cb00c86a0 con 0x7f9cb00c5ca0 2024-05-09T05:10:38.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.323+0000 7f9cb513f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f9cb00cb910 con 0x7f9cb0156e90 2024-05-09T05:10:38.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.323+0000 7f9cb513f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9cb00c8980 con 0x7f9cb0156a80 2024-05-09T05:10:38.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.323+0000 7f9caed9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9cb00c5ca0 0x7f9cb00ca4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53812/0 (socket says 172.21.15.1:53812) 2024-05-09T05:10:38.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9caed9d640 1 -- 172.21.15.1:0/272364024 learned_addr learned my addr 172.21.15.1:0/272364024 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:38.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9cae59c640 1 --2- 172.21.15.1:0/272364024 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156a80 0x7f9cb006df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9caf59e640 1 --2- 172.21.15.1:0/272364024 >> v2:172.21.15.196:3300/0 conn(0x7f9cb0156e90 0x7f9cb006e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9cadd9b640 1 -- 172.21.15.1:0/272364024 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3697173245 0 0) 0x7f9cb00c86a0 con 0x7f9cb00c5ca0 2024-05-09T05:10:38.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9cadd9b640 1 -- 172.21.15.1:0/272364024 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9c98003520 con 0x7f9cb00c5ca0 2024-05-09T05:10:38.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9cae59c640 1 -- 172.21.15.1:0/272364024 >> v2:172.21.15.196:3300/0 conn(0x7f9cb0156e90 msgr2=0x7f9cb006e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:38.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9cae59c640 1 --2- 172.21.15.1:0/272364024 >> v2:172.21.15.196:3300/0 conn(0x7f9cb0156e90 0x7f9cb006e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9cae59c640 1 -- 172.21.15.1:0/272364024 >> v1:172.21.15.1:6789/0 conn(0x7f9cb00c5ca0 legacy=0x7f9cb00ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9cae59c640 1 -- 172.21.15.1:0/272364024 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb00c8c60 con 0x7f9cb0156a80 2024-05-09T05:10:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.324+0000 7f9caf59e640 1 --2- 172.21.15.1:0/272364024 >> v2:172.21.15.196:3300/0 conn(0x7f9cb0156e90 0x7f9cb006e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:10:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.325+0000 7f9cae59c640 1 --2- 172.21.15.1:0/272364024 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156a80 0x7f9cb006df20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40097a0 tx=0x7f9ca402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=143ce31f01d2da5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.325+0000 7f9cadd9b640 1 -- 172.21.15.1:0/272364024 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4037070 con 0x7f9cb0156a80 2024-05-09T05:10:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cadd9b640 1 -- 172.21.15.1:0/272364024 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca4032440 con 0x7f9cb0156a80 2024-05-09T05:10:38.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cadd9b640 1 -- 172.21.15.1:0/272364024 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca40327f0 con 0x7f9cb0156a80 2024-05-09T05:10:38.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cb513f640 1 -- 172.21.15.1:0/272364024 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156a80 msgr2=0x7f9cb006df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cb513f640 1 --2- 172.21.15.1:0/272364024 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156a80 0x7f9cb006df20 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40097a0 tx=0x7f9ca402a210 comp rx=0 tx=0).stop 2024-05-09T05:10:38.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cb513f640 1 -- 172.21.15.1:0/272364024 shutdown_connections 2024-05-09T05:10:38.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cb513f640 1 --2- 172.21.15.1:0/272364024 >> v2:172.21.15.196:3300/0 conn(0x7f9cb0156e90 0x7f9cb006e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cb513f640 1 --2- 172.21.15.1:0/272364024 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156a80 0x7f9cb006df20 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cb513f640 1 -- 172.21.15.1:0/272364024 >> 172.21.15.1:0/272364024 conn(0x7f9cb00745a0 msgr2=0x7f9cb0071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cb513f640 1 -- 172.21.15.1:0/272364024 shutdown_connections 2024-05-09T05:10:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.326+0000 7f9cb513f640 1 -- 172.21.15.1:0/272364024 wait complete. 2024-05-09T05:10:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.329+0000 7f9cb513f640 1 Processor -- start 2024-05-09T05:10:38.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.329+0000 7f9cb513f640 1 -- start start 2024-05-09T05:10:38.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cb513f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb00c5ca0 0x7f9cb00cd710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cb513f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb0156a80 0x7f9cb00ce890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cb513f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156e90 0x7f9cb00cedd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cb513f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb00cad60 con 0x7f9cb0156a80 2024-05-09T05:10:38.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cb513f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9cb00cb5a0 con 0x7f9cb00c5ca0 2024-05-09T05:10:38.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cb513f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9cb00c7750 con 0x7f9cb0156e90 2024-05-09T05:10:38.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cae59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb0156a80 0x7f9cb00ce890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cae59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb0156a80 0x7f9cb00ce890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58150/0 (socket says 172.21.15.1:58150) 2024-05-09T05:10:38.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9cae59c640 1 -- 172.21.15.1:0/3430916775 learned_addr learned my addr 172.21.15.1:0/3430916775 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:38.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9caf59e640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156e90 0x7f9cb00cedd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.330+0000 7f9caed9d640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb00c5ca0 0x7f9cb00cd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9cae59c640 1 -- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156e90 msgr2=0x7f9cb00cedd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9cae59c640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156e90 0x7f9cb00cedd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9cae59c640 1 -- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb00c5ca0 msgr2=0x7f9cb00cd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9cae59c640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb00c5ca0 0x7f9cb00cd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9cae59c640 1 -- 172.21.15.1:0/3430916775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cb00cdc50 con 0x7f9cb0156a80 2024-05-09T05:10:38.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9cae59c640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb0156a80 0x7f9cb00ce890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40097a0 tx=0x7f9ca40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9c977fe640 1 -- 172.21.15.1:0/3430916775 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4037050 con 0x7f9cb0156a80 2024-05-09T05:10:38.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9c977fe640 1 -- 172.21.15.1:0/3430916775 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9ca4043070 con 0x7f9cb0156a80 2024-05-09T05:10:38.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9c977fe640 1 -- 172.21.15.1:0/3430916775 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4006e10 con 0x7f9cb0156a80 2024-05-09T05:10:38.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cb00cd230 con 0x7f9cb0156a80 2024-05-09T05:10:38.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.331+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cb0159580 con 0x7f9cb0156a80 2024-05-09T05:10:38.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.332+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9cb0050870 con 0x7f9cb0156a80 2024-05-09T05:10:38.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.333+0000 7f9c977fe640 1 -- 172.21.15.1:0/3430916775 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9ca4032810 con 0x7f9cb0156a80 2024-05-09T05:10:38.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.334+0000 7f9c977fe640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c9c076300 0x7f9c9c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:38.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.334+0000 7f9c977fe640 1 -- 172.21.15.1:0/3430916775 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9ca40b7f40 con 0x7f9cb0156a80 2024-05-09T05:10:38.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.334+0000 7f9caed9d640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c9c076300 0x7f9c9c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:38.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.334+0000 7f9caed9d640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c9c076300 0x7f9c9c078810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005880 tx=0x7f9ca0006aa0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:38.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.340+0000 7f9c977fe640 1 -- 172.21.15.1:0/3430916775 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9ca4081820 con 0x7f9cb0156a80 2024-05-09T05:10:38.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.670+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f9cb00cb000 con 0x7f9cb0156a80 2024-05-09T05:10:38.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.671+0000 7f9c977fe640 1 -- 172.21.15.1:0/3430916775 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f9ca4081120 con 0x7f9cb0156a80 2024-05-09T05:10:38.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c9c076300 msgr2=0x7f9c9c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c9c076300 0x7f9c9c078810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9ca0005880 tx=0x7f9ca0006aa0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb0156a80 msgr2=0x7f9cb00ce890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:38.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb0156a80 0x7f9cb00ce890 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40097a0 tx=0x7f9ca40046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 shutdown_connections 2024-05-09T05:10:38.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c9c076300 0x7f9c9c078810 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9cb0156e90 0x7f9cb00cedd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9cb0156a80 0x7f9cb00ce890 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 --2- 172.21.15.1:0/3430916775 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9cb00c5ca0 0x7f9cb00cd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:38.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 >> 172.21.15.1:0/3430916775 conn(0x7f9cb00745a0 msgr2=0x7f9cb00c5520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:38.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 shutdown_connections 2024-05-09T05:10:38.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:38.674+0000 7f9cb513f640 1 -- 172.21.15.1:0/3430916775 wait complete. 2024-05-09T05:10:38.688 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:10:48.689 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:10:48.689 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:10:48.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.809+0000 7f4efeffa640 1 Processor -- start 2024-05-09T05:10:48.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.810+0000 7f4efeffa640 1 -- start start 2024-05-09T05:10:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.811+0000 7f4efeffa640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef8072120 0x7f4ef8156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.811+0000 7f4efeffa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ef815d1d0 0x7f4ef8070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.811+0000 7f4efeffa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4ef80c5e00 con 0x7f4ef8071360 2024-05-09T05:10:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.811+0000 7f4efeffa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4ef80ca340 con 0x7f4ef815d1d0 2024-05-09T05:10:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.811+0000 7f4efeffa640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef80ca620 con 0x7f4ef8072120 2024-05-09T05:10:48.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.811+0000 7f4ef7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4ef815d1d0 0x7f4ef8070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.811+0000 7f4efd597640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4ef8071360 0x7f4ef80caab0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56368/0 (socket says 172.21.15.1:56368) 2024-05-09T05:10:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.811+0000 7f4efd597640 1 -- 172.21.15.1:0/3333319067 learned_addr learned my addr 172.21.15.1:0/3333319067 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.812+0000 7f4efcd96640 1 --2- 172.21.15.1:0/3333319067 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef8072120 0x7f4ef8156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.812+0000 7f4ef7fff640 1 -- 172.21.15.1:0/3333319067 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef8072120 msgr2=0x7f4ef8156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.812+0000 7f4ef7fff640 1 --2- 172.21.15.1:0/3333319067 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef8072120 0x7f4ef8156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.812+0000 7f4ef7fff640 1 -- 172.21.15.1:0/3333319067 >> v1:172.21.15.1:6789/0 conn(0x7f4ef8071360 legacy=0x7f4ef80caab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:48.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.812+0000 7f4ef7fff640 1 -- 172.21.15.1:0/3333319067 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef80c60e0 con 0x7f4ef815d1d0 2024-05-09T05:10:48.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.813+0000 7f4ef7fff640 1 --2- 172.21.15.1:0/3333319067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef815d1d0 0x7f4ef8070e20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40097a0 tx=0x7f4ee402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90e8a93871db3404 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:48.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.813+0000 7f4ef77fe640 1 -- 172.21.15.1:0/3333319067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4037070 con 0x7f4ef815d1d0 2024-05-09T05:10:48.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.813+0000 7f4ef77fe640 1 -- 172.21.15.1:0/3333319067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4ee4032440 con 0x7f4ef815d1d0 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.813+0000 7f4ef77fe640 1 -- 172.21.15.1:0/3333319067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ee4032810 con 0x7f4ef815d1d0 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.814+0000 7f4efeffa640 1 -- 172.21.15.1:0/3333319067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef815d1d0 msgr2=0x7f4ef8070e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.814+0000 7f4efeffa640 1 --2- 172.21.15.1:0/3333319067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef815d1d0 0x7f4ef8070e20 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f4ee40097a0 tx=0x7f4ee402a180 comp rx=0 tx=0).stop 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.814+0000 7f4efeffa640 1 -- 172.21.15.1:0/3333319067 shutdown_connections 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.814+0000 7f4efeffa640 1 --2- 172.21.15.1:0/3333319067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef815d1d0 0x7f4ef8070e20 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.814+0000 7f4efeffa640 1 --2- 172.21.15.1:0/3333319067 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef8072120 0x7f4ef8156a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.814+0000 7f4efeffa640 1 -- 172.21.15.1:0/3333319067 >> 172.21.15.1:0/3333319067 conn(0x7f4ef80731d0 msgr2=0x7f4ef80cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.814+0000 7f4efeffa640 1 -- 172.21.15.1:0/3333319067 shutdown_connections 2024-05-09T05:10:48.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.814+0000 7f4efeffa640 1 -- 172.21.15.1:0/3333319067 wait complete. 2024-05-09T05:10:48.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.816+0000 7f4efeffa640 1 Processor -- start 2024-05-09T05:10:48.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.816+0000 7f4efeffa640 1 -- start start 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efeffa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef8071360 0x7f4ef8159920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efeffa640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8072120 0x7f4ef8158d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efeffa640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef815d1d0 0x7f4ef8155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efeffa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef815d5e0 con 0x7f4ef8071360 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efeffa640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4ef80c66a0 con 0x7f4ef8072120 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efeffa640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4ef806d800 con 0x7f4ef815d1d0 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efcd96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef8071360 0x7f4ef8159920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efcd96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef8071360 0x7f4ef8159920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47652/0 (socket says 172.21.15.1:47652) 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efcd96640 1 -- 172.21.15.1:0/1239014881 learned_addr learned my addr 172.21.15.1:0/1239014881 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:48.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4efd597640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef815d1d0 0x7f4ef8155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.817+0000 7f4ef7fff640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8072120 0x7f4ef8158d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4efcd96640 1 -- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef815d1d0 msgr2=0x7f4ef8155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4efcd96640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef815d1d0 0x7f4ef8155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4efcd96640 1 -- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8072120 msgr2=0x7f4ef8158d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4efcd96640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8072120 0x7f4ef8158d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4efcd96640 1 -- 172.21.15.1:0/1239014881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ef81556b0 con 0x7f4ef8071360 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4efcd96640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef8071360 0x7f4ef8159920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00d5d0 tx=0x7f4eec00da90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4ef57fa640 1 -- 172.21.15.1:0/1239014881 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec004240 con 0x7f4ef8071360 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4ef57fa640 1 -- 172.21.15.1:0/1239014881 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4eec0043e0 con 0x7f4ef8071360 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4ef57fa640 1 -- 172.21.15.1:0/1239014881 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4eec003830 con 0x7f4ef8071360 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ef81f60d0 con 0x7f4ef8071360 2024-05-09T05:10:48.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.818+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ef81f6690 con 0x7f4ef8071360 2024-05-09T05:10:48.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.819+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ef8071180 con 0x7f4ef8071360 2024-05-09T05:10:48.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.820+0000 7f4ef57fa640 1 -- 172.21.15.1:0/1239014881 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4eec0039d0 con 0x7f4ef8071360 2024-05-09T05:10:48.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.821+0000 7f4ef57fa640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4edc076300 0x7f4edc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:48.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.821+0000 7f4ef57fa640 1 -- 172.21.15.1:0/1239014881 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4eec097720 con 0x7f4ef8071360 2024-05-09T05:10:48.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.821+0000 7f4ef7fff640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4edc076300 0x7f4edc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:48.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.822+0000 7f4ef7fff640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4edc076300 0x7f4edc078810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4034000 tx=0x7f4ee402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:48.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:48.827+0000 7f4ef57fa640 1 -- 172.21.15.1:0/1239014881 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4ef8071180 con 0x7f4ef8071360 2024-05-09T05:10:49.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.125+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4ef8050870 con 0x7f4ef8071360 2024-05-09T05:10:49.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.126+0000 7f4ef57fa640 1 -- 172.21.15.1:0/1239014881 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f4eec060a40 con 0x7f4ef8071360 2024-05-09T05:10:49.127 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:10:49.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4edc076300 msgr2=0x7f4edc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4edc076300 0x7f4edc078810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4ee4034000 tx=0x7f4ee402e000 comp rx=0 tx=0).stop 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef8071360 msgr2=0x7f4ef8159920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef8071360 0x7f4ef8159920 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f4eec00d5d0 tx=0x7f4eec00da90 comp rx=0 tx=0).stop 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 shutdown_connections 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4edc076300 0x7f4edc078810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4ef8072120 0x7f4ef8158d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4ef815d1d0 0x7f4ef8155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 --2- 172.21.15.1:0/1239014881 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4ef8071360 0x7f4ef8159920 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 >> 172.21.15.1:0/1239014881 conn(0x7f4ef80731d0 msgr2=0x7f4ef81516d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 shutdown_connections 2024-05-09T05:10:49.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.128+0000 7f4efeffa640 1 -- 172.21.15.1:0/1239014881 wait complete. 2024-05-09T05:10:49.140 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:10:49.140 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:10:49.140 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:10:49.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:10:49.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.266+0000 7f955349a640 1 Processor -- start 2024-05-09T05:10:49.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.266+0000 7f955349a640 1 -- start start 2024-05-09T05:10:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f955349a640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f954c0b4ec0 0x7f954c0cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f955349a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954c15c7b0 con 0x7f954c160460 2024-05-09T05:10:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f955349a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f954c0cca50 con 0x7f954c0ce010 2024-05-09T05:10:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f955349a640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f954c15d0c0 con 0x7f954c0b4ec0 2024-05-09T05:10:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f9551a37640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f954c160460 0x7f954c167130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56372/0 (socket says 172.21.15.1:56372) 2024-05-09T05:10:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f9551a37640 1 -- 172.21.15.1:0/1306416312 learned_addr learned my addr 172.21.15.1:0/1306416312 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f9551236640 1 --2- 172.21.15.1:0/1306416312 >> v2:172.21.15.196:3300/0 conn(0x7f954c0b4ec0 0x7f954c0cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f9543fff640 1 -- 172.21.15.1:0/1306416312 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1726035220 0 0) 0x7f954c15c7b0 con 0x7f954c160460 2024-05-09T05:10:49.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f9543fff640 1 -- 172.21.15.1:0/1306416312 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9538003520 con 0x7f954c160460 2024-05-09T05:10:49.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.268+0000 7f9543fff640 1 -- 172.21.15.1:0/1306416312 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 735328098 0 0) 0x7f954c0cca50 con 0x7f954c0ce010 2024-05-09T05:10:49.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.269+0000 7f9543fff640 1 -- 172.21.15.1:0/1306416312 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f954c15c7b0 con 0x7f954c0ce010 2024-05-09T05:10:49.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.269+0000 7f9551236640 1 -- 172.21.15.1:0/1306416312 >> v1:172.21.15.196:6789/0 conn(0x7f954c0ce010 legacy=0x7f954c15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.269+0000 7f9551236640 1 -- 172.21.15.1:0/1306416312 >> v1:172.21.15.1:6789/0 conn(0x7f954c160460 legacy=0x7f954c167130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.269+0000 7f9551236640 1 -- 172.21.15.1:0/1306416312 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954c0ccd30 con 0x7f954c0b4ec0 2024-05-09T05:10:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.270+0000 7f9551236640 1 --2- 172.21.15.1:0/1306416312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f954c0b4ec0 0x7f954c0cf9d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f95480097a0 tx=0x7f954802a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eaa46b71f68c910 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.270+0000 7f9543fff640 1 -- 172.21.15.1:0/1306416312 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548037070 con 0x7f954c0b4ec0 2024-05-09T05:10:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.270+0000 7f9543fff640 1 -- 172.21.15.1:0/1306416312 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9548032640 con 0x7f954c0b4ec0 2024-05-09T05:10:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.270+0000 7f9543fff640 1 -- 172.21.15.1:0/1306416312 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95480329f0 con 0x7f954c0b4ec0 2024-05-09T05:10:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.271+0000 7f955349a640 1 -- 172.21.15.1:0/1306416312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f954c0b4ec0 msgr2=0x7f954c0cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.271+0000 7f955349a640 1 --2- 172.21.15.1:0/1306416312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f954c0b4ec0 0x7f954c0cf9d0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f95480097a0 tx=0x7f954802a2c0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.271+0000 7f955349a640 1 -- 172.21.15.1:0/1306416312 shutdown_connections 2024-05-09T05:10:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.271+0000 7f955349a640 1 --2- 172.21.15.1:0/1306416312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f954c0b4ec0 0x7f954c0cf9d0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.271+0000 7f955349a640 1 -- 172.21.15.1:0/1306416312 >> 172.21.15.1:0/1306416312 conn(0x7f954c0d2360 msgr2=0x7f954c0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.271+0000 7f955349a640 1 -- 172.21.15.1:0/1306416312 shutdown_connections 2024-05-09T05:10:49.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.271+0000 7f955349a640 1 -- 172.21.15.1:0/1306416312 wait complete. 2024-05-09T05:10:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.273+0000 7f955349a640 1 Processor -- start 2024-05-09T05:10:49.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.273+0000 7f955349a640 1 -- start start 2024-05-09T05:10:49.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f955349a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f954c0b4ec0 0x7f954c1f2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f955349a640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f954c0ce010 0x7f954c14c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f955349a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c160460 0x7f954c15dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f955349a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f954c0cc750 con 0x7f954c160460 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f955349a640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f954c15cd50 con 0x7f954c0ce010 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f955349a640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f954c0cff10 con 0x7f954c0b4ec0 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551a37640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c160460 0x7f954c15dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551a37640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c160460 0x7f954c15dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47668/0 (socket says 172.21.15.1:47668) 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551a37640 1 -- 172.21.15.1:0/3803478180 learned_addr learned my addr 172.21.15.1:0/3803478180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9550a35640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f954c0ce010 0x7f954c14c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551236640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f954c0b4ec0 0x7f954c1f2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551a37640 1 -- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f954c0b4ec0 msgr2=0x7f954c1f2cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551a37640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f954c0b4ec0 0x7f954c1f2cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551a37640 1 -- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f954c0ce010 msgr2=0x7f954c14c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551a37640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f954c0ce010 0x7f954c14c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.274+0000 7f9551a37640 1 -- 172.21.15.1:0/3803478180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f954c15e500 con 0x7f954c160460 2024-05-09T05:10:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.275+0000 7f9551a37640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c160460 0x7f954c15dfc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f954400ca70 tx=0x7f954400cf30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.275+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f954400abd0 con 0x7f954c160460 2024-05-09T05:10:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.275+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f954c15e9d0 con 0x7f954c160460 2024-05-09T05:10:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.275+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f954400ad70 con 0x7f954c160460 2024-05-09T05:10:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.275+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9544012600 con 0x7f954c160460 2024-05-09T05:10:49.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.275+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f954c1e7800 con 0x7f954c160460 2024-05-09T05:10:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.276+0000 7f955349a640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c1e8830 0x7f954c1e79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.276+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f954c050870 con 0x7f954c1e8830 2024-05-09T05:10:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.276+0000 7f9551236640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c1e8830 0x7f954c1e79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.277+0000 7f9551236640 1 -- 172.21.15.1:0/3803478180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f954c156430 con 0x7f954c1e8830 2024-05-09T05:10:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.277+0000 7f9551236640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c1e8830 0x7f954c1e79d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f954c08fd40 tx=0x7f95480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.277+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548042070 con 0x7f954c1e8830 2024-05-09T05:10:49.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95440127a0 con 0x7f954c160460 2024-05-09T05:10:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9541ffb640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95300761d0 0x7f95300786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f95480093c0 con 0x7f954c1e8830 2024-05-09T05:10:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c1e8830 msgr2=0x7f954c1e79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9541ffb640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c1e8830 0x7f954c1e79d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f954c08fd40 tx=0x7f95480046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c1e8830 msgr2=0x7f954c1e79d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9541ffb640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c1e8830 0x7f954c1e79d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9544097b50 con 0x7f954c160460 2024-05-09T05:10:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.278+0000 7f9550a35640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95300761d0 0x7f95300786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.279+0000 7f9550a35640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95300761d0 0x7f95300786e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f954c155e70 tx=0x7f953c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.288+0000 7f955349a640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c0d0180 0x7f954c0d0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.288+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f954c0d0ad0 con 0x7f954c0d0180 2024-05-09T05:10:49.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9551236640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c0d0180 0x7f954c0d0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9551236640 1 -- 172.21.15.1:0/3803478180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f954c0cd2b0 con 0x7f954c0d0180 2024-05-09T05:10:49.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9551236640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c0d0180 0x7f954c0d0590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95480097a0 tx=0x7f9548009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9548037040 con 0x7f954c0d0180 2024-05-09T05:10:49.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f9548048a70 con 0x7f954c0d0180 2024-05-09T05:10:49.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c0d0180 msgr2=0x7f954c0d0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9541ffb640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c0d0180 0x7f954c0d0590 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f95480097a0 tx=0x7f9548009190 comp rx=0 tx=0).stop 2024-05-09T05:10:49.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9541ffb640 1 -- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c0d0180 msgr2=0x7f954c0d0590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:49.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.289+0000 7f9541ffb640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c0d0180 0x7f954c0d0590 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.291 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:10:49.291 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:10:49.291 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:10:49.291 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:10:49.291 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-05-09T05:10:49.291 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 37884, 2024-05-09T05:10:49.291 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:10:49.291 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.292 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.293 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:49.294 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.295 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.296 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.297 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.298 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:10:49.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:10:49.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:10:49.302 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:10:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95300761d0 msgr2=0x7f95300786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95300761d0 0x7f95300786e0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f954c155e70 tx=0x7f953c009040 comp rx=0 tx=0).stop 2024-05-09T05:10:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c160460 msgr2=0x7f954c15dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f9551a37640 1 -- 172.21.15.1:0/3803478180 reap_dead start 2024-05-09T05:10:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c160460 0x7f954c15dfc0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f954400ca70 tx=0x7f954400cf30 comp rx=0 tx=0).stop 2024-05-09T05:10:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 shutdown_connections 2024-05-09T05:10:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 --2- 172.21.15.1:0/3803478180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f954c160460 0x7f954c15dfc0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 >> 172.21.15.1:0/3803478180 conn(0x7f954c0d2360 msgr2=0x7f954c0b5dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 shutdown_connections 2024-05-09T05:10:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.290+0000 7f955349a640 1 -- 172.21.15.1:0/3803478180 wait complete. 2024-05-09T05:10:49.304 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:10:49.304 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:10:49.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:10:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.432+0000 7fe79a28b640 1 Processor -- start 2024-05-09T05:10:49.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.432+0000 7fe79a28b640 1 -- start start 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe79a28b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe7940c6900 0x7fe7941569e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe79a28b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe79406d800 0x7fe7940ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe79a28b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe79415ca00 con 0x7fe794072390 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe79a28b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe794156f20 con 0x7fe79406d800 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe79a28b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe79415cca0 con 0x7fe7940c6900 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe798828640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe79406d800 0x7fe7940ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe7937fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe794072390 0x7fe794070f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56386/0 (socket says 172.21.15.1:56386) 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe7937fe640 1 -- 172.21.15.1:0/312958826 learned_addr learned my addr 172.21.15.1:0/312958826 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:49.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe793fff640 1 --2- 172.21.15.1:0/312958826 >> v2:172.21.15.196:3300/0 conn(0x7fe7940c6900 0x7fe7941569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe798828640 1 -- 172.21.15.1:0/312958826 >> v2:172.21.15.196:3300/0 conn(0x7fe7940c6900 msgr2=0x7fe7941569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe798828640 1 --2- 172.21.15.1:0/312958826 >> v2:172.21.15.196:3300/0 conn(0x7fe7940c6900 0x7fe7941569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe798828640 1 -- 172.21.15.1:0/312958826 >> v1:172.21.15.1:6789/0 conn(0x7fe794072390 legacy=0x7fe794070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe798828640 1 -- 172.21.15.1:0/312958826 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe79415cf80 con 0x7fe79406d800 2024-05-09T05:10:49.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.434+0000 7fe792ffd640 1 -- 172.21.15.1:0/312958826 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 936639562 0 0) 0x7fe79415ca00 con 0x7fe794072390 2024-05-09T05:10:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.435+0000 7fe798828640 1 --2- 172.21.15.1:0/312958826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe79406d800 0x7fe7940ca7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe794074a90 tx=0x7fe78c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=102b929a789dc266 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.435+0000 7fe792ffd640 1 -- 172.21.15.1:0/312958826 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c007740 con 0x7fe79406d800 2024-05-09T05:10:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.435+0000 7fe792ffd640 1 -- 172.21.15.1:0/312958826 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe78c0078e0 con 0x7fe79406d800 2024-05-09T05:10:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.435+0000 7fe792ffd640 1 -- 172.21.15.1:0/312958826 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78c007cb0 con 0x7fe79406d800 2024-05-09T05:10:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.436+0000 7fe79a28b640 1 -- 172.21.15.1:0/312958826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe79406d800 msgr2=0x7fe7940ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.436+0000 7fe79a28b640 1 --2- 172.21.15.1:0/312958826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe79406d800 0x7fe7940ca7a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe794074a90 tx=0x7fe78c02a680 comp rx=0 tx=0).stop 2024-05-09T05:10:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.436+0000 7fe79a28b640 1 -- 172.21.15.1:0/312958826 shutdown_connections 2024-05-09T05:10:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.436+0000 7fe79a28b640 1 --2- 172.21.15.1:0/312958826 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe79406d800 0x7fe7940ca7a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.436+0000 7fe79a28b640 1 --2- 172.21.15.1:0/312958826 >> v2:172.21.15.196:3300/0 conn(0x7fe7940c6900 0x7fe7941569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.436+0000 7fe79a28b640 1 -- 172.21.15.1:0/312958826 >> 172.21.15.1:0/312958826 conn(0x7fe7940d0470 msgr2=0x7fe7940cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:49.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.436+0000 7fe79a28b640 1 -- 172.21.15.1:0/312958826 shutdown_connections 2024-05-09T05:10:49.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.436+0000 7fe79a28b640 1 -- 172.21.15.1:0/312958826 wait complete. 2024-05-09T05:10:49.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.439+0000 7fe79a28b640 1 Processor -- start 2024-05-09T05:10:49.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.439+0000 7fe79a28b640 1 -- start start 2024-05-09T05:10:49.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe79a28b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe79406d800 0x7fe7941f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe79a28b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe794072390 0x7fe7941511f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe79a28b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7940c6900 0x7fe794147e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe79a28b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe794071610 con 0x7fe794072390 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe79a28b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe79415d540 con 0x7fe7940c6900 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe79a28b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe794071790 con 0x7fe79406d800 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe7937fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe794072390 0x7fe7941511f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe7937fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe794072390 0x7fe7941511f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47706/0 (socket says 172.21.15.1:47706) 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe7937fe640 1 -- 172.21.15.1:0/2093871357 learned_addr learned my addr 172.21.15.1:0/2093871357 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe793fff640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe79406d800 0x7fe7941f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe798828640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7940c6900 0x7fe794147e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe7937fe640 1 -- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe79406d800 msgr2=0x7fe7941f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe7937fe640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe79406d800 0x7fe7941f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe7937fe640 1 -- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7940c6900 msgr2=0x7fe794147e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe7937fe640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe7940c6900 0x7fe794147e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.441+0000 7fe7937fe640 1 -- 172.21.15.1:0/2093871357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7941f3300 con 0x7fe794072390 2024-05-09T05:10:49.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.442+0000 7fe7937fe640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe794072390 0x7fe7941511f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c3f0 tx=0x7fe78800c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.442+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7880074e0 con 0x7fe794072390 2024-05-09T05:10:49.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.442+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe7880047e0 con 0x7fe794072390 2024-05-09T05:10:49.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.442+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7941e8890 con 0x7fe794072390 2024-05-09T05:10:49.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.442+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe788004bd0 con 0x7fe794072390 2024-05-09T05:10:49.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.442+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7940cf4b0 con 0x7fe794072390 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.443+0000 7fe79a28b640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148eb0 0x7fe794148350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.443+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe7941e6d80 con 0x7fe794148eb0 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.443+0000 7fe793fff640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148eb0 0x7fe794148350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.443+0000 7fe793fff640 1 -- 172.21.15.1:0/2093871357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7941e8640 con 0x7fe794148eb0 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.444+0000 7fe793fff640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148eb0 0x7fe794148350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe784010820 tx=0x7fe784010ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.444+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe788011070 con 0x7fe794072390 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.444+0000 7fe790ff9640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe780076300 0x7fe780078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.444+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe7880970c0 con 0x7fe794072390 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.444+0000 7fe798828640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe780076300 0x7fe780078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.445+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe78400ea30 con 0x7fe794148eb0 2024-05-09T05:10:49.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.445+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe78400ebd0 con 0x7fe794148eb0 2024-05-09T05:10:49.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.445+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148eb0 msgr2=0x7fe794148350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.445+0000 7fe790ff9640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148eb0 0x7fe794148350 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe784010820 tx=0x7fe784010ce0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.445+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148eb0 msgr2=0x7fe794148350 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:49.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.445+0000 7fe790ff9640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148eb0 0x7fe794148350 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.445+0000 7fe798828640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe780076300 0x7fe780078810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe78c034000 tx=0x7fe78c02f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.452+0000 7fe79a28b640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148890 0x7fe7941f5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.452+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe794148ca0 con 0x7fe794148890 2024-05-09T05:10:49.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.452+0000 7fe793fff640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148890 0x7fe7941f5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.453+0000 7fe793fff640 1 -- 172.21.15.1:0/2093871357 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe794050870 con 0x7fe794148890 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.453+0000 7fe793fff640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148890 0x7fe7941f5a70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe784019000 tx=0x7fe7840046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.453+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7840044f0 con 0x7fe794148890 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.453+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7fe7840044f0 con 0x7fe794148890 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.453+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148890 msgr2=0x7fe7941f5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.453+0000 7fe790ff9640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148890 0x7fe7941f5a70 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe784019000 tx=0x7fe7840046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.453+0000 7fe790ff9640 1 -- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148890 msgr2=0x7fe7941f5a70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.453+0000 7fe790ff9640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe794148890 0x7fe7941f5a70 secure :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fe784019000 tx=0x7fe7840046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:10:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 93026, 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:10:49.455 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.456 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:10:49.457 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.458 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:10:49.459 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.460 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.461 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:10:49.462 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.463 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:10:49.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:10:49.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.466 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:10:49.467 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe780076300 msgr2=0x7fe780078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe79a28b640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe780076300 0x7fe780078810 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe78c034000 tx=0x7fe78c02f000 comp rx=0 tx=0).stop 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe794072390 msgr2=0x7fe7941511f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe79a28b640 1 --2- 172.21.15.1:0/2093871357 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe794072390 0x7fe7941511f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe78800c3f0 tx=0x7fe78800c8b0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe798828640 1 -- 172.21.15.1:0/2093871357 reap_dead start 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe798828640 1 -- 172.21.15.1:0/2093871357 reap_dead start 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 shutdown_connections 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 >> 172.21.15.1:0/2093871357 conn(0x7fe7940d0470 msgr2=0x7fe794154060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 shutdown_connections 2024-05-09T05:10:49.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.454+0000 7fe79a28b640 1 -- 172.21.15.1:0/2093871357 wait complete. 2024-05-09T05:10:49.469 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:10:49.469 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:10:49.469 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:10:49.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.602+0000 7fe76db84640 1 Processor -- start 2024-05-09T05:10:49.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.602+0000 7fe76db84640 1 -- start start 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe76db84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 0x7fe768156b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe76db84640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe7680c9520 0x7fe7680717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe76db84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe7680cb450 con 0x7fe768071d30 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe76db84640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe7680c9e50 con 0x7fe7680c9520 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe76db84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe7680caed0 con 0x7fe7680c8b70 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe767fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe768071d30 0x7fe768159700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56394/0 (socket says 172.21.15.1:56394) 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe767fff640 1 -- 172.21.15.1:0/724313819 learned_addr learned my addr 172.21.15.1:0/724313819 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe7677fe640 1 --2- 172.21.15.1:0/724313819 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 0x7fe768156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe75ffff640 1 --2- 172.21.15.1:0/724313819 >> v2:172.21.15.196:3300/0 conn(0x7fe7680c9520 0x7fe7680717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe7677fe640 1 -- 172.21.15.1:0/724313819 >> v2:172.21.15.196:3300/0 conn(0x7fe7680c9520 msgr2=0x7fe7680717f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe766ffd640 1 -- 172.21.15.1:0/724313819 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1607006670 0 0) 0x7fe7680cb450 con 0x7fe768071d30 2024-05-09T05:10:49.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe7677fe640 1 --2- 172.21.15.1:0/724313819 >> v2:172.21.15.196:3300/0 conn(0x7fe7680c9520 0x7fe7680717f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe7677fe640 1 -- 172.21.15.1:0/724313819 >> v1:172.21.15.1:6789/0 conn(0x7fe768071d30 legacy=0x7fe768159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.604+0000 7fe7677fe640 1 -- 172.21.15.1:0/724313819 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7680cb170 con 0x7fe7680c8b70 2024-05-09T05:10:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.605+0000 7fe7677fe640 1 --2- 172.21.15.1:0/724313819 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 0x7fe768156b90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe7580097a0 tx=0x7fe75802a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ab6b451571789c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.606+0000 7fe766ffd640 1 -- 172.21.15.1:0/724313819 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758037070 con 0x7fe7680c8b70 2024-05-09T05:10:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.606+0000 7fe766ffd640 1 -- 172.21.15.1:0/724313819 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe758032440 con 0x7fe7680c8b70 2024-05-09T05:10:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.606+0000 7fe766ffd640 1 -- 172.21.15.1:0/724313819 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758032810 con 0x7fe7680c8b70 2024-05-09T05:10:49.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.607+0000 7fe76db84640 1 -- 172.21.15.1:0/724313819 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 msgr2=0x7fe768156b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.607+0000 7fe76db84640 1 --2- 172.21.15.1:0/724313819 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 0x7fe768156b90 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fe7580097a0 tx=0x7fe75802a180 comp rx=0 tx=0).stop 2024-05-09T05:10:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.607+0000 7fe76db84640 1 -- 172.21.15.1:0/724313819 shutdown_connections 2024-05-09T05:10:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.607+0000 7fe76db84640 1 --2- 172.21.15.1:0/724313819 >> v2:172.21.15.196:3300/0 conn(0x7fe7680c9520 0x7fe7680717f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.607+0000 7fe76db84640 1 --2- 172.21.15.1:0/724313819 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 0x7fe768156b90 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.607+0000 7fe76db84640 1 -- 172.21.15.1:0/724313819 >> 172.21.15.1:0/724313819 conn(0x7fe76806d800 msgr2=0x7fe76806dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.607+0000 7fe76db84640 1 -- 172.21.15.1:0/724313819 shutdown_connections 2024-05-09T05:10:49.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.607+0000 7fe76db84640 1 -- 172.21.15.1:0/724313819 wait complete. 2024-05-09T05:10:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.609+0000 7fe76db84640 1 Processor -- start 2024-05-09T05:10:49.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.609+0000 7fe76db84640 1 -- start start 2024-05-09T05:10:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.610+0000 7fe76db84640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe768071d30 0x7fe7681f34a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.610+0000 7fe76db84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 0x7fe76814f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.610+0000 7fe76db84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7680c9520 0x7fe7680c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.610+0000 7fe76db84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe768071210 con 0x7fe7680c9520 2024-05-09T05:10:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.610+0000 7fe76db84640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe7680c3f30 con 0x7fe768071d30 2024-05-09T05:10:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.610+0000 7fe76db84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe7681570d0 con 0x7fe7680c8b70 2024-05-09T05:10:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.610+0000 7fe767fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7680c9520 0x7fe7680c46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe767fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7680c9520 0x7fe7680c46c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47712/0 (socket says 172.21.15.1:47712) 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe767fff640 1 -- 172.21.15.1:0/3764411695 learned_addr learned my addr 172.21.15.1:0/3764411695 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe75ffff640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 0x7fe76814f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe7677fe640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe768071d30 0x7fe7681f34a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe767fff640 1 -- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 msgr2=0x7fe76814f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe767fff640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c8b70 0x7fe76814f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe767fff640 1 -- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe768071d30 msgr2=0x7fe7681f34a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe767fff640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe768071d30 0x7fe7681f34a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe767fff640 1 -- 172.21.15.1:0/3764411695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe76814e510 con 0x7fe7680c9520 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe767fff640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7680c9520 0x7fe7680c46c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe76000ecb0 tx=0x7fe76000c640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe760015070 con 0x7fe7680c9520 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe760011670 con 0x7fe7680c9520 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7681f29c0 con 0x7fe7680c9520 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe76000ee60 con 0x7fe7680c9520 2024-05-09T05:10:49.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.611+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7680cdc90 con 0x7fe7680c9520 2024-05-09T05:10:49.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.612+0000 7fe76db84640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c7d40 0x7fe7681ef3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.612+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe7680696d0 con 0x7fe7680c7d40 2024-05-09T05:10:49.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.613+0000 7fe7677fe640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c7d40 0x7fe7681ef3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.613+0000 7fe7677fe640 1 -- 172.21.15.1:0/3764411695 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe7680c8150 con 0x7fe7680c7d40 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.613+0000 7fe7677fe640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c7d40 0x7fe7681ef3b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe768073f00 tx=0x7fe7580046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.613+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe758037040 con 0x7fe7680c7d40 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.614+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe758038040 con 0x7fe7680c7d40 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.614+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c7d40 msgr2=0x7fe7681ef3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.614+0000 7fe764ff9640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c7d40 0x7fe7681ef3b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe768073f00 tx=0x7fe7580046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.614+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c7d40 msgr2=0x7fe7681ef3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.614+0000 7fe764ff9640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe7680c7d40 0x7fe7681ef3b0 secure :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fe768073f00 tx=0x7fe7580046c0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.614+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe760018070 con 0x7fe7680c9520 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.614+0000 7fe764ff9640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe748075db0 0x7fe7480782c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.614+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe760099fe0 con 0x7fe7680c9520 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.615+0000 7fe7677fe640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe748075db0 0x7fe7480782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.615+0000 7fe7677fe640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe748075db0 0x7fe7480782c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe758040000 tx=0x7fe75803d000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.624+0000 7fe76db84640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe768159120 0x7fe768159530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.624+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe768159a70 con 0x7fe768159120 2024-05-09T05:10:49.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.624+0000 7fe75ffff640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe768159120 0x7fe768159530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.624+0000 7fe75ffff640 1 -- 172.21.15.1:0/3764411695 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe7680c5de0 con 0x7fe768159120 2024-05-09T05:10:49.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe75ffff640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe768159120 0x7fe768159530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7681578c0 tx=0x7fe750011cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe750019070 con 0x7fe768159120 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7fe750004240 con 0x7fe768159120 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe768159120 msgr2=0x7fe768159530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe764ff9640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe768159120 0x7fe768159530 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fe7681578c0 tx=0x7fe750011cf0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe764ff9640 1 -- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe768159120 msgr2=0x7fe768159530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe764ff9640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe768159120 0x7fe768159530 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:10:49.626 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 44, 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 207622, 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.627 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.628 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:10:49.629 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:10:49.630 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:10:49.631 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.636 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:10:49.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:10:49.638 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:10:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:10:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:10:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:10:49.639 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:10:49.639 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:10:49.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe748075db0 msgr2=0x7fe7480782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe76db84640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe748075db0 0x7fe7480782c0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fe758040000 tx=0x7fe75803d000 comp rx=0 tx=0).stop 2024-05-09T05:10:49.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7680c9520 msgr2=0x7fe7680c46c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe76db84640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7680c9520 0x7fe7680c46c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fe76000ecb0 tx=0x7fe76000c640 comp rx=0 tx=0).stop 2024-05-09T05:10:49.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.625+0000 7fe767fff640 1 -- 172.21.15.1:0/3764411695 reap_dead start 2024-05-09T05:10:49.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.626+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 shutdown_connections 2024-05-09T05:10:49.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.626+0000 7fe76db84640 1 --2- 172.21.15.1:0/3764411695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe7680c9520 0x7fe7680c46c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.626+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 >> 172.21.15.1:0/3764411695 conn(0x7fe76806d800 msgr2=0x7fe768150b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:49.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.626+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 shutdown_connections 2024-05-09T05:10:49.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.626+0000 7fe76db84640 1 -- 172.21.15.1:0/3764411695 wait complete. 2024-05-09T05:10:49.641 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:10:49.641 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-05-09T05:10:49.641 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:10:49.641 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-05-09T05:10:49.641 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:10:49.642 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T05:10:49.642 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:10:49.656 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T05:10:49.656 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:10:49.656 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:10:49.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.784+0000 7fe10ff14640 1 Processor -- start 2024-05-09T05:10:49.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.785+0000 7fe10ff14640 1 -- start start 2024-05-09T05:10:49.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10ff14640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1080d16b0 0x7fe1080d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10ff14640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe10815ca90 0x7fe10815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10ff14640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe10806f6d0 con 0x7fe10806f1b0 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10ff14640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe1080cee50 con 0x7fe10815ca90 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10ff14640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe1080cd0e0 con 0x7fe1080d16b0 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10d4af640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fe10806f1b0 legacy=0x7fe1080ce740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10e4b1640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7fe10815ca90 msgr2=0x7fe10815cea0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10e4b1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe10815ca90 0x7fe10815cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10dcb0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1080d16b0 0x7fe1080d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10dcb0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1080d16b0 0x7fe1080d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:52334/0 (socket says 172.21.15.1:52334) 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.786+0000 7fe10dcb0640 1 -- 172.21.15.1:0/2968851687 learned_addr learned my addr 172.21.15.1:0/2968851687 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.787+0000 7fe10dcb0640 1 -- 172.21.15.1:0/2968851687 >> v2:172.21.15.196:3300/0 conn(0x7fe10815ca90 msgr2=0x7fe10815cea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.787+0000 7fe10dcb0640 1 --2- 172.21.15.1:0/2968851687 >> v2:172.21.15.196:3300/0 conn(0x7fe10815ca90 0x7fe10815cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.787+0000 7fe10dcb0640 1 -- 172.21.15.1:0/2968851687 >> v1:172.21.15.196:6789/0 conn(0x7fe10806f1b0 legacy=0x7fe1080ce740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.787+0000 7fe10dcb0640 1 -- 172.21.15.1:0/2968851687 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1080cc4d0 con 0x7fe1080d16b0 2024-05-09T05:10:49.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.788+0000 7fe10dcb0640 1 --2- 172.21.15.1:0/2968851687 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1080d16b0 0x7fe1080d03d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe104009870 tx=0x7fe10402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91e8ab1d3cd3f03d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.788+0000 7fe10ccae640 1 -- 172.21.15.1:0/2968851687 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104037070 con 0x7fe1080d16b0 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.788+0000 7fe10ccae640 1 -- 172.21.15.1:0/2968851687 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe104032440 con 0x7fe1080d16b0 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.788+0000 7fe10ccae640 1 -- 172.21.15.1:0/2968851687 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe104032810 con 0x7fe1080d16b0 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.789+0000 7fe10ff14640 1 -- 172.21.15.1:0/2968851687 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1080d16b0 msgr2=0x7fe1080d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.789+0000 7fe10ff14640 1 --2- 172.21.15.1:0/2968851687 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1080d16b0 0x7fe1080d03d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fe104009870 tx=0x7fe10402a270 comp rx=0 tx=0).stop 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.789+0000 7fe10ff14640 1 -- 172.21.15.1:0/2968851687 shutdown_connections 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.789+0000 7fe10ff14640 1 --2- 172.21.15.1:0/2968851687 >> v2:172.21.15.196:3300/0 conn(0x7fe10815ca90 0x7fe10815cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.789+0000 7fe10ff14640 1 --2- 172.21.15.1:0/2968851687 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1080d16b0 0x7fe1080d03d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.789+0000 7fe10ff14640 1 -- 172.21.15.1:0/2968851687 >> 172.21.15.1:0/2968851687 conn(0x7fe1080b3660 msgr2=0x7fe1080b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.789+0000 7fe10ff14640 1 -- 172.21.15.1:0/2968851687 shutdown_connections 2024-05-09T05:10:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.789+0000 7fe10ff14640 1 -- 172.21.15.1:0/2968851687 wait complete. 2024-05-09T05:10:49.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.791+0000 7fe10ff14640 1 Processor -- start 2024-05-09T05:10:49.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.791+0000 7fe10ff14640 1 -- start start 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10ff14640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe10806f1b0 0x7fe1080b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10ff14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1080d16b0 0x7fe1080cafe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10ff14640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe10815ca90 0x7fe1081f6780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10ff14640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe1080ccd70 con 0x7fe1080d16b0 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10ff14640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe1080cca90 con 0x7fe10806f1b0 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10ff14640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe1080d0910 con 0x7fe10815ca90 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10dcb0640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe10806f1b0 msgr2=0x7fe1080b45c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10dcb0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe10806f1b0 0x7fe1080b45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10d4af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1080d16b0 0x7fe1080cafe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10d4af640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1080d16b0 0x7fe1080cafe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37228/0 (socket says 172.21.15.1:37228) 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10d4af640 1 -- 172.21.15.1:0/366839834 learned_addr learned my addr 172.21.15.1:0/366839834 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.792+0000 7fe10e4b1640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe10815ca90 0x7fe1081f6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10d4af640 1 -- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe10815ca90 msgr2=0x7fe1081f6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10d4af640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe10815ca90 0x7fe1081f6780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10d4af640 1 -- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe10806f1b0 msgr2=0x7fe1080b45c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10d4af640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe10806f1b0 0x7fe1080b45c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10d4af640 1 -- 172.21.15.1:0/366839834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1080ca290 con 0x7fe1080d16b0 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10e4b1640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe10815ca90 0x7fe1081f6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10d4af640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1080d16b0 0x7fe1080cafe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800b4d0 tx=0x7fe0f800b990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe0fe7fc640 1 -- 172.21.15.1:0/366839834 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f8003d70 con 0x7fe1080d16b0 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe0fe7fc640 1 -- 172.21.15.1:0/366839834 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0f8015070 con 0x7fe1080d16b0 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1080b4b00 con 0x7fe1080d16b0 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe0fe7fc640 1 -- 172.21.15.1:0/366839834 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0f80052b0 con 0x7fe1080d16b0 2024-05-09T05:10:49.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.793+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1081f6cc0 con 0x7fe1080d16b0 2024-05-09T05:10:49.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.794+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe108050870 con 0x7fe1080d16b0 2024-05-09T05:10:49.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.795+0000 7fe0fe7fc640 1 -- 172.21.15.1:0/366839834 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe0f8005450 con 0x7fe1080d16b0 2024-05-09T05:10:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.796+0000 7fe0fe7fc640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0ec076280 0x7fe0ec078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.796+0000 7fe0fe7fc640 1 -- 172.21.15.1:0/366839834 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe0f80973d0 con 0x7fe1080d16b0 2024-05-09T05:10:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.796+0000 7fe10dcb0640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0ec076280 0x7fe0ec078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:49.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.797+0000 7fe10dcb0640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0ec076280 0x7fe0ec078790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe104034000 tx=0x7fe10402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:49.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:49.801+0000 7fe0fe7fc640 1 -- 172.21.15.1:0/366839834 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe0f8060d40 con 0x7fe1080d16b0 2024-05-09T05:10:50.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.127+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe1080cc090 con 0x7fe1080d16b0 2024-05-09T05:10:50.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.127+0000 7fe0fe7fc640 1 -- 172.21.15.1:0/366839834 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe0f8060640 con 0x7fe1080d16b0 2024-05-09T05:10:50.128 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:10:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0ec076280 msgr2=0x7fe0ec078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0ec076280 0x7fe0ec078790 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe104034000 tx=0x7fe10402e000 comp rx=0 tx=0).stop 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1080d16b0 msgr2=0x7fe1080cafe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1080d16b0 0x7fe1080cafe0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800b4d0 tx=0x7fe0f800b990 comp rx=0 tx=0).stop 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 shutdown_connections 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0ec076280 0x7fe0ec078790 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe10815ca90 0x7fe1081f6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1080d16b0 0x7fe1080cafe0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 --2- 172.21.15.1:0/366839834 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe10806f1b0 0x7fe1080b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 >> 172.21.15.1:0/366839834 conn(0x7fe1080b3660 msgr2=0x7fe108075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 shutdown_connections 2024-05-09T05:10:50.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:50.130+0000 7fe10ff14640 1 -- 172.21.15.1:0/366839834 wait complete. 2024-05-09T05:10:53.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:10:53.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.271+0000 7ff6ab91b640 1 Processor -- start 2024-05-09T05:10:53.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.271+0000 7ff6ab91b640 1 -- start start 2024-05-09T05:10:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6ab91b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff6a40d16b0 0x7ff6a40d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6ab91b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a415ca90 0x7ff6a415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6ab91b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a406f6d0 con 0x7ff6a406f1b0 2024-05-09T05:10:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6ab91b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff6a40cee50 con 0x7ff6a40d16b0 2024-05-09T05:10:53.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6ab91b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a40cd0e0 con 0x7ff6a415ca90 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6a8eb6640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff6a406f1b0 0x7ff6a40ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60992/0 (socket says 172.21.15.1:60992) 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6a96b7640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7ff6a40d16b0 msgr2=0x7ff6a40d03d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6a8eb6640 1 -- 172.21.15.1:0/535470983 learned_addr learned my addr 172.21.15.1:0/535470983 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6a96b7640 1 --2- 172.21.15.1:0/535470983 >> v2:172.21.15.196:3300/0 conn(0x7ff6a40d16b0 0x7ff6a40d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff6a9eb8640 1 --2- 172.21.15.1:0/535470983 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a415ca90 0x7ff6a415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff69bfff640 1 -- 172.21.15.1:0/535470983 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 235208450 0 0) 0x7ff6a406f6d0 con 0x7ff6a406f1b0 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.273+0000 7ff69bfff640 1 -- 172.21.15.1:0/535470983 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff690003520 con 0x7ff6a406f1b0 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.274+0000 7ff6a9eb8640 1 -- 172.21.15.1:0/535470983 >> v2:172.21.15.196:3300/0 conn(0x7ff6a40d16b0 msgr2=0x7ff6a40d03d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.274+0000 7ff6a9eb8640 1 --2- 172.21.15.1:0/535470983 >> v2:172.21.15.196:3300/0 conn(0x7ff6a40d16b0 0x7ff6a40d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.274+0000 7ff6a9eb8640 1 -- 172.21.15.1:0/535470983 >> v1:172.21.15.1:6789/0 conn(0x7ff6a406f1b0 legacy=0x7ff6a40ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:53.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.274+0000 7ff6a9eb8640 1 -- 172.21.15.1:0/535470983 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a40cc4d0 con 0x7ff6a415ca90 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.275+0000 7ff6a9eb8640 1 --2- 172.21.15.1:0/535470983 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a415ca90 0x7ff6a415cea0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009d90 tx=0x7ff69c02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=863cee7b7ba87770 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.275+0000 7ff69bfff640 1 -- 172.21.15.1:0/535470983 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c007740 con 0x7ff6a415ca90 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.275+0000 7ff69bfff640 1 -- 172.21.15.1:0/535470983 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff69c0078e0 con 0x7ff6a415ca90 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.275+0000 7ff69bfff640 1 -- 172.21.15.1:0/535470983 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c007cb0 con 0x7ff6a415ca90 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.276+0000 7ff6ab91b640 1 -- 172.21.15.1:0/535470983 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a415ca90 msgr2=0x7ff6a415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.276+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/535470983 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a415ca90 0x7ff6a415cea0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009d90 tx=0x7ff69c02a710 comp rx=0 tx=0).stop 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.276+0000 7ff6ab91b640 1 -- 172.21.15.1:0/535470983 shutdown_connections 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.276+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/535470983 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a415ca90 0x7ff6a415cea0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.276+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/535470983 >> v2:172.21.15.196:3300/0 conn(0x7ff6a40d16b0 0x7ff6a40d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.276+0000 7ff6ab91b640 1 -- 172.21.15.1:0/535470983 >> 172.21.15.1:0/535470983 conn(0x7ff6a40b3660 msgr2=0x7ff6a40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.276+0000 7ff6ab91b640 1 -- 172.21.15.1:0/535470983 shutdown_connections 2024-05-09T05:10:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.276+0000 7ff6ab91b640 1 -- 172.21.15.1:0/535470983 wait complete. 2024-05-09T05:10:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.278+0000 7ff6ab91b640 1 Processor -- start 2024-05-09T05:10:53.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.278+0000 7ff6ab91b640 1 -- start start 2024-05-09T05:10:53.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6ab91b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a406f1b0 0x7ff6a40ad200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:53.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6ab91b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a40d16b0 0x7ff6a40c5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:53.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6ab91b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a415ca90 0x7ff6a40c7d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:53.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6ab91b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a40ccd70 con 0x7ff6a415ca90 2024-05-09T05:10:53.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6ab91b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a40cca90 con 0x7ff6a40d16b0 2024-05-09T05:10:53.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6ab91b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a40d0910 con 0x7ff6a406f1b0 2024-05-09T05:10:53.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6a8eb6640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a40d16b0 msgr2=0x7ff6a40c5ba0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:10:53.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6a8eb6640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a40d16b0 0x7ff6a40c5ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6a9eb8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a415ca90 0x7ff6a40c7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6a9eb8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a415ca90 0x7ff6a40c7d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37244/0 (socket says 172.21.15.1:37244) 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6a9eb8640 1 -- 172.21.15.1:0/3989551212 learned_addr learned my addr 172.21.15.1:0/3989551212 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.279+0000 7ff6a96b7640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a406f1b0 0x7ff6a40ad200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6a96b7640 1 -- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a40d16b0 msgr2=0x7ff6a40c5ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6a96b7640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a40d16b0 0x7ff6a40c5ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6a96b7640 1 -- 172.21.15.1:0/3989551212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a415ca90 msgr2=0x7ff6a40c7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6a96b7640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a415ca90 0x7ff6a40c7d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6a96b7640 1 -- 172.21.15.1:0/3989551212 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a40ad890 con 0x7ff6a406f1b0 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6a9eb8640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a415ca90 0x7ff6a40c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6a96b7640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a406f1b0 0x7ff6a40ad200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0003930 tx=0x7ff6a000b9b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:53.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff699ffb640 1 -- 172.21.15.1:0/3989551212 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0003dc0 con 0x7ff6a406f1b0 2024-05-09T05:10:53.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff699ffb640 1 -- 172.21.15.1:0/3989551212 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6a0015070 con 0x7ff6a406f1b0 2024-05-09T05:10:53.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff699ffb640 1 -- 172.21.15.1:0/3989551212 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6a0005290 con 0x7ff6a406f1b0 2024-05-09T05:10:53.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a415a0e0 con 0x7ff6a406f1b0 2024-05-09T05:10:53.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.280+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a41f6050 con 0x7ff6a406f1b0 2024-05-09T05:10:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.282+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a4050870 con 0x7ff6a406f1b0 2024-05-09T05:10:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.282+0000 7ff699ffb640 1 -- 172.21.15.1:0/3989551212 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff6a000f070 con 0x7ff6a406f1b0 2024-05-09T05:10:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.283+0000 7ff699ffb640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff688076280 0x7ff688078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.283+0000 7ff699ffb640 1 -- 172.21.15.1:0/3989551212 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff6a0096e20 con 0x7ff6a406f1b0 2024-05-09T05:10:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.283+0000 7ff6a8eb6640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff688076280 0x7ff688078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:53.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.284+0000 7ff6a8eb6640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff688076280 0x7ff688078790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6a41ecef0 tx=0x7ff6940079b0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.288+0000 7ff699ffb640 1 -- 172.21.15.1:0/3989551212 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff6a000d040 con 0x7ff6a406f1b0 2024-05-09T05:10:53.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.611+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff6a40cfad0 con 0x7ff6a406f1b0 2024-05-09T05:10:53.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.611+0000 7ff699ffb640 1 -- 172.21.15.1:0/3989551212 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff6a0060700 con 0x7ff6a406f1b0 2024-05-09T05:10:53.612 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff688076280 msgr2=0x7ff688078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff688076280 0x7ff688078790 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff6a41ecef0 tx=0x7ff6940079b0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a406f1b0 msgr2=0x7ff6a40ad200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a406f1b0 0x7ff6a40ad200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7ff6a0003930 tx=0x7ff6a000b9b0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 shutdown_connections 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff688076280 0x7ff688078790 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a40d16b0 0x7ff6a40c5ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a415ca90 0x7ff6a40c7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 --2- 172.21.15.1:0/3989551212 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a406f1b0 0x7ff6a40ad200 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 >> 172.21.15.1:0/3989551212 conn(0x7ff6a40b3660 msgr2=0x7ff6a4075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 shutdown_connections 2024-05-09T05:10:53.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:53.614+0000 7ff6ab91b640 1 -- 172.21.15.1:0/3989551212 wait complete. 2024-05-09T05:10:56.627 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:10:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.746+0000 7f1e00ac6640 1 Processor -- start 2024-05-09T05:10:56.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.746+0000 7f1e00ac6640 1 -- start start 2024-05-09T05:10:56.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1e00ac6640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc156a80 0x7f1dfc15d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:56.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1e00ac6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dfc0caab0 0x7f1dfc0caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:56.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1e00ac6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1dfc0c5e00 con 0x7f1dfc072120 2024-05-09T05:10:56.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1e00ac6640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f1dfc15d710 con 0x7f1dfc0caab0 2024-05-09T05:10:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1e00ac6640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1dfc0ca340 con 0x7f1dfc156a80 2024-05-09T05:10:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1dfa59c640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f1dfc072120 legacy=0x7f1dfc070e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:10:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1dfad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dfc0caab0 0x7f1dfc0caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1dfad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1dfc0caab0 0x7f1dfc0caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37254/0 (socket says 172.21.15.1:37254) 2024-05-09T05:10:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1dfad9d640 1 -- 172.21.15.1:0/330068458 learned_addr learned my addr 172.21.15.1:0/330068458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:56.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1df9d9b640 1 --2- 172.21.15.1:0/330068458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc156a80 0x7f1dfc15d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1dfad9d640 1 -- 172.21.15.1:0/330068458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc156a80 msgr2=0x7f1dfc15d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:10:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1dfad9d640 1 --2- 172.21.15.1:0/330068458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc156a80 0x7f1dfc15d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1dfad9d640 1 -- 172.21.15.1:0/330068458 >> v1:172.21.15.196:6789/0 conn(0x7f1dfc072120 legacy=0x7f1dfc070e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:10:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.748+0000 7f1dfad9d640 1 -- 172.21.15.1:0/330068458 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc0c60e0 con 0x7f1dfc0caab0 2024-05-09T05:10:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.749+0000 7f1dfad9d640 1 --2- 172.21.15.1:0/330068458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc0caab0 0x7f1dfc0caec0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009e50 tx=0x7f1df402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4526f689ed6d2f6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:56.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1df959a640 1 -- 172.21.15.1:0/330068458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4036070 con 0x7f1dfc0caab0 2024-05-09T05:10:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1df959a640 1 -- 172.21.15.1:0/330068458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df4031600 con 0x7f1dfc0caab0 2024-05-09T05:10:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1df959a640 1 -- 172.21.15.1:0/330068458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df40319d0 con 0x7f1dfc0caab0 2024-05-09T05:10:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1e00ac6640 1 -- 172.21.15.1:0/330068458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc0caab0 msgr2=0x7f1dfc0caec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/330068458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc0caab0 0x7f1dfc0caec0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009e50 tx=0x7f1df402a680 comp rx=0 tx=0).stop 2024-05-09T05:10:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1e00ac6640 1 -- 172.21.15.1:0/330068458 shutdown_connections 2024-05-09T05:10:56.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/330068458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc0caab0 0x7f1dfc0caec0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/330068458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc156a80 0x7f1dfc15d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1e00ac6640 1 -- 172.21.15.1:0/330068458 >> 172.21.15.1:0/330068458 conn(0x7f1dfc0731d0 msgr2=0x7f1dfc0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1e00ac6640 1 -- 172.21.15.1:0/330068458 shutdown_connections 2024-05-09T05:10:56.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.750+0000 7f1e00ac6640 1 -- 172.21.15.1:0/330068458 wait complete. 2024-05-09T05:10:56.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.752+0000 7f1e00ac6640 1 Processor -- start 2024-05-09T05:10:56.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.752+0000 7f1e00ac6640 1 -- start start 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1e00ac6640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dfc072120 0x7f1dfc0b3e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1e00ac6640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc0caab0 0x7f1dfc152c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1e00ac6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc156a80 0x7f1dfc159390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1e00ac6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc156e90 con 0x7f1dfc156a80 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1e00ac6640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1dfc0ca620 con 0x7f1dfc072120 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1e00ac6640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1dfc0c66a0 con 0x7f1dfc0caab0 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfa59c640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dfc072120 msgr2=0x7f1dfc0b3e70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfa59c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dfc072120 0x7f1dfc0b3e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc156a80 0x7f1dfc159390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:56.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc156a80 0x7f1dfc159390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37258/0 (socket says 172.21.15.1:37258) 2024-05-09T05:10:56.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 -- 172.21.15.1:0/3764667466 learned_addr learned my addr 172.21.15.1:0/3764667466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:10:56.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1df9d9b640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc0caab0 0x7f1dfc152c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:56.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 -- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc0caab0 msgr2=0x7f1dfc152c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:56.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc0caab0 0x7f1dfc152c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:56.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 -- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dfc072120 msgr2=0x7f1dfc0b3e70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:10:56.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dfc072120 0x7f1dfc0b3e70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:56.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 -- 172.21.15.1:0/3764667466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1dfc1f3b10 con 0x7f1dfc156a80 2024-05-09T05:10:56.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.754+0000 7f1dfad9d640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc156a80 0x7f1dfc159390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009e50 tx=0x7f1df4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:56.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.755+0000 7f1deaffd640 1 -- 172.21.15.1:0/3764667466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4036050 con 0x7f1dfc156a80 2024-05-09T05:10:56.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.755+0000 7f1deaffd640 1 -- 172.21.15.1:0/3764667466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1df4042070 con 0x7f1dfc156a80 2024-05-09T05:10:56.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.755+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1dfc0b4a00 con 0x7f1dfc156a80 2024-05-09T05:10:56.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.755+0000 7f1deaffd640 1 -- 172.21.15.1:0/3764667466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1df4009510 con 0x7f1dfc156a80 2024-05-09T05:10:56.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.755+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1dfc1f3030 con 0x7f1dfc156a80 2024-05-09T05:10:56.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.756+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1dfc050870 con 0x7f1dfc156a80 2024-05-09T05:10:56.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.757+0000 7f1deaffd640 1 -- 172.21.15.1:0/3764667466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1df4007bf0 con 0x7f1dfc156a80 2024-05-09T05:10:56.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.758+0000 7f1deaffd640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1ddc0762b0 0x7f1ddc0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:10:56.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.758+0000 7f1deaffd640 1 -- 172.21.15.1:0/3764667466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1df40b7e80 con 0x7f1dfc156a80 2024-05-09T05:10:56.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.758+0000 7f1dfa59c640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1ddc0762b0 0x7f1ddc0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:10:56.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.759+0000 7f1dfa59c640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1ddc0762b0 0x7f1ddc0787c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1dec006fd0 tx=0x7f1dec006d00 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:10:56.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:56.762+0000 7f1deaffd640 1 -- 172.21.15.1:0/3764667466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1df404b020 con 0x7f1dfc156a80 2024-05-09T05:10:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.085+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1dfc15a3a0 con 0x7f1dfc156a80 2024-05-09T05:10:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.086+0000 7f1deaffd640 1 -- 172.21.15.1:0/3764667466 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f1df40816d0 con 0x7f1dfc156a80 2024-05-09T05:10:57.086 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":44,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:10:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1ddc0762b0 msgr2=0x7f1ddc0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1ddc0762b0 0x7f1ddc0787c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1dec006fd0 tx=0x7f1dec006d00 comp rx=0 tx=0).stop 2024-05-09T05:10:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc156a80 msgr2=0x7f1dfc159390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:10:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc156a80 0x7f1dfc159390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1df4009e50 tx=0x7f1df4004a40 comp rx=0 tx=0).stop 2024-05-09T05:10:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 shutdown_connections 2024-05-09T05:10:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1ddc0762b0 0x7f1ddc0787c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1dfc156a80 0x7f1dfc159390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1dfc0caab0 0x7f1dfc152c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 --2- 172.21.15.1:0/3764667466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1dfc072120 0x7f1dfc0b3e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:10:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 >> 172.21.15.1:0/3764667466 conn(0x7f1dfc0731d0 msgr2=0x7f1dfc15a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:10:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 shutdown_connections 2024-05-09T05:10:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:10:57.088+0000 7f1e00ac6640 1 -- 172.21.15.1:0/3764667466 wait complete. 2024-05-09T05:11:00.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:11:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.227+0000 7f324111f640 1 Processor -- start 2024-05-09T05:11:00.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.227+0000 7f324111f640 1 -- start start 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.228+0000 7f324111f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.228+0000 7f324111f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.228+0000 7f324111f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f323c0c5e00 con 0x7f323c15d1d0 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.228+0000 7f324111f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f323c0ca340 con 0x7f323c0caab0 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.228+0000 7f324111f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f323c0ca620 con 0x7f323c072120 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.228+0000 7f323b59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.228+0000 7f323a59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f323c15d1d0 0x7f323c070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:56048/0 (socket says 172.21.15.1:56048) 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.228+0000 7f323a59c640 1 -- 172.21.15.1:0/1012958448 learned_addr learned my addr 172.21.15.1:0/1012958448 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:11:00.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:00.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:00.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:00.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:00.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:00.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:00.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.229+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:00.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.430+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:11:00.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:00.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:11:00.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:00.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:00.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:11:00.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:00.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:00.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:00.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:00.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:00.831+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:11:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:01.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:11:01.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:11:01.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:01.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:01.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:01.632+0000 7f323ad9d640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:11:02.698 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:11:02.701 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:11:03.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.228+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 msgr2=0x7f323c156a80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:03.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.228+0000 7f3238d99640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f323c072120 0x7f323c156a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:03.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.228+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 msgr2=0x7f323c0caec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:03.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.228+0000 7f3238d99640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f323c0caab0 0x7f323c0caec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:03.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.228+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 >> v1:172.21.15.1:6789/0 conn(0x7f323c15d1d0 legacy=0x7f323c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:03.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.228+0000 7f3238d99640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 0x7f3228009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:03.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.229+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f323c0c60e0 con 0x7f32280036a0 2024-05-09T05:11:03.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.229+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f323c0ca340 con 0x7f32280073a0 2024-05-09T05:11:03.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.229+0000 7f323a59c640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 msgr2=0x7f3228009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:03.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.229+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> v1:172.21.15.196:6789/0 conn(0x7f32280036a0 legacy=0x7f3228005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:03.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.229+0000 7f323a59c640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 0x7f3228009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:03.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.430+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> v1:172.21.15.196:6789/0 conn(0x7f32280036a0 legacy=0x7f3228005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:03.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.430+0000 7f323a59c640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 msgr2=0x7f3228009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:03.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.430+0000 7f323a59c640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 0x7f3228009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:11:03.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.831+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> v1:172.21.15.196:6789/0 conn(0x7f32280036a0 legacy=0x7f3228005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:03.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.831+0000 7f323a59c640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 msgr2=0x7f3228009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:03.831+0000 7f323a59c640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 0x7f3228009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:11:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:04.632+0000 7f323a59c640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 msgr2=0x7f3228009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:04.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:04.632+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> v1:172.21.15.196:6789/0 conn(0x7f32280036a0 legacy=0x7f3228005b70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:04.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:04.632+0000 7f323a59c640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 0x7f3228009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:11:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 msgr2=0x7f3228009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f3238d99640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.196:3300/0 conn(0x7f32280073a0 0x7f3228009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 >> v1:172.21.15.196:6789/0 conn(0x7f32280036a0 legacy=0x7f3228005b70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:06.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f323b59e640 1 -- 172.21.15.1:0/1012958448 reap_dead start 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f3238d99640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f322800aaf0 0x7f3228005ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f3238d99640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32280098b0 0x7f322800d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f322800b010 con 0x7f3228009f10 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f323c0ca340 con 0x7f322800aaf0 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f3238d99640 1 -- 172.21.15.1:0/1012958448 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f323c0ca620 con 0x7f32280098b0 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f323ad9d640 1 -- 172.21.15.1:0/1012958448 >> v1:172.21.15.196:6789/0 conn(0x7f3228009f10 legacy=0x7f322800a320 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f323a59c640 1 --2- 172.21.15.1:0/1012958448 >> v2:172.21.15.1:3300/0 conn(0x7f322800aaf0 0x7f3228005ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f323b59e640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32280098b0 0x7f322800d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f323a59c640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32280098b0 msgr2=0x7f322800d450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f323a59c640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32280098b0 0x7f322800d450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f323a59c640 1 -- 172.21.15.1:0/1012958448 >> v1:172.21.15.196:6789/0 conn(0x7f3228009f10 legacy=0x7f322800a320 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:06.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.229+0000 7f323a59c640 1 -- 172.21.15.1:0/1012958448 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323c0c60e0 con 0x7f322800aaf0 2024-05-09T05:11:06.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.230+0000 7f323a59c640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322800aaf0 0x7f3228005ac0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3230007fd0 tx=0x7f323002a2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e10b9577fad91404 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.231+0000 7f3239d9b640 1 -- 172.21.15.1:0/1012958448 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32300048a0 con 0x7f322800aaf0 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.231+0000 7f3239d9b640 1 -- 172.21.15.1:0/1012958448 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f323002e070 con 0x7f322800aaf0 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.231+0000 7f3239d9b640 1 -- 172.21.15.1:0/1012958448 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230004c60 con 0x7f322800aaf0 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.232+0000 7f324111f640 1 -- 172.21.15.1:0/1012958448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322800aaf0 msgr2=0x7f3228005ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.232+0000 7f324111f640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322800aaf0 0x7f3228005ac0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3230007fd0 tx=0x7f323002a2b0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.232+0000 7f324111f640 1 -- 172.21.15.1:0/1012958448 shutdown_connections 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.232+0000 7f324111f640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32280098b0 0x7f322800d450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.232+0000 7f324111f640 1 --2- 172.21.15.1:0/1012958448 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f322800aaf0 0x7f3228005ac0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.232+0000 7f324111f640 1 -- 172.21.15.1:0/1012958448 >> 172.21.15.1:0/1012958448 conn(0x7f323c0731d0 msgr2=0x7f323c0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.232+0000 7f324111f640 1 -- 172.21.15.1:0/1012958448 shutdown_connections 2024-05-09T05:11:06.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.232+0000 7f324111f640 1 -- 172.21.15.1:0/1012958448 wait complete. 2024-05-09T05:11:06.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.234+0000 7f324111f640 1 Processor -- start 2024-05-09T05:11:06.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.234+0000 7f324111f640 1 -- start start 2024-05-09T05:11:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f324111f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32280098b0 0x7f323c1f3180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f324111f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3228009f10 0x7f323c1506a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f324111f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f322800aaf0 0x7f323c15eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f324111f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f323c15d5e0 con 0x7f3228009f10 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f324111f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f323c0c66a0 con 0x7f32280098b0 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f324111f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f323c06d800 con 0x7f322800aaf0 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323ad9d640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32280098b0 msgr2=0x7f323c1f3180 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3228009f10 0x7f323c1506a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323ad9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32280098b0 0x7f323c1f3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3228009f10 0x7f323c1506a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37600/0 (socket says 172.21.15.1:37600) 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323a59c640 1 -- 172.21.15.1:0/438956070 learned_addr learned my addr 172.21.15.1:0/438956070 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323b59e640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f322800aaf0 0x7f323c15eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323a59c640 1 -- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f322800aaf0 msgr2=0x7f323c15eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323a59c640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f322800aaf0 0x7f323c15eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323a59c640 1 -- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32280098b0 msgr2=0x7f323c1f3180 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323a59c640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32280098b0 0x7f323c1f3180 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.236+0000 7f323a59c640 1 -- 172.21.15.1:0/438956070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f323c1f36c0 con 0x7f3228009f10 2024-05-09T05:11:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.237+0000 7f323a59c640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3228009f10 0x7f323c1506a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f323002a8a0 tx=0x7f32300012d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.237+0000 7f32237fe640 1 -- 172.21.15.1:0/438956070 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230007270 con 0x7f3228009f10 2024-05-09T05:11:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.237+0000 7f32237fe640 1 -- 172.21.15.1:0/438956070 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3230007410 con 0x7f3228009f10 2024-05-09T05:11:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.237+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f323c15fac0 con 0x7f3228009f10 2024-05-09T05:11:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.237+0000 7f32237fe640 1 -- 172.21.15.1:0/438956070 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3230004e20 con 0x7f3228009f10 2024-05-09T05:11:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.237+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f323c15fd80 con 0x7f3228009f10 2024-05-09T05:11:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.238+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f323c0cb610 con 0x7f3228009f10 2024-05-09T05:11:06.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.239+0000 7f32237fe640 1 -- 172.21.15.1:0/438956070 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3230030040 con 0x7f3228009f10 2024-05-09T05:11:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.240+0000 7f32237fe640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3228077e90 0x7f322807a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.240+0000 7f32237fe640 1 -- 172.21.15.1:0/438956070 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f32300b6ce0 con 0x7f3228009f10 2024-05-09T05:11:06.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.240+0000 7f323ad9d640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3228077e90 0x7f322807a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.240+0000 7f323ad9d640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3228077e90 0x7f322807a3a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3228007370 tx=0x7f322c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.245+0000 7f32237fe640 1 -- 172.21.15.1:0/438956070 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f323c0cb610 con 0x7f3228009f10 2024-05-09T05:11:06.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.552+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f323c06f6f0 con 0x7f3228009f10 2024-05-09T05:11:06.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.552+0000 7f32237fe640 1 -- 172.21.15.1:0/438956070 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f323c06f6f0 con 0x7f3228009f10 2024-05-09T05:11:06.553 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":48,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:11:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3228077e90 msgr2=0x7f322807a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3228077e90 0x7f322807a3a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f3228007370 tx=0x7f322c008040 comp rx=0 tx=0).stop 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3228009f10 msgr2=0x7f323c1506a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3228009f10 0x7f323c1506a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f323002a8a0 tx=0x7f32300012d0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 shutdown_connections 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3228077e90 0x7f322807a3a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f322800aaf0 0x7f323c15eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3228009f10 0x7f323c1506a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 --2- 172.21.15.1:0/438956070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32280098b0 0x7f323c1f3180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 >> 172.21.15.1:0/438956070 conn(0x7f323c0731d0 msgr2=0x7f323c0cf840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 shutdown_connections 2024-05-09T05:11:06.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.555+0000 7f324111f640 1 -- 172.21.15.1:0/438956070 wait complete. 2024-05-09T05:11:06.569 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:11:06.569 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:11:06.569 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:11:06.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.703+0000 7f5422619640 1 Processor -- start 2024-05-09T05:11:06.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.703+0000 7f5422619640 1 -- start start 2024-05-09T05:11:06.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f5422619640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15e300 0x7f541c15e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f5422619640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f541c0c7810 0x7f541c0c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f5422619640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c0cd730 con 0x7f541c1547d0 2024-05-09T05:11:06.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f5422619640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f541c15de70 con 0x7f541c0c7810 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f5422619640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f541c0cc970 con 0x7f541c15e300 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f5420bb6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f541c0c7810 0x7f541c0c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f541bfff640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f541c1547d0 legacy=0x7f541c0cbf60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f5420bb6640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f541c0c7810 0x7f541c0c7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37612/0 (socket says 172.21.15.1:37612) 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.704+0000 7f5420bb6640 1 -- 172.21.15.1:0/510524538 learned_addr learned my addr 172.21.15.1:0/510524538 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.705+0000 7f541b7fe640 1 --2- 172.21.15.1:0/510524538 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15e300 0x7f541c15e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.705+0000 7f5420bb6640 1 -- 172.21.15.1:0/510524538 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15e300 msgr2=0x7f541c15e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.705+0000 7f5420bb6640 1 --2- 172.21.15.1:0/510524538 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15e300 0x7f541c15e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.705+0000 7f5420bb6640 1 -- 172.21.15.1:0/510524538 >> v1:172.21.15.196:6789/0 conn(0x7f541c1547d0 legacy=0x7f541c0cbf60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:06.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.705+0000 7f5420bb6640 1 -- 172.21.15.1:0/510524538 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c06c990 con 0x7f541c0c7810 2024-05-09T05:11:06.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.705+0000 7f5420bb6640 1 --2- 172.21.15.1:0/510524538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0c7810 0x7f541c0c7c20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5414009e50 tx=0x7f541402a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81811e9301fa5fe0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.706+0000 7f541affd640 1 -- 172.21.15.1:0/510524538 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414036070 con 0x7f541c0c7810 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.706+0000 7f541affd640 1 -- 172.21.15.1:0/510524538 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5414031600 con 0x7f541c0c7810 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.706+0000 7f541affd640 1 -- 172.21.15.1:0/510524538 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54140319d0 con 0x7f541c0c7810 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.706+0000 7f5422619640 1 -- 172.21.15.1:0/510524538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0c7810 msgr2=0x7f541c0c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.706+0000 7f5422619640 1 --2- 172.21.15.1:0/510524538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0c7810 0x7f541c0c7c20 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f5414009e50 tx=0x7f541402a5f0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.707+0000 7f5422619640 1 -- 172.21.15.1:0/510524538 shutdown_connections 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.707+0000 7f5422619640 1 --2- 172.21.15.1:0/510524538 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0c7810 0x7f541c0c7c20 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.707+0000 7f5422619640 1 --2- 172.21.15.1:0/510524538 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15e300 0x7f541c15e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.707+0000 7f5422619640 1 -- 172.21.15.1:0/510524538 >> 172.21.15.1:0/510524538 conn(0x7f541c0ce180 msgr2=0x7f541c0ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.707+0000 7f5422619640 1 -- 172.21.15.1:0/510524538 shutdown_connections 2024-05-09T05:11:06.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.707+0000 7f5422619640 1 -- 172.21.15.1:0/510524538 wait complete. 2024-05-09T05:11:06.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.708+0000 7f5422619640 1 Processor -- start 2024-05-09T05:11:06.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.709+0000 7f5422619640 1 -- start start 2024-05-09T05:11:06.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5422619640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0c7810 0x7f541c0caeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5422619640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c1547d0 0x7f541c154dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5422619640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c15e300 0x7f541c13f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5422619640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f541c0cce90 con 0x7f541c15e300 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5422619640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f541c0cc670 con 0x7f541c1547d0 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5422619640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f541c0cd170 con 0x7f541c0c7810 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f541b7fe640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c1547d0 msgr2=0x7f541c154dc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f541b7fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c1547d0 0x7f541c154dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5420bb6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c15e300 0x7f541c13f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5420bb6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c15e300 0x7f541c13f910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37616/0 (socket says 172.21.15.1:37616) 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5420bb6640 1 -- 172.21.15.1:0/479989622 learned_addr learned my addr 172.21.15.1:0/479989622 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f541bfff640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0c7810 0x7f541c0caeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5420bb6640 1 -- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0c7810 msgr2=0x7f541c0caeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5420bb6640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0c7810 0x7f541c0caeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5420bb6640 1 -- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c1547d0 msgr2=0x7f541c154dc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:06.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5420bb6640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c1547d0 0x7f541c154dc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.710+0000 7f5420bb6640 1 -- 172.21.15.1:0/479989622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c0cb3f0 con 0x7f541c15e300 2024-05-09T05:11:06.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.711+0000 7f5420bb6640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c15e300 0x7f541c13f910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5414009f70 tx=0x7f5414004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.711+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f541403f020 con 0x7f541c15e300 2024-05-09T05:11:06.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.711+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5414037040 con 0x7f541c15e300 2024-05-09T05:11:06.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.711+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c1f4310 con 0x7f541c15e300 2024-05-09T05:11:06.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.711+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414036070 con 0x7f541c15e300 2024-05-09T05:11:06.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.711+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c1f48d0 con 0x7f541c15e300 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.712+0000 7f5422619640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c1e7710 0x7f541c13fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.712+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f541c0cb8c0 con 0x7f541c1e7710 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.712+0000 7f541bfff640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c1e7710 0x7f541c13fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.712+0000 7f541bfff640 1 -- 172.21.15.1:0/479989622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f541c06c6f0 con 0x7f541c1e7710 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.713+0000 7f541bfff640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c1e7710 0x7f541c13fe50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f540c003930 tx=0x7f540c009b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.713+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c016070 con 0x7f541c1e7710 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.713+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5414042020 con 0x7f541c15e300 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.713+0000 7f5418ff9640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5408076200 0x7f5408078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.714+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f54140b7b40 con 0x7f541c15e300 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.714+0000 7f541b7fe640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5408076200 0x7f5408078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.714+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f540c012600 con 0x7f541c1e7710 2024-05-09T05:11:06.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.714+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c1e7710 msgr2=0x7f541c13fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.714+0000 7f5418ff9640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c1e7710 0x7f541c13fe50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f540c003930 tx=0x7f540c009b00 comp rx=0 tx=0).stop 2024-05-09T05:11:06.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.714+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c1e7710 msgr2=0x7f541c13fe50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:06.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.714+0000 7f5418ff9640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c1e7710 0x7f541c13fe50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.714+0000 7f541b7fe640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5408076200 0x7f5408078710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f541c140530 tx=0x7f5410008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.720+0000 7f5422619640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c14e480 0x7f541c1f0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.720+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f541c14e8e0 con 0x7f541c14e480 2024-05-09T05:11:06.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.721+0000 7f541bfff640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c14e480 0x7f541c1f0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.721+0000 7f541bfff640 1 -- 172.21.15.1:0/479989622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f541c06cdb0 con 0x7f541c14e480 2024-05-09T05:11:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.721+0000 7f541bfff640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c14e480 0x7f541c1f0760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f540c018000 tx=0x7f540c0059d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.721+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c026600 con 0x7f541c14e480 2024-05-09T05:11:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4881 (secure 0 0 0) 0x7f540c00e420 con 0x7f541c14e480 2024-05-09T05:11:06.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c14e480 msgr2=0x7f541c1f0760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5418ff9640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c14e480 0x7f541c1f0760 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f540c018000 tx=0x7f540c0059d0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5418ff9640 1 -- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c14e480 msgr2=0x7f541c1f0760 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5418ff9640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c14e480 0x7f541c1f0760 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 48, 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 55316, 2024-05-09T05:11:06.723 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:06.724 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:06.725 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:06.726 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:06.727 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:06.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:06.729 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.731 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.732 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.733 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.734 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:06.735 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:06.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5408076200 msgr2=0x7f5408078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5408076200 0x7f5408078710 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f541c140530 tx=0x7f5410008040 comp rx=0 tx=0).stop 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c15e300 msgr2=0x7f541c13f910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c15e300 0x7f541c13f910 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f5414009f70 tx=0x7f5414004a40 comp rx=0 tx=0).stop 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5420bb6640 1 -- 172.21.15.1:0/479989622 reap_dead start 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 shutdown_connections 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 --2- 172.21.15.1:0/479989622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c15e300 0x7f541c13f910 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 >> 172.21.15.1:0/479989622 conn(0x7f541c0ce180 msgr2=0x7f541c152c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 shutdown_connections 2024-05-09T05:11:06.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.722+0000 7f5422619640 1 -- 172.21.15.1:0/479989622 wait complete. 2024-05-09T05:11:06.737 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:11:06.738 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:11:06.738 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:11:06.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.859+0000 7f782c8b8640 1 Processor -- start 2024-05-09T05:11:06.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.859+0000 7f782c8b8640 1 -- start start 2024-05-09T05:11:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f782c8b8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7828152a40 0x7f7828152e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f782c8b8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 0x7f7828070e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f782c8b8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78280719e0 con 0x7f78280c7740 2024-05-09T05:11:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f782c8b8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f78280c5ed0 con 0x7f7828152a40 2024-05-09T05:11:06.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f782c8b8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7828071ca0 con 0x7f7828070a70 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f782659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7828152a40 0x7f7828152e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f7826d9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f78280c7740 legacy=0x7f78280c7b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f782659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7828152a40 0x7f7828152e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37632/0 (socket says 172.21.15.1:37632) 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f782659c640 1 -- 172.21.15.1:0/2945327824 learned_addr learned my addr 172.21.15.1:0/2945327824 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.861+0000 7f7825d9b640 1 --2- 172.21.15.1:0/2945327824 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 0x7f7828070e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.862+0000 7f782659c640 1 -- 172.21.15.1:0/2945327824 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 msgr2=0x7f7828070e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.862+0000 7f782659c640 1 --2- 172.21.15.1:0/2945327824 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 0x7f7828070e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.862+0000 7f782659c640 1 -- 172.21.15.1:0/2945327824 >> v1:172.21.15.196:6789/0 conn(0x7f78280c7740 legacy=0x7f78280c7b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:06.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.862+0000 7f782659c640 1 -- 172.21.15.1:0/2945327824 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7828071f80 con 0x7f7828152a40 2024-05-09T05:11:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.863+0000 7f782659c640 1 --2- 172.21.15.1:0/2945327824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7828152a40 0x7f7828152e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7818009870 tx=0x7f781802a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83fb493bbd1ef09f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.863+0000 7f782559a640 1 -- 172.21.15.1:0/2945327824 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818037070 con 0x7f7828152a40 2024-05-09T05:11:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.863+0000 7f782559a640 1 -- 172.21.15.1:0/2945327824 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7818032440 con 0x7f7828152a40 2024-05-09T05:11:06.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.863+0000 7f782559a640 1 -- 172.21.15.1:0/2945327824 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818032810 con 0x7f7828152a40 2024-05-09T05:11:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.864+0000 7f782c8b8640 1 -- 172.21.15.1:0/2945327824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7828152a40 msgr2=0x7f7828152e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.864+0000 7f782c8b8640 1 --2- 172.21.15.1:0/2945327824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7828152a40 0x7f7828152e50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7818009870 tx=0x7f781802a250 comp rx=0 tx=0).stop 2024-05-09T05:11:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.864+0000 7f782c8b8640 1 -- 172.21.15.1:0/2945327824 shutdown_connections 2024-05-09T05:11:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.864+0000 7f782c8b8640 1 --2- 172.21.15.1:0/2945327824 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 0x7f7828070e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.864+0000 7f782c8b8640 1 --2- 172.21.15.1:0/2945327824 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7828152a40 0x7f7828152e50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.864+0000 7f782c8b8640 1 -- 172.21.15.1:0/2945327824 >> 172.21.15.1:0/2945327824 conn(0x7f78280caf40 msgr2=0x7f78280cb350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.864+0000 7f782c8b8640 1 -- 172.21.15.1:0/2945327824 shutdown_connections 2024-05-09T05:11:06.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.864+0000 7f782c8b8640 1 -- 172.21.15.1:0/2945327824 wait complete. 2024-05-09T05:11:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.866+0000 7f782c8b8640 1 Processor -- start 2024-05-09T05:11:06.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.867+0000 7f782c8b8640 1 -- start start 2024-05-09T05:11:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f782c8b8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 0x7f7828143d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f782c8b8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78280c7740 0x7f782815c110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f782c8b8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7828152a40 0x7f782815c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f782c8b8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f78280c5b60 con 0x7f78280c7740 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f782c8b8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f78280c5880 con 0x7f7828152a40 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f782c8b8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f78280c72d0 con 0x7f7828070a70 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f7825d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78280c7740 0x7f782815c110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f7826d9d640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7828152a40 msgr2=0x7f782815c650 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.868+0000 7f7826d9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7828152a40 0x7f782815c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7825d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78280c7740 0x7f782815c110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37644/0 (socket says 172.21.15.1:37644) 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7825d9b640 1 -- 172.21.15.1:0/1162647573 learned_addr learned my addr 172.21.15.1:0/1162647573 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f782659c640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 0x7f7828143d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7825d9b640 1 -- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 msgr2=0x7f7828143d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7825d9b640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7828070a70 0x7f7828143d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7825d9b640 1 -- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7828152a40 msgr2=0x7f782815c650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7825d9b640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7828152a40 0x7f782815c650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7825d9b640 1 -- 172.21.15.1:0/1162647573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f782814c170 con 0x7f78280c7740 2024-05-09T05:11:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7825d9b640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78280c7740 0x7f782815c110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f781c00e8a0 tx=0x7f781c00ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c017070 con 0x7f78280c7740 2024-05-09T05:11:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f781c00c940 con 0x7f78280c7740 2024-05-09T05:11:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f782814a020 con 0x7f78280c7740 2024-05-09T05:11:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781c00cd30 con 0x7f78280c7740 2024-05-09T05:11:06.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.869+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f782814a5e0 con 0x7f78280c7740 2024-05-09T05:11:06.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.870+0000 7f782c8b8640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281f3900 0x7f7828157a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.871+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f782814c450 con 0x7f78281f3900 2024-05-09T05:11:06.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.871+0000 7f782659c640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281f3900 0x7f7828157a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.871+0000 7f782659c640 1 -- 172.21.15.1:0/1162647573 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7828050870 con 0x7f78281f3900 2024-05-09T05:11:06.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.871+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f781c01a070 con 0x7f78280c7740 2024-05-09T05:11:06.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.871+0000 7f782659c640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281f3900 0x7f7828157a70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f78280c7660 tx=0x7f78180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.872+0000 7f7816ffd640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080762b0 0x7f78080787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.872+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f781c097340 con 0x7f78280c7740 2024-05-09T05:11:06.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.872+0000 7f7826d9d640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080762b0 0x7f78080787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.872+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818037040 con 0x7f78281f3900 2024-05-09T05:11:06.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.873+0000 7f7826d9d640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080762b0 0x7f78080787c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f782814b140 tx=0x7f78200091f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.875+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f781803e660 con 0x7f78281f3900 2024-05-09T05:11:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.875+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281f3900 msgr2=0x7f7828157a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.875+0000 7f7816ffd640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281f3900 0x7f7828157a70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f78280c7660 tx=0x7f78180046c0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.875+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281f3900 msgr2=0x7f7828157a70 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:06.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.875+0000 7f7816ffd640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281f3900 0x7f7828157a70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.882+0000 7f782c8b8640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281536f0 0x7f7828157fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:06.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.882+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f782814a7b0 con 0x7f78281536f0 2024-05-09T05:11:06.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.882+0000 7f782659c640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281536f0 0x7f7828157fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:06.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.882+0000 7f782659c640 1 -- 172.21.15.1:0/1162647573 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f78280cb610 con 0x7f78281536f0 2024-05-09T05:11:06.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.882+0000 7f782659c640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281536f0 0x7f7828157fb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f78280780e0 tx=0x7f7818008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7818037030 con 0x7f78281536f0 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5050 (secure 0 0 0) 0x7f781803ece0 con 0x7f78281536f0 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281536f0 msgr2=0x7f7828157fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f7816ffd640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281536f0 0x7f7828157fb0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f78280780e0 tx=0x7f7818008030 comp rx=0 tx=0).stop 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f7816ffd640 1 -- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281536f0 msgr2=0x7f7828157fb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f7816ffd640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78281536f0 0x7f7828157fb0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 48, 2024-05-09T05:11:06.884 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 224881, 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:06.885 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:06.886 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:06.887 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:06.889 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.890 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:06.891 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:06.892 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.893 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.894 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.895 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.896 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:11:06.897 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:06.898 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:06.898 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:06.898 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:11:06.898 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:06.898 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:06.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:06.898 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:06.898 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080762b0 msgr2=0x7f78080787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f782c8b8640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080762b0 0x7f78080787c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f782814b140 tx=0x7f78200091f0 comp rx=0 tx=0).stop 2024-05-09T05:11:06.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78280c7740 msgr2=0x7f782815c110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f782c8b8640 1 --2- 172.21.15.1:0/1162647573 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78280c7740 0x7f782815c110 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f781c00e8a0 tx=0x7f781c00ed60 comp rx=0 tx=0).stop 2024-05-09T05:11:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f7826d9d640 1 -- 172.21.15.1:0/1162647573 reap_dead start 2024-05-09T05:11:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f7826d9d640 1 -- 172.21.15.1:0/1162647573 reap_dead start 2024-05-09T05:11:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 shutdown_connections 2024-05-09T05:11:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 >> 172.21.15.1:0/1162647573 conn(0x7f78280caf40 msgr2=0x7f78280ca250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 shutdown_connections 2024-05-09T05:11:06.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:06.883+0000 7f782c8b8640 1 -- 172.21.15.1:0/1162647573 wait complete. 2024-05-09T05:11:06.900 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:11:06.900 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:11:26.901 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:11:26.901 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-05-09T05:11:26.901 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T05:11:26.901 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T05:11:26.904 INFO:tasks.ceph.mon.b:Started 2024-05-09T05:11:26.904 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:11:26.904 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:11:27.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.024+0000 7f5a515e3640 1 Processor -- start 2024-05-09T05:11:27.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.024+0000 7f5a515e3640 1 -- start start 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.026+0000 7f5a515e3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5a4c0cd350 0x7f5a4c15ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.026+0000 7f5a515e3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a4c06d8a0 con 0x7f5a4c068a50 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a515e3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5a4c15a680 con 0x7f5a4c06c6f0 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a515e3640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f5a4c06db40 con 0x7f5a4c0cd350 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5a4c068a50 0x7f5a4c151070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51684/0 (socket says 172.21.15.1:51684) 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4affd640 1 -- 172.21.15.1:0/2495085554 learned_addr learned my addr 172.21.15.1:0/2495085554 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4b7fe640 1 --2- 172.21.15.1:0/2495085554 >> v2:172.21.15.196:3300/0 conn(0x7f5a4c0cd350 0x7f5a4c15ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4b7fe640 1 -- 172.21.15.1:0/2495085554 >> v2:172.21.15.196:3300/0 conn(0x7f5a4c0cd350 msgr2=0x7f5a4c15ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4b7fe640 1 -- 172.21.15.1:0/2495085554 >> v2:172.21.15.196:3300/0 conn(0x7f5a4c0cd350 msgr2=0x7f5a4c15ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4b7fe640 1 --2- 172.21.15.1:0/2495085554 >> v2:172.21.15.196:3300/0 conn(0x7f5a4c0cd350 0x7f5a4c15ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4b7fe640 1 --2- 172.21.15.1:0/2495085554 >> v2:172.21.15.196:3300/0 conn(0x7f5a4c0cd350 0x7f5a4c15ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 298462619 0 0) 0x7f5a4c06d8a0 con 0x7f5a4c068a50 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5a38003520 con 0x7f5a4c068a50 2024-05-09T05:11:27.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3724807465 0 0) 0x7f5a38003520 con 0x7f5a4c068a50 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 >> v2:172.21.15.196:3300/0 conn(0x7f5a4c0cd350 msgr2=0x7f5a4c15ab80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4a7fc640 1 --2- 172.21.15.1:0/2495085554 >> v2:172.21.15.196:3300/0 conn(0x7f5a4c0cd350 0x7f5a4c15ab80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 >> v1:172.21.15.196:6789/0 conn(0x7f5a4c06c6f0 legacy=0x7f5a4c0c77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4c06de20 con 0x7f5a4c068a50 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.027+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f5a3c001710 con 0x7f5a4c068a50 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 >> v1:172.21.15.1:6789/0 conn(0x7f5a4c068a50 legacy=0x7f5a4c151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4a7fc640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a38003a50 0x7f5a38003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4a7fc640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a38004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4a7fc640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a380053f0 0x7f5a3800ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c06db40 con 0x7f5a38004800 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5a38010770 con 0x7f5a380053f0 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5a38010970 con 0x7f5a38003a50 2024-05-09T05:11:27.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a43fff640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a38004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4b7fe640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a380053f0 0x7f5a3800ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4affd640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a38003a50 0x7f5a38003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a43fff640 1 -- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a38003a50 msgr2=0x7f5a38003ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a43fff640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a38003a50 0x7f5a38003ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a43fff640 1 -- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a380053f0 msgr2=0x7f5a3800ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a43fff640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a380053f0 0x7f5a3800ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a43fff640 1 -- 172.21.15.1:0/2495085554 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a38003520 con 0x7f5a38004800 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4b7fe640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a380053f0 0x7f5a3800ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:11:27.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.028+0000 7f5a4b7fe640 1 -- 172.21.15.1:0/2495085554 reap_dead start 2024-05-09T05:11:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.029+0000 7f5a43fff640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a38004c10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5a34001790 tx=0x7f5a38005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51c32fad83d2eb1e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.030+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a340013b0 con 0x7f5a38004800 2024-05-09T05:11:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.030+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a340052d0 con 0x7f5a38004800 2024-05-09T05:11:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.030+0000 7f5a4a7fc640 1 -- 172.21.15.1:0/2495085554 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3402aac0 con 0x7f5a38004800 2024-05-09T05:11:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.031+0000 7f5a515e3640 1 -- 172.21.15.1:0/2495085554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 msgr2=0x7f5a38004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:27.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.031+0000 7f5a515e3640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a38004c10 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f5a34001790 tx=0x7f5a38005830 comp rx=0 tx=0).stop 2024-05-09T05:11:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.031+0000 7f5a515e3640 1 -- 172.21.15.1:0/2495085554 shutdown_connections 2024-05-09T05:11:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.031+0000 7f5a515e3640 1 --2- 172.21.15.1:0/2495085554 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a38004c10 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.031+0000 7f5a515e3640 1 -- 172.21.15.1:0/2495085554 >> 172.21.15.1:0/2495085554 conn(0x7f5a4c0ce110 msgr2=0x7f5a4c0ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.031+0000 7f5a515e3640 1 -- 172.21.15.1:0/2495085554 shutdown_connections 2024-05-09T05:11:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.031+0000 7f5a515e3640 1 -- 172.21.15.1:0/2495085554 wait complete. 2024-05-09T05:11:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.033+0000 7f5a515e3640 1 Processor -- start 2024-05-09T05:11:27.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.033+0000 7f5a515e3640 1 -- start start 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a515e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a4c15e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a515e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a4c157770 0x7f5a4c0c9e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a515e3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a4c0ca3d0 0x7f5a4c1ed9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a515e3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c0cc4b0 con 0x7f5a38004800 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a515e3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4c0cc1d0 con 0x7f5a4c0ca3d0 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a515e3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5a4c151070 con 0x7f5a4c157770 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a4affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a4c15e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a4affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a4c15e120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33260/0 (socket says 172.21.15.1:33260) 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.034+0000 7f5a4affd640 1 -- 172.21.15.1:0/812371066 learned_addr learned my addr 172.21.15.1:0/812371066 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:27.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a43fff640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a4c157770 0x7f5a4c0c9e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a4b7fe640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a4c0ca3d0 0x7f5a4c1ed9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a4affd640 1 -- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a4c157770 msgr2=0x7f5a4c0c9e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a4affd640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a4c157770 0x7f5a4c0c9e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a4affd640 1 -- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a4c0ca3d0 msgr2=0x7f5a4c1ed9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a4affd640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a4c0ca3d0 0x7f5a4c1ed9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a4affd640 1 -- 172.21.15.1:0/812371066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a4c0c8810 con 0x7f5a38004800 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a4affd640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a4c15e120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0072c0 tx=0x7f5a3c007780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a437fe640 1 -- 172.21.15.1:0/812371066 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c00f8e0 con 0x7f5a38004800 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a437fe640 1 -- 172.21.15.1:0/812371066 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5a3c018070 con 0x7f5a38004800 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a4c15e660 con 0x7f5a38004800 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a437fe640 1 -- 172.21.15.1:0/812371066 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a3c00fce0 con 0x7f5a38004800 2024-05-09T05:11:27.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.035+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a4c1f25e0 con 0x7f5a38004800 2024-05-09T05:11:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.036+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a4c06d3d0 con 0x7f5a38004800 2024-05-09T05:11:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.037+0000 7f5a437fe640 1 -- 172.21.15.1:0/812371066 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5a3c01d460 con 0x7f5a38004800 2024-05-09T05:11:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.037+0000 7f5a437fe640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5a2c0762d0 0x7f5a2c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.038+0000 7f5a437fe640 1 -- 172.21.15.1:0/812371066 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5a3c097760 con 0x7f5a38004800 2024-05-09T05:11:27.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.038+0000 7f5a43fff640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5a2c0762d0 0x7f5a2c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:27.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.038+0000 7f5a43fff640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5a2c0762d0 0x7f5a2c0787e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a34032000 tx=0x7f5a3402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:27.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.042+0000 7f5a437fe640 1 -- 172.21.15.1:0/812371066 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5a4c0c8810 con 0x7f5a38004800 2024-05-09T05:11:27.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.388+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5a4c06cdd0 con 0x7f5a38004800 2024-05-09T05:11:27.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.389+0000 7f5a437fe640 1 -- 172.21.15.1:0/812371066 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f5a3c060fb0 con 0x7f5a38004800 2024-05-09T05:11:27.390 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":48,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:11:27.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5a2c0762d0 msgr2=0x7f5a2c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:27.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5a2c0762d0 0x7f5a2c0787e0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f5a34032000 tx=0x7f5a3402e000 comp rx=0 tx=0).stop 2024-05-09T05:11:27.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 msgr2=0x7f5a4c15e120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:27.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a4c15e120 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5a3c0072c0 tx=0x7f5a3c007780 comp rx=0 tx=0).stop 2024-05-09T05:11:27.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 shutdown_connections 2024-05-09T05:11:27.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5a2c0762d0 0x7f5a2c0787e0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5a4c0ca3d0 0x7f5a4c1ed9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5a4c157770 0x7f5a4c0c9e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 --2- 172.21.15.1:0/812371066 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5a38004800 0x7f5a4c15e120 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:27.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 >> 172.21.15.1:0/812371066 conn(0x7f5a4c0ce110 msgr2=0x7f5a4c14cc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:27.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 shutdown_connections 2024-05-09T05:11:27.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:27.391+0000 7f5a515e3640 1 -- 172.21.15.1:0/812371066 wait complete. 2024-05-09T05:11:30.405 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:11:30.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.535+0000 7f035c6d9640 1 Processor -- start 2024-05-09T05:11:30.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.535+0000 7f035c6d9640 1 -- start start 2024-05-09T05:11:30.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035c6d9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0354142970 0x7f0354074510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:30.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035c6d9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03540c96a0 0x7f03540ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:30.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035c6d9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 0x7f035415a980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035c6d9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f0354074a50 con 0x7f03540c96a0 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035c6d9640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0354074cc0 con 0x7f03540cede0 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035c6d9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0354159590 con 0x7f0354142970 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f0359c74640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03540c96a0 0x7f03540ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f0359c74640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f03540c96a0 0x7f03540ce8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39440/0 (socket says 172.21.15.1:39440) 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f0359c74640 1 -- 172.21.15.1:0/1257942559 learned_addr learned my addr 172.21.15.1:0/1257942559 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035a475640 1 --2- 172.21.15.1:0/1257942559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0354142970 0x7f0354074510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035ac76640 1 --2- 172.21.15.1:0/1257942559 >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 0x7f035415a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035ac76640 1 -- 172.21.15.1:0/1257942559 >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 msgr2=0x7f035415a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f0359c74640 1 -- 172.21.15.1:0/1257942559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0354142970 msgr2=0x7f0354074510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035ac76640 1 -- 172.21.15.1:0/1257942559 >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 msgr2=0x7f035415a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035ac76640 1 --2- 172.21.15.1:0/1257942559 >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 0x7f035415a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f0359c74640 1 --2- 172.21.15.1:0/1257942559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0354142970 0x7f0354074510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f0359c74640 1 -- 172.21.15.1:0/1257942559 >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 msgr2=0x7f035415a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:30.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035ac76640 1 --2- 172.21.15.1:0/1257942559 >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 0x7f035415a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:30.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f0359c74640 1 --2- 172.21.15.1:0/1257942559 >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 0x7f035415a980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f0359c74640 1 -- 172.21.15.1:0/1257942559 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0354159870 con 0x7f03540c96a0 2024-05-09T05:11:30.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.537+0000 7f035a475640 1 --2- 172.21.15.1:0/1257942559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0354142970 0x7f0354074510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:11:30.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.538+0000 7f0359c74640 1 --2- 172.21.15.1:0/1257942559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540c96a0 0x7f03540ce8a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f03540c55b0 tx=0x7f034402c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5ff88143e828af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:30.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.539+0000 7f0359473640 1 -- 172.21.15.1:0/1257942559 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344032790 con 0x7f03540c96a0 2024-05-09T05:11:30.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.539+0000 7f0359473640 1 -- 172.21.15.1:0/1257942559 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0344038070 con 0x7f03540c96a0 2024-05-09T05:11:30.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.539+0000 7f0359473640 1 -- 172.21.15.1:0/1257942559 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344032b50 con 0x7f03540c96a0 2024-05-09T05:11:30.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 -- 172.21.15.1:0/1257942559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540c96a0 msgr2=0x7f03540ce8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:30.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 --2- 172.21.15.1:0/1257942559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540c96a0 0x7f03540ce8a0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f03540c55b0 tx=0x7f034402c890 comp rx=0 tx=0).stop 2024-05-09T05:11:30.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 -- 172.21.15.1:0/1257942559 shutdown_connections 2024-05-09T05:11:30.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 --2- 172.21.15.1:0/1257942559 >> v2:172.21.15.196:3300/0 conn(0x7f03540cede0 0x7f035415a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 --2- 172.21.15.1:0/1257942559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540c96a0 0x7f03540ce8a0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 --2- 172.21.15.1:0/1257942559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0354142970 0x7f0354074510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 -- 172.21.15.1:0/1257942559 >> 172.21.15.1:0/1257942559 conn(0x7f035406e400 msgr2=0x7f0354071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:30.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 -- 172.21.15.1:0/1257942559 shutdown_connections 2024-05-09T05:11:30.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.540+0000 7f035c6d9640 1 -- 172.21.15.1:0/1257942559 wait complete. 2024-05-09T05:11:30.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.542+0000 7f035c6d9640 1 Processor -- start 2024-05-09T05:11:30.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.542+0000 7f035c6d9640 1 -- start start 2024-05-09T05:11:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035c6d9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 0x7f03540c78e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035c6d9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cede0 0x7f03540caf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035c6d9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f035415b280 0x7f035415b690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035c6d9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f03540c5920 con 0x7f03540cede0 2024-05-09T05:11:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035c6d9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0354159e30 con 0x7f03540c96a0 2024-05-09T05:11:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035c6d9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f03540c5aa0 con 0x7f035415b280 2024-05-09T05:11:30.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f0359c74640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cede0 0x7f03540caf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f0359c74640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cede0 0x7f03540caf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39446/0 (socket says 172.21.15.1:39446) 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f0359c74640 1 -- 172.21.15.1:0/562554311 learned_addr learned my addr 172.21.15.1:0/562554311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035a475640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 0x7f03540c78e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035ac76640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f035415b280 0x7f035415b690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035a475640 1 -- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 msgr2=0x7f03540c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035a475640 1 -- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 msgr2=0x7f03540c78e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035a475640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 0x7f03540c78e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f035a475640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 0x7f03540c78e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f0359c74640 1 -- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f035415b280 msgr2=0x7f035415b690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f0359c74640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f035415b280 0x7f035415b690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f0359c74640 1 -- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 msgr2=0x7f03540c78e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f0359c74640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 0x7f03540c78e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.544+0000 7f0359c74640 1 -- 172.21.15.1:0/562554311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f035415bfd0 con 0x7f03540cede0 2024-05-09T05:11:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.545+0000 7f0359c74640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cede0 0x7f03540caf50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0344000c00 tx=0x7f0344004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.545+0000 7f034affd640 1 -- 172.21.15.1:0/562554311 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0344043070 con 0x7f03540cede0 2024-05-09T05:11:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.545+0000 7f034affd640 1 -- 172.21.15.1:0/562554311 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f034403c620 con 0x7f03540cede0 2024-05-09T05:11:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.545+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03541e5610 con 0x7f03540cede0 2024-05-09T05:11:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.545+0000 7f034affd640 1 -- 172.21.15.1:0/562554311 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f034403ca10 con 0x7f03540cede0 2024-05-09T05:11:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.545+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03540c70e0 con 0x7f03540cede0 2024-05-09T05:11:30.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.546+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0354050870 con 0x7f03540cede0 2024-05-09T05:11:30.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.547+0000 7f034affd640 1 -- 172.21.15.1:0/562554311 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0344046070 con 0x7f03540cede0 2024-05-09T05:11:30.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.547+0000 7f034affd640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f03380762d0 0x7f03380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:30.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.548+0000 7f034affd640 1 -- 172.21.15.1:0/562554311 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f03440b8180 con 0x7f03540cede0 2024-05-09T05:11:30.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.548+0000 7f035a475640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f03380762d0 0x7f03380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:30.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.548+0000 7f035a475640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f03380762d0 0x7f03380787e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0350006fd0 tx=0x7f0350008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:30.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.553+0000 7f034affd640 1 -- 172.21.15.1:0/562554311 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0344081610 con 0x7f03540cede0 2024-05-09T05:11:30.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.865+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f03540c9480 con 0x7f03540cede0 2024-05-09T05:11:30.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.865+0000 7f034affd640 1 -- 172.21.15.1:0/562554311 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f0344085910 con 0x7f03540cede0 2024-05-09T05:11:30.866 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:11:30.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f03380762d0 msgr2=0x7f03380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:30.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f03380762d0 0x7f03380787e0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0350006fd0 tx=0x7f0350008040 comp rx=0 tx=0).stop 2024-05-09T05:11:30.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cede0 msgr2=0x7f03540caf50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:30.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cede0 0x7f03540caf50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f0344000c00 tx=0x7f0344004730 comp rx=0 tx=0).stop 2024-05-09T05:11:30.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 shutdown_connections 2024-05-09T05:11:30.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f03380762d0 0x7f03380787e0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f035415b280 0x7f035415b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f03540cede0 0x7f03540caf50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 --2- 172.21.15.1:0/562554311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f03540c96a0 0x7f03540c78e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:30.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 >> 172.21.15.1:0/562554311 conn(0x7f035406e400 msgr2=0x7f035413f8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:30.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 shutdown_connections 2024-05-09T05:11:30.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:30.867+0000 7f035c6d9640 1 -- 172.21.15.1:0/562554311 wait complete. 2024-05-09T05:11:30.882 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:11:30.882 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:11:30.882 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:11:31.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.011+0000 7f7a38430640 1 Processor -- start 2024-05-09T05:11:31.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.011+0000 7f7a38430640 1 -- start start 2024-05-09T05:11:31.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a38430640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 0x7f7a30154530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a38430640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a30070f00 0x7f7a30071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a38430640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a300cbf90 con 0x7f7a300cae10 2024-05-09T05:11:31.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a38430640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f7a300ca3c0 con 0x7f7a30072400 2024-05-09T05:11:31.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a38430640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7a3015a780 con 0x7f7a30070f00 2024-05-09T05:11:31.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a369cd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7a300cae10 0x7f7a3015db60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57014/0 (socket says 172.21.15.1:57014) 2024-05-09T05:11:31.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a369cd640 1 -- 172.21.15.1:0/4200995076 learned_addr learned my addr 172.21.15.1:0/4200995076 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4200995076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a30070f00 0x7f7a30071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a361cc640 1 --2- 172.21.15.1:0/4200995076 >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 0x7f7a30154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a361cc640 1 -- 172.21.15.1:0/4200995076 >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 msgr2=0x7f7a30154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a361cc640 1 -- 172.21.15.1:0/4200995076 >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 msgr2=0x7f7a30154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a361cc640 1 --2- 172.21.15.1:0/4200995076 >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 0x7f7a30154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a351ca640 1 -- 172.21.15.1:0/4200995076 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1330405633 0 0) 0x7f7a300cbf90 con 0x7f7a300cae10 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a361cc640 1 --2- 172.21.15.1:0/4200995076 >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 0x7f7a30154530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a351ca640 1 -- 172.21.15.1:0/4200995076 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a1c003520 con 0x7f7a300cae10 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a359cb640 1 -- 172.21.15.1:0/4200995076 >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 msgr2=0x7f7a30154530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4200995076 >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 0x7f7a30154530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a359cb640 1 -- 172.21.15.1:0/4200995076 >> v1:172.21.15.1:6789/0 conn(0x7f7a300cae10 legacy=0x7f7a3015db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.013+0000 7f7a359cb640 1 -- 172.21.15.1:0/4200995076 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a300ca610 con 0x7f7a30070f00 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.014+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4200995076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a30070f00 0x7f7a30071310 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7a200097a0 tx=0x7f7a2002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ca8e5e8b6790f67 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.015+0000 7f7a351ca640 1 -- 172.21.15.1:0/4200995076 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20037070 con 0x7f7a30070f00 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.015+0000 7f7a351ca640 1 -- 172.21.15.1:0/4200995076 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a20032440 con 0x7f7a30070f00 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.015+0000 7f7a351ca640 1 -- 172.21.15.1:0/4200995076 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20032810 con 0x7f7a30070f00 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.015+0000 7f7a38430640 1 -- 172.21.15.1:0/4200995076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a30070f00 msgr2=0x7f7a30071310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.015+0000 7f7a38430640 1 --2- 172.21.15.1:0/4200995076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a30070f00 0x7f7a30071310 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7a200097a0 tx=0x7f7a2002a180 comp rx=0 tx=0).stop 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.015+0000 7f7a38430640 1 -- 172.21.15.1:0/4200995076 shutdown_connections 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.016+0000 7f7a38430640 1 --2- 172.21.15.1:0/4200995076 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a30070f00 0x7f7a30071310 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.016+0000 7f7a38430640 1 --2- 172.21.15.1:0/4200995076 >> v2:172.21.15.196:3300/0 conn(0x7f7a30072400 0x7f7a30154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.016+0000 7f7a38430640 1 -- 172.21.15.1:0/4200995076 >> 172.21.15.1:0/4200995076 conn(0x7f7a300d0470 msgr2=0x7f7a300cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.016+0000 7f7a38430640 1 -- 172.21.15.1:0/4200995076 shutdown_connections 2024-05-09T05:11:31.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.016+0000 7f7a38430640 1 -- 172.21.15.1:0/4200995076 wait complete. 2024-05-09T05:11:31.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.017+0000 7f7a38430640 1 Processor -- start 2024-05-09T05:11:31.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.017+0000 7f7a38430640 1 -- start start 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a38430640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30070f00 0x7f7a301f3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a38430640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a30072400 0x7f7a301f63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a38430640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a300cae10 0x7f7a300b0840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a38430640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300b27b0 con 0x7f7a30070f00 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a38430640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300b2630 con 0x7f7a30072400 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a38430640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7a300cb910 con 0x7f7a300cae10 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a361cc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30070f00 0x7f7a301f3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a361cc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30070f00 0x7f7a301f3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39450/0 (socket says 172.21.15.1:39450) 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a361cc640 1 -- 172.21.15.1:0/4119532520 learned_addr learned my addr 172.21.15.1:0/4119532520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:31.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a30072400 0x7f7a301f63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a369cd640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a300cae10 0x7f7a300b0840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a359cb640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a30072400 msgr2=0x7f7a301f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a359cb640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a30072400 msgr2=0x7f7a301f63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a30072400 0x7f7a301f63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a30072400 0x7f7a301f63d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a361cc640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a300cae10 msgr2=0x7f7a300b0840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a361cc640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a300cae10 0x7f7a300b0840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a361cc640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a30072400 msgr2=0x7f7a301f63d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a361cc640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a30072400 0x7f7a301f63d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.019+0000 7f7a361cc640 1 -- 172.21.15.1:0/4119532520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a301e53c0 con 0x7f7a30070f00 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.020+0000 7f7a361cc640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30070f00 0x7f7a301f3040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00d7d0 tx=0x7f7a2c00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.020+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c00b660 con 0x7f7a30070f00 2024-05-09T05:11:31.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.020+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a2c019070 con 0x7f7a30070f00 2024-05-09T05:11:31.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.020+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a300cf080 con 0x7f7a30070f00 2024-05-09T05:11:31.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.020+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2c00ba60 con 0x7f7a30070f00 2024-05-09T05:11:31.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.020+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a301f2840 con 0x7f7a30070f00 2024-05-09T05:11:31.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.021+0000 7f7a38430640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30150eb0 0x7f7a30152a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.021+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7a300cee30 con 0x7f7a30150eb0 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.021+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30150eb0 0x7f7a30152a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.021+0000 7f7a359cb640 1 -- 172.21.15.1:0/4119532520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7a300cbf90 con 0x7f7a30150eb0 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.022+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30150eb0 0x7f7a30152a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7a300cd130 tx=0x7f7a200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.022+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20041020 con 0x7f7a30150eb0 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.022+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7a2c00bc00 con 0x7f7a30070f00 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a26ffd640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7a14076200 0x7f7a14078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7a2c097b10 con 0x7f7a30070f00 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a369cd640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7a14076200 0x7f7a14078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7a2003d620 con 0x7f7a30150eb0 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30150eb0 msgr2=0x7f7a30152a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a26ffd640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30150eb0 0x7f7a30152a40 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7a300cd130 tx=0x7f7a200046c0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30150eb0 msgr2=0x7f7a30152a40 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:31.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a26ffd640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30150eb0 0x7f7a30152a40 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.023+0000 7f7a369cd640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7a14076200 0x7f7a14078710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7a28005c00 tx=0x7f7a28009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.032+0000 7f7a38430640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a301e5590 0x7f7a30152f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.032+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7a30050870 con 0x7f7a301e5590 2024-05-09T05:11:31.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.033+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a301e5590 0x7f7a30152f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.033+0000 7f7a359cb640 1 -- 172.21.15.1:0/4119532520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7a300cbcd0 con 0x7f7a301e5590 2024-05-09T05:11:31.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.033+0000 7f7a359cb640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a301e5590 0x7f7a30152f80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7a3006d4d0 tx=0x7f7a20008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.033+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20037040 con 0x7f7a301e5590 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4892 (secure 0 0 0) 0x7f7a2003dcd0 con 0x7f7a301e5590 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a301e5590 msgr2=0x7f7a30152f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a26ffd640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a301e5590 0x7f7a30152f80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7a3006d4d0 tx=0x7f7a20008030 comp rx=0 tx=0).stop 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a26ffd640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a301e5590 msgr2=0x7f7a30152f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a26ffd640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a301e5590 0x7f7a30152f80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:11:31.035 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 50, 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 79628, 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.036 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.037 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:31.038 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.039 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:31.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.041 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.043 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.045 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.046 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:31.047 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7a14076200 msgr2=0x7f7a14078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a38430640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7a14076200 0x7f7a14078710 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7a28005c00 tx=0x7f7a28009040 comp rx=0 tx=0).stop 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30070f00 msgr2=0x7f7a301f3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a38430640 1 --2- 172.21.15.1:0/4119532520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a30070f00 0x7f7a301f3040 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f7a2c00d7d0 tx=0x7f7a2c00dc90 comp rx=0 tx=0).stop 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a369cd640 1 -- 172.21.15.1:0/4119532520 reap_dead start 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a369cd640 1 -- 172.21.15.1:0/4119532520 reap_dead start 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 shutdown_connections 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 >> 172.21.15.1:0/4119532520 conn(0x7f7a300d0470 msgr2=0x7f7a300b3ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 shutdown_connections 2024-05-09T05:11:31.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.034+0000 7f7a38430640 1 -- 172.21.15.1:0/4119532520 wait complete. 2024-05-09T05:11:31.049 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:11:31.050 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:11:31.050 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:11:31.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.171+0000 7f9960b00640 1 Processor -- start 2024-05-09T05:11:31.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.171+0000 7f9960b00640 1 -- start start 2024-05-09T05:11:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f9960b00640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 0x7f995c153730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f9960b00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c0cae10 0x7f995c15c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f9960b00640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f995c0c8a50 con 0x7f995c15ceb0 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f9960b00640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f995c0c9d00 con 0x7f995c0ca970 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f9960b00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f995c0c9870 con 0x7f995c0cae10 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f9959d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c0cae10 0x7f995c15c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f995a59c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 0x7f995c153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f9959d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c0cae10 0x7f995c15c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:53048/0 (socket says 172.21.15.1:53048) 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f9959d9b640 1 -- 172.21.15.1:0/1966254701 learned_addr learned my addr 172.21.15.1:0/1966254701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f995ad9d640 1 --1- 172.21.15.1:0/1966254701 >> v1:172.21.15.196:6789/0 conn(0x7f995c15ceb0 0x7f995c0c67b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:43580/0 (socket says 172.21.15.1:43580) 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f995a59c640 1 -- 172.21.15.1:0/1966254701 >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 msgr2=0x7f995c153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f995a59c640 1 -- 172.21.15.1:0/1966254701 >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 msgr2=0x7f995c153730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.173+0000 7f995a59c640 1 --2- 172.21.15.1:0/1966254701 >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 0x7f995c153730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.174+0000 7f995a59c640 1 --2- 172.21.15.1:0/1966254701 >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 0x7f995c153730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.174+0000 7f9959d9b640 1 -- 172.21.15.1:0/1966254701 >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 msgr2=0x7f995c153730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:31.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.174+0000 7f9959d9b640 1 --2- 172.21.15.1:0/1966254701 >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 0x7f995c153730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.174+0000 7f9959d9b640 1 -- 172.21.15.1:0/1966254701 >> v1:172.21.15.196:6789/0 conn(0x7f995c15ceb0 legacy=0x7f995c0c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.174+0000 7f9959d9b640 1 -- 172.21.15.1:0/1966254701 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995c0c8d10 con 0x7f995c0cae10 2024-05-09T05:11:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.174+0000 7f9959d9b640 1 --2- 172.21.15.1:0/1966254701 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c0cae10 0x7f995c15c970 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9950009870 tx=0x7f995002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=432c3cf98dfe981f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.175+0000 7f995959a640 1 -- 172.21.15.1:0/1966254701 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950037070 con 0x7f995c0cae10 2024-05-09T05:11:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.175+0000 7f995959a640 1 -- 172.21.15.1:0/1966254701 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9950032440 con 0x7f995c0cae10 2024-05-09T05:11:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.175+0000 7f995959a640 1 -- 172.21.15.1:0/1966254701 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950032810 con 0x7f995c0cae10 2024-05-09T05:11:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.175+0000 7f9960b00640 1 -- 172.21.15.1:0/1966254701 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c0cae10 msgr2=0x7f995c15c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.175+0000 7f9960b00640 1 --2- 172.21.15.1:0/1966254701 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c0cae10 0x7f995c15c970 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f9950009870 tx=0x7f995002a270 comp rx=0 tx=0).stop 2024-05-09T05:11:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.176+0000 7f9960b00640 1 -- 172.21.15.1:0/1966254701 shutdown_connections 2024-05-09T05:11:31.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.176+0000 7f9960b00640 1 --2- 172.21.15.1:0/1966254701 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c0cae10 0x7f995c15c970 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.176+0000 7f9960b00640 1 --2- 172.21.15.1:0/1966254701 >> v2:172.21.15.196:3300/0 conn(0x7f995c0ca970 0x7f995c153730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.176+0000 7f9960b00640 1 -- 172.21.15.1:0/1966254701 >> 172.21.15.1:0/1966254701 conn(0x7f995c0ce910 msgr2=0x7f995c0ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:31.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.176+0000 7f9960b00640 1 -- 172.21.15.1:0/1966254701 shutdown_connections 2024-05-09T05:11:31.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.176+0000 7f9960b00640 1 -- 172.21.15.1:0/1966254701 wait complete. 2024-05-09T05:11:31.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.178+0000 7f9960b00640 1 Processor -- start 2024-05-09T05:11:31.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.178+0000 7f9960b00640 1 -- start start 2024-05-09T05:11:31.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f9960b00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f995c0ca970 0x7f995c15a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f9960b00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c0cae10 0x7f995c15c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f9960b00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c15ceb0 0x7f995c13c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f9960b00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f995c0c63d0 con 0x7f995c0ca970 2024-05-09T05:11:31.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f9960b00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f995c0c60f0 con 0x7f995c0cae10 2024-05-09T05:11:31.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f9960b00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f995c0c9500 con 0x7f995c15ceb0 2024-05-09T05:11:31.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f995a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f995c0ca970 0x7f995c15a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f995a59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f995c0ca970 0x7f995c15a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39478/0 (socket says 172.21.15.1:39478) 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f9959d9b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c0cae10 0x7f995c15c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f995ad9d640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c15ceb0 0x7f995c13c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.180+0000 7f995a59c640 1 -- 172.21.15.1:0/3387368708 learned_addr learned my addr 172.21.15.1:0/3387368708 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f995a59c640 1 -- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c15ceb0 msgr2=0x7f995c13c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f995a59c640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c15ceb0 0x7f995c13c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f995a59c640 1 -- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c0cae10 msgr2=0x7f995c15c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f995a59c640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c0cae10 0x7f995c15c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f995a59c640 1 -- 172.21.15.1:0/3387368708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f995c14f7b0 con 0x7f995c0ca970 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f9959d9b640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c0cae10 0x7f995c15c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f995ad9d640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f995c15ceb0 0x7f995c13c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f995a59c640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f995c0ca970 0x7f995c15a540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f994c00d800 tx=0x7f994c00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994c00b6c0 con 0x7f995c0ca970 2024-05-09T05:11:31.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f994c019070 con 0x7f995c0ca970 2024-05-09T05:11:31.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f994c00bac0 con 0x7f995c0ca970 2024-05-09T05:11:31.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f995c14fa90 con 0x7f995c0ca970 2024-05-09T05:11:31.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.181+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f995c0cdb20 con 0x7f995c0ca970 2024-05-09T05:11:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.182+0000 7f9960b00640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c13b9a0 0x7f995c0cc710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.182+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f995c15b650 con 0x7f995c13b9a0 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.183+0000 7f9959d9b640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c13b9a0 0x7f995c0cc710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.183+0000 7f9959d9b640 1 -- 172.21.15.1:0/3387368708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f995c1573a0 con 0x7f995c13b9a0 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.183+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f994c00bc60 con 0x7f995c0ca970 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.183+0000 7f9959d9b640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c13b9a0 0x7f995c0cc710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f995c073e90 tx=0x7f99500046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f994affd640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f993c076300 0x7f993c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f994c0979e0 con 0x7f995c0ca970 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f995ad9d640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f993c076300 0x7f993c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950037040 con 0x7f995c13b9a0 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9950032440 con 0x7f995c13b9a0 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c13b9a0 msgr2=0x7f995c0cc710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f994affd640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c13b9a0 0x7f995c0cc710 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f995c073e90 tx=0x7f99500046c0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c13b9a0 msgr2=0x7f995c0cc710 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:31.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f994affd640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c13b9a0 0x7f995c0cc710 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.184+0000 7f995ad9d640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f993c076300 0x7f993c078810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f99540012e0 tx=0x7f9954001600 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.191+0000 7f9960b00640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c15aff0 0x7f995c0ccc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.192+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f995c15c0b0 con 0x7f995c15aff0 2024-05-09T05:11:31.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.192+0000 7f9959d9b640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c15aff0 0x7f995c0ccc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.192+0000 7f9959d9b640 1 -- 172.21.15.1:0/3387368708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f995c0cdde0 con 0x7f995c15aff0 2024-05-09T05:11:31.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.192+0000 7f9959d9b640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c15aff0 0x7f995c0ccc50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9950000c00 tx=0x7f9950008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9950037030 con 0x7f995c15aff0 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f995003c430 con 0x7f995c15aff0 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c15aff0 msgr2=0x7f995c0ccc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f994affd640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c15aff0 0x7f995c0ccc50 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f9950000c00 tx=0x7f9950008030 comp rx=0 tx=0).stop 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f994affd640 1 -- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c15aff0 msgr2=0x7f995c0ccc50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f994affd640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f995c15aff0 0x7f995c0ccc50 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:31.194 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 50, 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 4186, 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.195 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.196 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:31.197 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:31.198 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.199 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.200 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:31.201 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.204 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:31.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.206 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.206 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.206 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:31.206 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.206 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.206 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:31.206 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:31.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f993c076300 msgr2=0x7f993c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f993c076300 0x7f993c078810 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f99540012e0 tx=0x7f9954001600 comp rx=0 tx=0).stop 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f995c0ca970 msgr2=0x7f995c15a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f995c0ca970 0x7f995c15a540 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f994c00d800 tx=0x7f994c00dcc0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f995ad9d640 1 -- 172.21.15.1:0/3387368708 reap_dead start 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 shutdown_connections 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 --2- 172.21.15.1:0/3387368708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f995c0ca970 0x7f995c15a540 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 >> 172.21.15.1:0/3387368708 conn(0x7f995c0ce910 msgr2=0x7f995c06c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 shutdown_connections 2024-05-09T05:11:31.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.193+0000 7f9960b00640 1 -- 172.21.15.1:0/3387368708 wait complete. 2024-05-09T05:11:31.209 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:11:31.209 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:11:31.209 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:11:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.332+0000 7fe9eb042640 1 Processor -- start 2024-05-09T05:11:31.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.332+0000 7fe9eb042640 1 -- start start 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9eb042640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe9e4072400 0x7fe9e4154530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9eb042640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9e4071310 0x7fe9e4164ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9eb042640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe9e40cbf90 con 0x7fe9e4070f00 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9eb042640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe9e40ca3c0 con 0x7fe9e4071310 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9eb042640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe9e415a780 con 0x7fe9e4072400 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e95df640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe9e4071310 0x7fe9e4164ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe9e4070f00 0x7fe9e40cae10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57016/0 (socket says 172.21.15.1:57016) 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e3fff640 1 -- 172.21.15.1:0/2712854156 learned_addr learned my addr 172.21.15.1:0/2712854156 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:31.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e8dde640 1 --2- 172.21.15.1:0/2712854156 >> v2:172.21.15.196:3300/0 conn(0x7fe9e4072400 0x7fe9e4154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e95df640 1 -- 172.21.15.1:0/2712854156 >> v2:172.21.15.196:3300/0 conn(0x7fe9e4072400 msgr2=0x7fe9e4154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:31.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e95df640 1 --2- 172.21.15.1:0/2712854156 >> v2:172.21.15.196:3300/0 conn(0x7fe9e4072400 0x7fe9e4154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e95df640 1 -- 172.21.15.1:0/2712854156 >> v1:172.21.15.1:6789/0 conn(0x7fe9e4070f00 legacy=0x7fe9e40cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e95df640 1 -- 172.21.15.1:0/2712854156 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e40ca610 con 0x7fe9e4071310 2024-05-09T05:11:31.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.334+0000 7fe9e37fe640 1 -- 172.21.15.1:0/2712854156 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 811399400 0 0) 0x7fe9e40cbf90 con 0x7fe9e4070f00 2024-05-09T05:11:31.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.335+0000 7fe9e95df640 1 --2- 172.21.15.1:0/2712854156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4071310 0x7fe9e4164ef0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40cd070 tx=0x7fe9dc02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d35cc78492ede318 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.335+0000 7fe9e37fe640 1 -- 172.21.15.1:0/2712854156 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc007740 con 0x7fe9e4071310 2024-05-09T05:11:31.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.335+0000 7fe9e37fe640 1 -- 172.21.15.1:0/2712854156 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9dc0078e0 con 0x7fe9e4071310 2024-05-09T05:11:31.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.335+0000 7fe9e37fe640 1 -- 172.21.15.1:0/2712854156 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc007cb0 con 0x7fe9e4071310 2024-05-09T05:11:31.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.336+0000 7fe9eb042640 1 -- 172.21.15.1:0/2712854156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4071310 msgr2=0x7fe9e4164ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.336+0000 7fe9eb042640 1 --2- 172.21.15.1:0/2712854156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4071310 0x7fe9e4164ef0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fe9e40cd070 tx=0x7fe9dc02a680 comp rx=0 tx=0).stop 2024-05-09T05:11:31.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.336+0000 7fe9eb042640 1 -- 172.21.15.1:0/2712854156 shutdown_connections 2024-05-09T05:11:31.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.336+0000 7fe9eb042640 1 --2- 172.21.15.1:0/2712854156 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4071310 0x7fe9e4164ef0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.336+0000 7fe9eb042640 1 --2- 172.21.15.1:0/2712854156 >> v2:172.21.15.196:3300/0 conn(0x7fe9e4072400 0x7fe9e4154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.336+0000 7fe9eb042640 1 -- 172.21.15.1:0/2712854156 >> 172.21.15.1:0/2712854156 conn(0x7fe9e40d0470 msgr2=0x7fe9e40cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:31.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.336+0000 7fe9eb042640 1 -- 172.21.15.1:0/2712854156 shutdown_connections 2024-05-09T05:11:31.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.336+0000 7fe9eb042640 1 -- 172.21.15.1:0/2712854156 wait complete. 2024-05-09T05:11:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.339+0000 7fe9eb042640 1 Processor -- start 2024-05-09T05:11:31.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.339+0000 7fe9eb042640 1 -- start start 2024-05-09T05:11:31.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9eb042640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4070f00 0x7fe9e406ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9eb042640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e4071310 0x7fe9e40b19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9eb042640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4072400 0x7fe9e41571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9eb042640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e40b2630 con 0x7fe9e4072400 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9eb042640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe9e4071720 con 0x7fe9e4070f00 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9eb042640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e40b27b0 con 0x7fe9e4071310 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4072400 0x7fe9e41571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4072400 0x7fe9e41571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39496/0 (socket says 172.21.15.1:39496) 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 -- 172.21.15.1:0/2889866539 learned_addr learned my addr 172.21.15.1:0/2889866539 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e8dde640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4070f00 0x7fe9e406ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e3fff640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e4071310 0x7fe9e40b19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 -- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e4071310 msgr2=0x7fe9e40b19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e4071310 0x7fe9e40b19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 -- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4070f00 msgr2=0x7fe9e406ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4070f00 0x7fe9e406ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 -- 172.21.15.1:0/2889866539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe9e406f810 con 0x7fe9e4072400 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e3fff640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e4071310 0x7fe9e40b19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e8dde640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe9e4070f00 0x7fe9e406ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.341+0000 7fe9e95df640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4072400 0x7fe9e41571e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009e50 tx=0x7fe9dc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.342+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc03f070 con 0x7fe9e4072400 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.342+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe9dc031bf0 con 0x7fe9e4072400 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.342+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe9e41f3780 con 0x7fe9e4072400 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.342+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9dc009450 con 0x7fe9e4072400 2024-05-09T05:11:31.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.342+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe9e41f2ca0 con 0x7fe9e4072400 2024-05-09T05:11:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.343+0000 7fe9eb042640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e41f2ee0 0x7fe9e406d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.343+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e4050870 con 0x7fe9e41f2ee0 2024-05-09T05:11:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.343+0000 7fe9e8dde640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e41f2ee0 0x7fe9e406d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.343+0000 7fe9e8dde640 1 -- 172.21.15.1:0/2889866539 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe9e40cbcd0 con 0x7fe9e41f2ee0 2024-05-09T05:11:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.344+0000 7fe9e8dde640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e41f2ee0 0x7fe9e406d800 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8011600 tx=0x7fe9d8011ac0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.344+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe9dc03b410 con 0x7fe9e4072400 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.344+0000 7fe9e17fa640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe9c8076300 0x7fe9c8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.344+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe9dc0b8e30 con 0x7fe9e4072400 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.345+0000 7fe9e3fff640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe9c8076300 0x7fe9c8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.345+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d800f940 con 0x7fe9e41f2ee0 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.345+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe9d800fae0 con 0x7fe9e41f2ee0 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.345+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e41f2ee0 msgr2=0x7fe9e406d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.345+0000 7fe9e17fa640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e41f2ee0 0x7fe9e406d800 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8011600 tx=0x7fe9d8011ac0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.345+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e41f2ee0 msgr2=0x7fe9e406d800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.345+0000 7fe9e17fa640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e41f2ee0 0x7fe9e406d800 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.345+0000 7fe9e3fff640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe9c8076300 0x7fe9c8078810 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4071b70 tx=0x7fe9d0007cc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.353+0000 7fe9eb042640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e406dd40 0x7fe9e4159280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.353+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe9e4071ba0 con 0x7fe9e406dd40 2024-05-09T05:11:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.354+0000 7fe9e8dde640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e406dd40 0x7fe9e4159280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.354+0000 7fe9e8dde640 1 -- 172.21.15.1:0/2889866539 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe9e4154810 con 0x7fe9e406dd40 2024-05-09T05:11:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.354+0000 7fe9e8dde640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e406dd40 0x7fe9e4159280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8008010 tx=0x7fe9d80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.354+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe9d8003610 con 0x7fe9e406dd40 2024-05-09T05:11:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5061 (secure 0 0 0) 0x7fe9d8006040 con 0x7fe9e406dd40 2024-05-09T05:11:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e406dd40 msgr2=0x7fe9e4159280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9e17fa640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e406dd40 0x7fe9e4159280 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fe9d8008010 tx=0x7fe9d80046c0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9e17fa640 1 -- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e406dd40 msgr2=0x7fe9e4159280 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9e17fa640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe9e406dd40 0x7fe9e4159280 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 50, 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 249352, 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:31.356 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.357 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.358 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:31.359 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:31.360 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.361 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:31.363 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.364 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:31.365 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:31.366 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:11:31.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:31.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:31.369 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:31.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe9c8076300 msgr2=0x7fe9c8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9eb042640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe9c8076300 0x7fe9c8078810 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe9e4071b70 tx=0x7fe9d0007cc0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4072400 msgr2=0x7fe9e41571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9eb042640 1 --2- 172.21.15.1:0/2889866539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe9e4072400 0x7fe9e41571e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe9dc009e50 tx=0x7fe9dc004a40 comp rx=0 tx=0).stop 2024-05-09T05:11:31.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9e95df640 1 -- 172.21.15.1:0/2889866539 reap_dead start 2024-05-09T05:11:31.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9e95df640 1 -- 172.21.15.1:0/2889866539 reap_dead start 2024-05-09T05:11:31.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 shutdown_connections 2024-05-09T05:11:31.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 >> 172.21.15.1:0/2889866539 conn(0x7fe9e40d0470 msgr2=0x7fe9e40cad10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:31.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 shutdown_connections 2024-05-09T05:11:31.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.355+0000 7fe9eb042640 1 -- 172.21.15.1:0/2889866539 wait complete. 2024-05-09T05:11:31.372 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:11:31.372 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:11:31.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:11:31.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.494+0000 7fb653dea640 1 Processor -- start 2024-05-09T05:11:31.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.495+0000 7fb653dea640 1 -- start start 2024-05-09T05:11:31.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb653dea640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c072120 0x7fb64c0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb653dea640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64c15b110 con 0x7fb64c0c62e0 2024-05-09T05:11:31.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb653dea640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb64c15b3b0 con 0x7fb64c0ca340 2024-05-09T05:11:31.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb653dea640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb64c0c53f0 con 0x7fb64c072120 2024-05-09T05:11:31.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb652387640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb64c0c62e0 0x7fb64c070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57028/0 (socket says 172.21.15.1:57028) 2024-05-09T05:11:31.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb652387640 1 -- 172.21.15.1:0/1203762554 learned_addr learned my addr 172.21.15.1:0/1203762554 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:31.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb651385640 1 --2- 172.21.15.1:0/1203762554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c072120 0x7fb64c0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb650b84640 1 -- 172.21.15.1:0/1203762554 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1316772486 0 0) 0x7fb64c15b110 con 0x7fb64c0c62e0 2024-05-09T05:11:31.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb650b84640 1 -- 172.21.15.1:0/1203762554 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb638003520 con 0x7fb64c0c62e0 2024-05-09T05:11:31.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb650b84640 1 -- 172.21.15.1:0/1203762554 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 165192270 0 0) 0x7fb64c15b3b0 con 0x7fb64c0ca340 2024-05-09T05:11:31.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb650b84640 1 -- 172.21.15.1:0/1203762554 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb64c15b110 con 0x7fb64c0ca340 2024-05-09T05:11:31.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb651385640 1 -- 172.21.15.1:0/1203762554 >> v1:172.21.15.196:6789/0 conn(0x7fb64c0ca340 legacy=0x7fb64c1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb651385640 1 -- 172.21.15.1:0/1203762554 >> v1:172.21.15.1:6789/0 conn(0x7fb64c0c62e0 legacy=0x7fb64c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.497+0000 7fb651385640 1 -- 172.21.15.1:0/1203762554 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64c0c5800 con 0x7fb64c072120 2024-05-09T05:11:31.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.498+0000 7fb651385640 1 --2- 172.21.15.1:0/1203762554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c072120 0x7fb64c0c4eb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0097d0 tx=0x7fb63c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b351ff2dcb4b126d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.498+0000 7fb650b84640 1 -- 172.21.15.1:0/1203762554 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c037070 con 0x7fb64c072120 2024-05-09T05:11:31.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.498+0000 7fb650b84640 1 -- 172.21.15.1:0/1203762554 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb63c032440 con 0x7fb64c072120 2024-05-09T05:11:31.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.499+0000 7fb650b84640 1 -- 172.21.15.1:0/1203762554 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c032810 con 0x7fb64c072120 2024-05-09T05:11:31.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.499+0000 7fb653dea640 1 -- 172.21.15.1:0/1203762554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c072120 msgr2=0x7fb64c0c4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.499+0000 7fb653dea640 1 --2- 172.21.15.1:0/1203762554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c072120 0x7fb64c0c4eb0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb63c0097d0 tx=0x7fb63c02a180 comp rx=0 tx=0).stop 2024-05-09T05:11:31.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.499+0000 7fb653dea640 1 -- 172.21.15.1:0/1203762554 shutdown_connections 2024-05-09T05:11:31.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.499+0000 7fb653dea640 1 --2- 172.21.15.1:0/1203762554 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c072120 0x7fb64c0c4eb0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.499+0000 7fb653dea640 1 -- 172.21.15.1:0/1203762554 >> 172.21.15.1:0/1203762554 conn(0x7fb64c0731d0 msgr2=0x7fb64c0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:31.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.499+0000 7fb653dea640 1 -- 172.21.15.1:0/1203762554 shutdown_connections 2024-05-09T05:11:31.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.499+0000 7fb653dea640 1 -- 172.21.15.1:0/1203762554 wait complete. 2024-05-09T05:11:31.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.501+0000 7fb653dea640 1 Processor -- start 2024-05-09T05:11:31.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.501+0000 7fb653dea640 1 -- start start 2024-05-09T05:11:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.502+0000 7fb653dea640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb64c072120 0x7fb64c0afff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.502+0000 7fb653dea640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb64c0c62e0 0x7fb64c0cda80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.502+0000 7fb653dea640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c0ca340 0x7fb64c1f2dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.502+0000 7fb653dea640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c0c5dc0 con 0x7fb64c072120 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.502+0000 7fb653dea640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb64c0c55b0 con 0x7fb64c0c62e0 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.502+0000 7fb653dea640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb64c0c66f0 con 0x7fb64c0ca340 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.502+0000 7fb651b86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb64c072120 0x7fb64c0afff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651b86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb64c072120 0x7fb64c0afff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39502/0 (socket says 172.21.15.1:39502) 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651b86640 1 -- 172.21.15.1:0/3615201703 learned_addr learned my addr 172.21.15.1:0/3615201703 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651385640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb64c0c62e0 0x7fb64c0cda80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb652387640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c0ca340 0x7fb64c1f2dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651b86640 1 -- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c0ca340 msgr2=0x7fb64c1f2dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651b86640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c0ca340 0x7fb64c1f2dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651b86640 1 -- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb64c0c62e0 msgr2=0x7fb64c0cda80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651b86640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb64c0c62e0 0x7fb64c0cda80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651b86640 1 -- 172.21.15.1:0/3615201703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb64c1f3300 con 0x7fb64c072120 2024-05-09T05:11:31.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651385640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb64c0c62e0 0x7fb64c0cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:11:31.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb651b86640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb64c072120 0x7fb64c0afff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb64800c430 tx=0x7fb64800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb6427fc640 1 -- 172.21.15.1:0/3615201703 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648017070 con 0x7fb64c072120 2024-05-09T05:11:31.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb6427fc640 1 -- 172.21.15.1:0/3615201703 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6480052f0 con 0x7fb64c072120 2024-05-09T05:11:31.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb64c0ce700 con 0x7fb64c072120 2024-05-09T05:11:31.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb6427fc640 1 -- 172.21.15.1:0/3615201703 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb648003830 con 0x7fb64c072120 2024-05-09T05:11:31.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.503+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb64c0cfa60 con 0x7fb64c072120 2024-05-09T05:11:31.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.505+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb64c0cec20 con 0x7fb64c072120 2024-05-09T05:11:31.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.506+0000 7fb6427fc640 1 -- 172.21.15.1:0/3615201703 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb64801a070 con 0x7fb64c072120 2024-05-09T05:11:31.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.506+0000 7fb6427fc640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6300762d0 0x7fb6300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:31.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.506+0000 7fb6427fc640 1 -- 172.21.15.1:0/3615201703 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb648096930 con 0x7fb64c072120 2024-05-09T05:11:31.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.506+0000 7fb651385640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6300762d0 0x7fb6300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:31.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.507+0000 7fb651385640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6300762d0 0x7fb6300787e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb63c034000 tx=0x7fb63c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:31.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.511+0000 7fb6427fc640 1 -- 172.21.15.1:0/3615201703 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb64c0cec20 con 0x7fb64c072120 2024-05-09T05:11:31.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.831+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb64c050870 con 0x7fb64c072120 2024-05-09T05:11:31.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.833+0000 7fb6427fc640 1 -- 172.21.15.1:0/3615201703 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb648068020 con 0x7fb64c072120 2024-05-09T05:11:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6300762d0 msgr2=0x7fb6300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6300762d0 0x7fb6300787e0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb63c034000 tx=0x7fb63c02e000 comp rx=0 tx=0).stop 2024-05-09T05:11:31.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb64c072120 msgr2=0x7fb64c0afff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb64c072120 0x7fb64c0afff0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb64800c430 tx=0x7fb64800c8f0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 shutdown_connections 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6300762d0 0x7fb6300787e0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb64c0c62e0 0x7fb64c0cda80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb64c0ca340 0x7fb64c1f2dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 --2- 172.21.15.1:0/3615201703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb64c072120 0x7fb64c0afff0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.835+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 >> 172.21.15.1:0/3615201703 conn(0x7fb64c0731d0 msgr2=0x7fb64c0ca750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.836+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 shutdown_connections 2024-05-09T05:11:31.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:31.836+0000 7fb653dea640 1 -- 172.21.15.1:0/3615201703 wait complete. 2024-05-09T05:11:31.849 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:11:32.763 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:11:32.765 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:11:41.849 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:11:41.850 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:11:41.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.970+0000 7f7a173a8640 1 Processor -- start 2024-05-09T05:11:41.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.971+0000 7f7a173a8640 1 -- start start 2024-05-09T05:11:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a173a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a1015ca90 0x7f7a1015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a173a8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a1006f6d0 con 0x7f7a100d16b0 2024-05-09T05:11:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a173a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7a100cc4d0 con 0x7f7a1006f1b0 2024-05-09T05:11:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a173a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7a100cee50 con 0x7f7a1015ca90 2024-05-09T05:11:41.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a15144640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7a100d16b0 0x7f7a100d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44364/0 (socket says 172.21.15.1:44364) 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a15144640 1 -- 172.21.15.1:0/53681867 learned_addr learned my addr 172.21.15.1:0/53681867 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a15945640 1 --2- 172.21.15.1:0/53681867 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a1015ca90 0x7f7a1015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a07fff640 1 -- 172.21.15.1:0/53681867 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1484142448 0 0) 0x7f7a1006f6d0 con 0x7f7a100d16b0 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a07fff640 1 -- 172.21.15.1:0/53681867 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79fc003520 con 0x7f7a100d16b0 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.972+0000 7f7a07fff640 1 -- 172.21.15.1:0/53681867 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 246972440 0 0) 0x7f7a100cc4d0 con 0x7f7a1006f1b0 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.973+0000 7f7a07fff640 1 -- 172.21.15.1:0/53681867 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7a1006f6d0 con 0x7f7a1006f1b0 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.973+0000 7f7a15945640 1 -- 172.21.15.1:0/53681867 >> v1:172.21.15.196:6789/0 conn(0x7f7a1006f1b0 legacy=0x7f7a100ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.973+0000 7f7a15945640 1 -- 172.21.15.1:0/53681867 >> v1:172.21.15.1:6789/0 conn(0x7f7a100d16b0 legacy=0x7f7a100d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:41.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.973+0000 7f7a15945640 1 -- 172.21.15.1:0/53681867 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a100cc7b0 con 0x7f7a1015ca90 2024-05-09T05:11:41.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.973+0000 7f7a15945640 1 --2- 172.21.15.1:0/53681867 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a1015ca90 0x7f7a1015cea0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009dc0 tx=0x7f7a0802a760 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f1e9e3e71d3ee00d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:41.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.974+0000 7f7a07fff640 1 -- 172.21.15.1:0/53681867 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08007740 con 0x7f7a1015ca90 2024-05-09T05:11:41.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.974+0000 7f7a07fff640 1 -- 172.21.15.1:0/53681867 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a080078e0 con 0x7f7a1015ca90 2024-05-09T05:11:41.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.974+0000 7f7a07fff640 1 -- 172.21.15.1:0/53681867 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a08007cb0 con 0x7f7a1015ca90 2024-05-09T05:11:41.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.975+0000 7f7a173a8640 1 -- 172.21.15.1:0/53681867 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a1015ca90 msgr2=0x7f7a1015cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:41.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.975+0000 7f7a173a8640 1 --2- 172.21.15.1:0/53681867 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a1015ca90 0x7f7a1015cea0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f7a08009dc0 tx=0x7f7a0802a760 comp rx=0 tx=0).stop 2024-05-09T05:11:41.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.975+0000 7f7a173a8640 1 -- 172.21.15.1:0/53681867 shutdown_connections 2024-05-09T05:11:41.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.975+0000 7f7a173a8640 1 --2- 172.21.15.1:0/53681867 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a1015ca90 0x7f7a1015cea0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:41.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.975+0000 7f7a173a8640 1 -- 172.21.15.1:0/53681867 >> 172.21.15.1:0/53681867 conn(0x7f7a100b3660 msgr2=0x7f7a100b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:41.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.975+0000 7f7a173a8640 1 -- 172.21.15.1:0/53681867 shutdown_connections 2024-05-09T05:11:41.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.975+0000 7f7a173a8640 1 -- 172.21.15.1:0/53681867 wait complete. 2024-05-09T05:11:41.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.976+0000 7f7a173a8640 1 Processor -- start 2024-05-09T05:11:41.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.977+0000 7f7a173a8640 1 -- start start 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a173a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1006f1b0 0x7f7a100cbd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a173a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a100d16b0 0x7f7a10151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a173a8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a1015ca90 0x7f7a100c6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a173a8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7a100ccd70 con 0x7f7a1006f1b0 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a173a8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7a100cd0e0 con 0x7f7a1015ca90 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a173a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7a1015d3e0 con 0x7f7a100d16b0 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15144640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1006f1b0 0x7f7a100cbd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15144640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1006f1b0 0x7f7a100cbd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38668/0 (socket says 172.21.15.1:38668) 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15144640 1 -- 172.21.15.1:0/1120935300 learned_addr learned my addr 172.21.15.1:0/1120935300 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a14943640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a100d16b0 0x7f7a10151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15945640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a1015ca90 0x7f7a100c6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:41.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15144640 1 -- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a100d16b0 msgr2=0x7f7a10151f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15144640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a100d16b0 0x7f7a10151f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15144640 1 -- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a1015ca90 msgr2=0x7f7a100c6220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15144640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a1015ca90 0x7f7a100c6220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.978+0000 7f7a15144640 1 -- 172.21.15.1:0/1120935300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a100cb000 con 0x7f7a1006f1b0 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.979+0000 7f7a15144640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1006f1b0 0x7f7a100cbd80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c440 tx=0x7f7a0c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.979+0000 7f7a05ffb640 1 -- 172.21.15.1:0/1120935300 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c017070 con 0x7f7a1006f1b0 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.979+0000 7f7a05ffb640 1 -- 172.21.15.1:0/1120935300 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7a0c005da0 con 0x7f7a1006f1b0 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.979+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a100cb2a0 con 0x7f7a1006f1b0 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.979+0000 7f7a05ffb640 1 -- 172.21.15.1:0/1120935300 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a0c0072b0 con 0x7f7a1006f1b0 2024-05-09T05:11:41.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.979+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a100cb860 con 0x7f7a1006f1b0 2024-05-09T05:11:41.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.980+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a1015d200 con 0x7f7a1006f1b0 2024-05-09T05:11:41.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.981+0000 7f7a05ffb640 1 -- 172.21.15.1:0/1120935300 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7a0c01a070 con 0x7f7a1006f1b0 2024-05-09T05:11:41.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.982+0000 7f7a05ffb640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79f4076300 0x7f79f4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:41.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.982+0000 7f7a05ffb640 1 -- 172.21.15.1:0/1120935300 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7a0c096810 con 0x7f7a1006f1b0 2024-05-09T05:11:41.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.983+0000 7f7a14943640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79f4076300 0x7f79f4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:41.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.983+0000 7f7a14943640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79f4076300 0x7f79f4078810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7a100ca610 tx=0x7f7a00009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:41.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:41.988+0000 7f7a05ffb640 1 -- 172.21.15.1:0/1120935300 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7a1015d200 con 0x7f7a1006f1b0 2024-05-09T05:11:42.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.315+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7a10050870 con 0x7f7a1006f1b0 2024-05-09T05:11:42.316 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.315+0000 7f7a05ffb640 1 -- 172.21.15.1:0/1120935300 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f7a0c05faa0 con 0x7f7a1006f1b0 2024-05-09T05:11:42.316 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:11:42.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79f4076300 msgr2=0x7f79f4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79f4076300 0x7f79f4078810 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f7a100ca610 tx=0x7f7a00009040 comp rx=0 tx=0).stop 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1006f1b0 msgr2=0x7f7a100cbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1006f1b0 0x7f7a100cbd80 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7a0c00c440 tx=0x7f7a0c00c900 comp rx=0 tx=0).stop 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 shutdown_connections 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79f4076300 0x7f79f4078810 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7a1015ca90 0x7f7a100c6220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7a100d16b0 0x7f7a10151f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 --2- 172.21.15.1:0/1120935300 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7a1006f1b0 0x7f7a100cbd80 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 >> 172.21.15.1:0/1120935300 conn(0x7f7a100b3660 msgr2=0x7f7a1015a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 shutdown_connections 2024-05-09T05:11:42.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.318+0000 7f7a173a8640 1 -- 172.21.15.1:0/1120935300 wait complete. 2024-05-09T05:11:42.331 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:11:42.331 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:11:42.331 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:11:42.331 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:11:42.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.459+0000 7fd99aebd640 1 Processor -- start 2024-05-09T05:11:42.459 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.459+0000 7fd99aebd640 1 -- start start 2024-05-09T05:11:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99aebd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9940b4ec0 0x7fd9940cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99aebd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd9940ce010 0x7fd99415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99aebd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd994160290 0x7fd9941606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99aebd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd99415d0c0 con 0x7fd994160290 2024-05-09T05:11:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99aebd640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd994160be0 con 0x7fd9940ce010 2024-05-09T05:11:42.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99aebd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd99415c7b0 con 0x7fd9940b4ec0 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99945a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd994160290 0x7fd9941606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99945a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd994160290 0x7fd9941606a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38674/0 (socket says 172.21.15.1:38674) 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd99945a640 1 -- 172.21.15.1:0/3773617281 learned_addr learned my addr 172.21.15.1:0/3773617281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd993fff640 1 --2- 172.21.15.1:0/3773617281 >> v2:172.21.15.196:3300/0 conn(0x7fd9940ce010 0x7fd99415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.460+0000 7fd998c59640 1 --2- 172.21.15.1:0/3773617281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9940b4ec0 0x7fd9940cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.461+0000 7fd99945a640 1 -- 172.21.15.1:0/3773617281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9940b4ec0 msgr2=0x7fd9940cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.461+0000 7fd99945a640 1 --2- 172.21.15.1:0/3773617281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9940b4ec0 0x7fd9940cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.461+0000 7fd99945a640 1 -- 172.21.15.1:0/3773617281 >> v2:172.21.15.196:3300/0 conn(0x7fd9940ce010 msgr2=0x7fd99415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.461+0000 7fd99945a640 1 --2- 172.21.15.1:0/3773617281 >> v2:172.21.15.196:3300/0 conn(0x7fd9940ce010 0x7fd99415fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.461+0000 7fd99945a640 1 -- 172.21.15.1:0/3773617281 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9940cca50 con 0x7fd994160290 2024-05-09T05:11:42.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.461+0000 7fd99945a640 1 --2- 172.21.15.1:0/3773617281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd994160290 0x7fd9941606a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd9940d1680 tx=0x7fd98c02a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=362526a82986210c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.462+0000 7fd9937fe640 1 -- 172.21.15.1:0/3773617281 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c036070 con 0x7fd994160290 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.462+0000 7fd9937fe640 1 -- 172.21.15.1:0/3773617281 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd98c031600 con 0x7fd994160290 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.462+0000 7fd9937fe640 1 -- 172.21.15.1:0/3773617281 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c0319d0 con 0x7fd994160290 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.462+0000 7fd99aebd640 1 -- 172.21.15.1:0/3773617281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd994160290 msgr2=0x7fd9941606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.462+0000 7fd99aebd640 1 --2- 172.21.15.1:0/3773617281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd994160290 0x7fd9941606a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fd9940d1680 tx=0x7fd98c02a730 comp rx=0 tx=0).stop 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.462+0000 7fd99aebd640 1 -- 172.21.15.1:0/3773617281 shutdown_connections 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.463+0000 7fd99aebd640 1 --2- 172.21.15.1:0/3773617281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd994160290 0x7fd9941606a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.463+0000 7fd99aebd640 1 --2- 172.21.15.1:0/3773617281 >> v2:172.21.15.196:3300/0 conn(0x7fd9940ce010 0x7fd99415fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.463+0000 7fd99aebd640 1 --2- 172.21.15.1:0/3773617281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd9940b4ec0 0x7fd9940cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.463+0000 7fd99aebd640 1 -- 172.21.15.1:0/3773617281 >> 172.21.15.1:0/3773617281 conn(0x7fd9940d2360 msgr2=0x7fd9940ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.463+0000 7fd99aebd640 1 -- 172.21.15.1:0/3773617281 shutdown_connections 2024-05-09T05:11:42.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.463+0000 7fd99aebd640 1 -- 172.21.15.1:0/3773617281 wait complete. 2024-05-09T05:11:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.465+0000 7fd99aebd640 1 Processor -- start 2024-05-09T05:11:42.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.465+0000 7fd99aebd640 1 -- start start 2024-05-09T05:11:42.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd99aebd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9940b4ec0 0x7fd9941f31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd99aebd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9940ce010 0x7fd9941e8f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd99aebd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd994160290 0x7fd9941e94a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd99aebd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd9940cc750 con 0x7fd9940ce010 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd99aebd640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd99415cd50 con 0x7fd9940b4ec0 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd99aebd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd9940cff10 con 0x7fd994160290 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd993fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9940ce010 0x7fd9941e8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd993fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9940ce010 0x7fd9941e8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38676/0 (socket says 172.21.15.1:38676) 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd993fff640 1 -- 172.21.15.1:0/4020681520 learned_addr learned my addr 172.21.15.1:0/4020681520 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd998c59640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9940b4ec0 0x7fd9941f31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd99945a640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd994160290 0x7fd9941e94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd998c59640 1 -- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd994160290 msgr2=0x7fd9941e94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd998c59640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd994160290 0x7fd9941e94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd998c59640 1 -- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9940ce010 msgr2=0x7fd9941e8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd998c59640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9940ce010 0x7fd9941e8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd998c59640 1 -- 172.21.15.1:0/4020681520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd9941e8d10 con 0x7fd9940b4ec0 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd99945a640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd994160290 0x7fd9941e94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.467+0000 7fd993fff640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd9940ce010 0x7fd9941e8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.468+0000 7fd998c59640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9940b4ec0 0x7fd9941f31c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd98800d580 tx=0x7fd98800da40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.468+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988018020 con 0x7fd9940b4ec0 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.468+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd988014620 con 0x7fd9940b4ec0 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.468+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd9941f2700 con 0x7fd9940b4ec0 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.468+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd988014a10 con 0x7fd9940b4ec0 2024-05-09T05:11:42.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.468+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd9941f29c0 con 0x7fd9940b4ec0 2024-05-09T05:11:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.469+0000 7fd99aebd640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99406f060 0x7fd994151160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.469+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd994150e20 con 0x7fd99406f060 2024-05-09T05:11:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.469+0000 7fd993fff640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99406f060 0x7fd994151160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.469+0000 7fd993fff640 1 -- 172.21.15.1:0/4020681520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd994050870 con 0x7fd99406f060 2024-05-09T05:11:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.469+0000 7fd993fff640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99406f060 0x7fd994151160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd9940cdd60 tx=0x7fd98000e1d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.470+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd980018020 con 0x7fd99406f060 2024-05-09T05:11:42.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.470+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd98000ed40 con 0x7fd99406f060 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.470+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99406f060 msgr2=0x7fd994151160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.470+0000 7fd9917fa640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99406f060 0x7fd994151160 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fd9940cdd60 tx=0x7fd98000e1d0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.470+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99406f060 msgr2=0x7fd994151160 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.470+0000 7fd9917fa640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99406f060 0x7fd994151160 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.470+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd98801b020 con 0x7fd9940b4ec0 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.471+0000 7fd9917fa640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd978075db0 0x7fd9780782c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.471+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd988097a80 con 0x7fd9940b4ec0 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.471+0000 7fd993fff640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd978075db0 0x7fd9780782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.472+0000 7fd993fff640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd978075db0 0x7fd9780782c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd980000c00 tx=0x7fd980014000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.480+0000 7fd99aebd640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99414d200 0x7fd99414d610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.480+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd9940cd2b0 con 0x7fd99414d200 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.481+0000 7fd99945a640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99414d200 0x7fd99414d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.481+0000 7fd99945a640 1 -- 172.21.15.1:0/4020681520 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd9940cc000 con 0x7fd99414d200 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.481+0000 7fd99945a640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99414d200 0x7fd99414d610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9940b4a70 tx=0x7fd98c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.481+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd98c036040 con 0x7fd99414d200 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4893 (secure 0 0 0) 0x7fd98c037040 con 0x7fd99414d200 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99414d200 msgr2=0x7fd99414d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd9917fa640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99414d200 0x7fd99414d610 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fd9940b4a70 tx=0x7fd98c004a40 comp rx=0 tx=0).stop 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd9917fa640 1 -- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99414d200 msgr2=0x7fd99414d610 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:42.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd9917fa640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd99414d200 0x7fd99414d610 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 50, 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 91076, 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:42.484 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.485 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.486 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:42.487 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.488 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.489 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.490 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.491 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:42.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:42.493 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.494 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.495 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:42.496 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd978075db0 msgr2=0x7fd9780782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd978075db0 0x7fd9780782c0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fd980000c00 tx=0x7fd980014000 comp rx=0 tx=0).stop 2024-05-09T05:11:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9940b4ec0 msgr2=0x7fd9941f31c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9940b4ec0 0x7fd9941f31c0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fd98800d580 tx=0x7fd98800da40 comp rx=0 tx=0).stop 2024-05-09T05:11:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99945a640 1 -- 172.21.15.1:0/4020681520 reap_dead start 2024-05-09T05:11:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 shutdown_connections 2024-05-09T05:11:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 --2- 172.21.15.1:0/4020681520 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd9940b4ec0 0x7fd9941f31c0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 >> 172.21.15.1:0/4020681520 conn(0x7fd9940d2360 msgr2=0x7fd9940b60e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 shutdown_connections 2024-05-09T05:11:42.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.482+0000 7fd99aebd640 1 -- 172.21.15.1:0/4020681520 wait complete. 2024-05-09T05:11:42.499 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:11:42.499 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:11:42.499 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:11:42.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.623+0000 7f8c1e91f640 1 Processor -- start 2024-05-09T05:11:42.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.623+0000 7f8c1e91f640 1 -- start start 2024-05-09T05:11:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.624+0000 7f8c1e91f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c180736b0 0x7f8c1815d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.624+0000 7f8c1e91f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 0x7f8c1806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.624+0000 7f8c1e91f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8c180cc1b0 con 0x7f8c1806dde0 2024-05-09T05:11:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.624+0000 7f8c1e91f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8c1806d710 con 0x7f8c180736b0 2024-05-09T05:11:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.624+0000 7f8c1e91f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8c180cbf60 con 0x7f8c180c5600 2024-05-09T05:11:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.624+0000 7f8c177fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8c180736b0 0x7f8c1815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.624+0000 7f8c17fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8c1806dde0 0x7f8c180c70b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44376/0 (socket says 172.21.15.1:44376) 2024-05-09T05:11:42.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.624+0000 7f8c17fff640 1 -- 172.21.15.1:0/1343004075 learned_addr learned my addr 172.21.15.1:0/1343004075 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:42.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.625+0000 7f8c1cebc640 1 --2- 172.21.15.1:0/1343004075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 0x7f8c1806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.625+0000 7f8c177fe640 1 -- 172.21.15.1:0/1343004075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 msgr2=0x7f8c1806d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:42.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.625+0000 7f8c177fe640 1 --2- 172.21.15.1:0/1343004075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 0x7f8c1806d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.625+0000 7f8c177fe640 1 -- 172.21.15.1:0/1343004075 >> v1:172.21.15.1:6789/0 conn(0x7f8c1806dde0 legacy=0x7f8c180c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.625+0000 7f8c177fe640 1 -- 172.21.15.1:0/1343004075 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c180cc490 con 0x7f8c180736b0 2024-05-09T05:11:42.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.625+0000 7f8c177fe640 1 --2- 172.21.15.1:0/1343004075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c180736b0 0x7f8c1815d330 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0097a0 tx=0x7f8c0c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18ce1989a2a55afd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.626+0000 7f8c16ffd640 1 -- 172.21.15.1:0/1343004075 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c037070 con 0x7f8c180736b0 2024-05-09T05:11:42.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.626+0000 7f8c16ffd640 1 -- 172.21.15.1:0/1343004075 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c0c032440 con 0x7f8c180736b0 2024-05-09T05:11:42.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.626+0000 7f8c16ffd640 1 -- 172.21.15.1:0/1343004075 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c032810 con 0x7f8c180736b0 2024-05-09T05:11:42.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.626+0000 7f8c1e91f640 1 -- 172.21.15.1:0/1343004075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c180736b0 msgr2=0x7f8c1815d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.626+0000 7f8c1e91f640 1 --2- 172.21.15.1:0/1343004075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c180736b0 0x7f8c1815d330 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c0097a0 tx=0x7f8c0c02a180 comp rx=0 tx=0).stop 2024-05-09T05:11:42.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.627+0000 7f8c1e91f640 1 -- 172.21.15.1:0/1343004075 shutdown_connections 2024-05-09T05:11:42.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.627+0000 7f8c1e91f640 1 --2- 172.21.15.1:0/1343004075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 0x7f8c1806d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.627+0000 7f8c1e91f640 1 --2- 172.21.15.1:0/1343004075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c180736b0 0x7f8c1815d330 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.627+0000 7f8c1e91f640 1 -- 172.21.15.1:0/1343004075 >> 172.21.15.1:0/1343004075 conn(0x7f8c180ce180 msgr2=0x7f8c180ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:42.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.627+0000 7f8c1e91f640 1 -- 172.21.15.1:0/1343004075 shutdown_connections 2024-05-09T05:11:42.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.627+0000 7f8c1e91f640 1 -- 172.21.15.1:0/1343004075 wait complete. 2024-05-09T05:11:42.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.628+0000 7f8c1e91f640 1 Processor -- start 2024-05-09T05:11:42.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.629+0000 7f8c1e91f640 1 -- start start 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c1e91f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1806dde0 0x7f8c1815f460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c1e91f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c180736b0 0x7f8c180c8a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c1e91f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 0x7f8c181e96a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c1e91f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1806ccb0 con 0x7f8c1806dde0 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c1e91f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8c1806c9d0 con 0x7f8c180736b0 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c1e91f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8c180c77c0 con 0x7f8c180c5600 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1806dde0 0x7f8c1815f460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c17fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1806dde0 0x7f8c1815f460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38696/0 (socket says 172.21.15.1:38696) 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c17fff640 1 -- 172.21.15.1:0/2097551180 learned_addr learned my addr 172.21.15.1:0/2097551180 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c1cebc640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 0x7f8c181e96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c177fe640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c180736b0 0x7f8c180c8a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c17fff640 1 -- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 msgr2=0x7f8c181e96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c17fff640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8c180c5600 0x7f8c181e96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c17fff640 1 -- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c180736b0 msgr2=0x7f8c180c8a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c17fff640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c180736b0 0x7f8c180c8a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.630+0000 7f8c17fff640 1 -- 172.21.15.1:0/2097551180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c1815f9a0 con 0x7f8c1806dde0 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.631+0000 7f8c17fff640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1806dde0 0x7f8c1815f460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800c400 tx=0x7f8c0800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.631+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c08017070 con 0x7f8c1806dde0 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.631+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8c08005d00 con 0x7f8c1806dde0 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.631+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c180c9690 con 0x7f8c1806dde0 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.631+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c08007310 con 0x7f8c1806dde0 2024-05-09T05:11:42.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.631+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c1815ec60 con 0x7f8c1806dde0 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.632+0000 7f8c1e91f640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815fb70 0x7f8c1814f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.632+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8c180c9440 con 0x7f8c1815fb70 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.632+0000 7f8c177fe640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815fb70 0x7f8c1814f9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.633+0000 7f8c177fe640 1 -- 172.21.15.1:0/2097551180 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8c18050870 con 0x7f8c1815fb70 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.633+0000 7f8c177fe640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815fb70 0x7f8c1814f9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8c18072ba0 tx=0x7f8c0c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.633+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8c0801a070 con 0x7f8c1806dde0 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c14ff9640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8c040762d0 0x7f8c040787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8c08096ac0 con 0x7f8c1806dde0 2024-05-09T05:11:42.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c1cebc640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8c040762d0 0x7f8c040787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c041070 con 0x7f8c1815fb70 2024-05-09T05:11:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8c0c03c660 con 0x7f8c1815fb70 2024-05-09T05:11:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815fb70 msgr2=0x7f8c1814f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c14ff9640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815fb70 0x7f8c1814f9a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f8c18072ba0 tx=0x7f8c0c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815fb70 msgr2=0x7f8c1814f9a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.634+0000 7f8c14ff9640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815fb70 0x7f8c1814f9a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.635+0000 7f8c1cebc640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8c040762d0 0x7f8c040787e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8c18153b70 tx=0x7f8c1000b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.643+0000 7f8c1e91f640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815da70 0x7f8c1814fee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.643+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8c180748f0 con 0x7f8c1815da70 2024-05-09T05:11:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.643+0000 7f8c177fe640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815da70 0x7f8c1814fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.644+0000 7f8c177fe640 1 -- 172.21.15.1:0/2097551180 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8c180ce850 con 0x7f8c1815da70 2024-05-09T05:11:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.644+0000 7f8c177fe640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815da70 0x7f8c1814fee0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c02a1c0 tx=0x7f8c0c008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.644+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c0c037040 con 0x7f8c1815da70 2024-05-09T05:11:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.644+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f8c0c047e50 con 0x7f8c1815da70 2024-05-09T05:11:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.644+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815da70 msgr2=0x7f8c1814fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.644+0000 7f8c14ff9640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815da70 0x7f8c1814fee0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c02a1c0 tx=0x7f8c0c008030 comp rx=0 tx=0).stop 2024-05-09T05:11:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.644+0000 7f8c14ff9640 1 -- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815da70 msgr2=0x7f8c1814fee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.644+0000 7f8c14ff9640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8c1815da70 0x7f8c1814fee0 secure :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f8c0c02a1c0 tx=0x7f8c0c008030 comp rx=0 tx=0).stop 2024-05-09T05:11:42.645 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 50, 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15637, 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:42.646 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:42.647 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.648 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:42.649 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.650 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.651 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:42.652 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.653 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:42.654 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.655 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.656 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:42.657 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.657 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.657 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.657 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:42.657 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8c040762d0 msgr2=0x7f8c040787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1e91f640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8c040762d0 0x7f8c040787e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8c18153b70 tx=0x7f8c1000b040 comp rx=0 tx=0).stop 2024-05-09T05:11:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1806dde0 msgr2=0x7f8c1815f460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1e91f640 1 --2- 172.21.15.1:0/2097551180 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8c1806dde0 0x7f8c1815f460 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f8c0800c400 tx=0x7f8c0800c8c0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1cebc640 1 -- 172.21.15.1:0/2097551180 reap_dead start 2024-05-09T05:11:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1cebc640 1 -- 172.21.15.1:0/2097551180 reap_dead start 2024-05-09T05:11:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 shutdown_connections 2024-05-09T05:11:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 >> 172.21.15.1:0/2097551180 conn(0x7f8c180ce180 msgr2=0x7f8c18154340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 shutdown_connections 2024-05-09T05:11:42.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.645+0000 7f8c1e91f640 1 -- 172.21.15.1:0/2097551180 wait complete. 2024-05-09T05:11:42.659 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:11:42.659 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:11:42.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:11:42.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.782+0000 7fb8949fa640 1 Processor -- start 2024-05-09T05:11:42.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.782+0000 7fb8949fa640 1 -- start start 2024-05-09T05:11:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.783+0000 7fb8949fa640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8900b4ec0 0x7fb8900cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.783+0000 7fb8949fa640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb8900ce010 0x7fb89015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.784 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.783+0000 7fb8949fa640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890160290 0x7fb8901606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.783+0000 7fb8949fa640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb89015d0c0 con 0x7fb8900b4ec0 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.783+0000 7fb8949fa640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb890160be0 con 0x7fb8900ce010 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.783+0000 7fb8949fa640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb89015c7b0 con 0x7fb890160290 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8900b4ec0 0x7fb8900cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb8900b4ec0 0x7fb8900cf9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38702/0 (socket says 172.21.15.1:38702) 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88e59c640 1 -- 172.21.15.1:0/1334959897 learned_addr learned my addr 172.21.15.1:0/1334959897 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88dd9b640 1 --2- 172.21.15.1:0/1334959897 >> v2:172.21.15.196:3300/0 conn(0x7fb8900ce010 0x7fb89015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88ed9d640 1 --2- 172.21.15.1:0/1334959897 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890160290 0x7fb8901606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88e59c640 1 -- 172.21.15.1:0/1334959897 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890160290 msgr2=0x7fb8901606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88e59c640 1 --2- 172.21.15.1:0/1334959897 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890160290 0x7fb8901606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88e59c640 1 -- 172.21.15.1:0/1334959897 >> v2:172.21.15.196:3300/0 conn(0x7fb8900ce010 msgr2=0x7fb89015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88e59c640 1 --2- 172.21.15.1:0/1334959897 >> v2:172.21.15.196:3300/0 conn(0x7fb8900ce010 0x7fb89015fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88e59c640 1 -- 172.21.15.1:0/1334959897 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb8900cca50 con 0x7fb8900b4ec0 2024-05-09T05:11:42.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.784+0000 7fb88ed9d640 1 --2- 172.21.15.1:0/1334959897 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890160290 0x7fb8901606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.785+0000 7fb88e59c640 1 --2- 172.21.15.1:0/1334959897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8900b4ec0 0x7fb8900cf9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb8900c8b40 tx=0x7fb88002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e6d4250749a275 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.785+0000 7fb88d59a640 1 -- 172.21.15.1:0/1334959897 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880037070 con 0x7fb8900b4ec0 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.785+0000 7fb88d59a640 1 -- 172.21.15.1:0/1334959897 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb880032440 con 0x7fb8900b4ec0 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.785+0000 7fb88d59a640 1 -- 172.21.15.1:0/1334959897 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880032810 con 0x7fb8900b4ec0 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 -- 172.21.15.1:0/1334959897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8900b4ec0 msgr2=0x7fb8900cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 --2- 172.21.15.1:0/1334959897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8900b4ec0 0x7fb8900cf9d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fb8900c8b40 tx=0x7fb88002a270 comp rx=0 tx=0).stop 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 -- 172.21.15.1:0/1334959897 shutdown_connections 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 --2- 172.21.15.1:0/1334959897 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890160290 0x7fb8901606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 --2- 172.21.15.1:0/1334959897 >> v2:172.21.15.196:3300/0 conn(0x7fb8900ce010 0x7fb89015fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 --2- 172.21.15.1:0/1334959897 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb8900b4ec0 0x7fb8900cf9d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 -- 172.21.15.1:0/1334959897 >> 172.21.15.1:0/1334959897 conn(0x7fb8900d2360 msgr2=0x7fb8900ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 -- 172.21.15.1:0/1334959897 shutdown_connections 2024-05-09T05:11:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.786+0000 7fb8949fa640 1 -- 172.21.15.1:0/1334959897 wait complete. 2024-05-09T05:11:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.788+0000 7fb8949fa640 1 Processor -- start 2024-05-09T05:11:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.788+0000 7fb8949fa640 1 -- start start 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb8949fa640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb8900b4ec0 0x7fb8901f3280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb8949fa640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8900ce010 0x7fb89015b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb8949fa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb890160290 0x7fb89015bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb8949fa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb8900cc750 con 0x7fb890160290 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb8949fa640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb89015cd50 con 0x7fb8900ce010 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb8949fa640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb8900cff10 con 0x7fb8900b4ec0 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb88ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb890160290 0x7fb89015bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb88ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb890160290 0x7fb89015bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38706/0 (socket says 172.21.15.1:38706) 2024-05-09T05:11:42.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.789+0000 7fb88ed9d640 1 -- 172.21.15.1:0/2261411082 learned_addr learned my addr 172.21.15.1:0/2261411082 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88e59c640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb8900b4ec0 0x7fb8901f3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88dd9b640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8900ce010 0x7fb89015b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88ed9d640 1 -- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb8900b4ec0 msgr2=0x7fb8901f3280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88ed9d640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb8900b4ec0 0x7fb8901f3280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88ed9d640 1 -- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8900ce010 msgr2=0x7fb89015b800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88ed9d640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8900ce010 0x7fb89015b800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88ed9d640 1 -- 172.21.15.1:0/2261411082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb89014dd00 con 0x7fb890160290 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88dd9b640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb8900ce010 0x7fb89015b800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb88ed9d640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb890160290 0x7fb89015bd40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb888007e60 tx=0x7fb88800d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb88801a070 con 0x7fb890160290 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb888015600 con 0x7fb890160290 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb890152380 con 0x7fb890160290 2024-05-09T05:11:42.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb8880159f0 con 0x7fb890160290 2024-05-09T05:11:42.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.790+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb890150a70 con 0x7fb890160290 2024-05-09T05:11:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.791+0000 7fb8949fa640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890150c40 0x7fb89006e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.791+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb8901f37c0 con 0x7fb890150c40 2024-05-09T05:11:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.792+0000 7fb88e59c640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890150c40 0x7fb89006e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.792+0000 7fb88e59c640 1 -- 172.21.15.1:0/2261411082 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb890152660 con 0x7fb890150c40 2024-05-09T05:11:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.792+0000 7fb88e59c640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890150c40 0x7fb89006e920 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8900d15d0 tx=0x7fb8800046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.792+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb88801d070 con 0x7fb890160290 2024-05-09T05:11:42.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb87effd640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb870076300 0x7fb870078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb8880977a0 con 0x7fb890160290 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb88dd9b640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb870076300 0x7fb870078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880042070 con 0x7fb890150c40 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb88003e660 con 0x7fb890150c40 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890150c40 msgr2=0x7fb89006e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb87effd640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890150c40 0x7fb89006e920 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb8900d15d0 tx=0x7fb8800046c0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890150c40 msgr2=0x7fb89006e920 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.793+0000 7fb87effd640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb890150c40 0x7fb89006e920 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.794+0000 7fb88dd9b640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb870076300 0x7fb870078810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb89015c320 tx=0x7fb884008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.801+0000 7fb8949fa640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb89014ded0 0x7fb89006ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.801+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb890152110 con 0x7fb89014ded0 2024-05-09T05:11:42.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.801+0000 7fb88e59c640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb89014ded0 0x7fb89006ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.801+0000 7fb88e59c640 1 -- 172.21.15.1:0/2261411082 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb890050870 con 0x7fb89014ded0 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.802+0000 7fb88e59c640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb89014ded0 0x7fb89006ee60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb88002a2b0 tx=0x7fb880008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.802+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb880037040 con 0x7fb89014ded0 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.802+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7fb88003ee10 con 0x7fb89014ded0 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.802+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb89014ded0 msgr2=0x7fb89006ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.802+0000 7fb87effd640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb89014ded0 0x7fb89006ee60 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fb88002a2b0 tx=0x7fb880008030 comp rx=0 tx=0).stop 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.802+0000 7fb87effd640 1 -- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb89014ded0 msgr2=0x7fb89006ee60 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.802+0000 7fb87effd640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb89014ded0 0x7fb89006ee60 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.803 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 50, 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 260799, 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:42.804 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.805 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.806 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:42.807 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:42.808 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:42.809 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.810 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.811 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:42.812 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.813 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:42.814 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.815 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:42.816 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:42.817 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb870076300 msgr2=0x7fb870078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb8949fa640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb870076300 0x7fb870078810 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fb89015c320 tx=0x7fb884008040 comp rx=0 tx=0).stop 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb890160290 msgr2=0x7fb89015bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb8949fa640 1 --2- 172.21.15.1:0/2261411082 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb890160290 0x7fb89015bd40 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fb888007e60 tx=0x7fb88800d4a0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb88ed9d640 1 -- 172.21.15.1:0/2261411082 reap_dead start 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb88ed9d640 1 -- 172.21.15.1:0/2261411082 reap_dead start 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 shutdown_connections 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 >> 172.21.15.1:0/2261411082 conn(0x7fb8900d2360 msgr2=0x7fb890152880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 shutdown_connections 2024-05-09T05:11:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.803+0000 7fb8949fa640 1 -- 172.21.15.1:0/2261411082 wait complete. 2024-05-09T05:11:42.819 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:11:42.819 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-05-09T05:11:42.819 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:11:42.819 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-05-09T05:11:42.819 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:11:42.819 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T05:11:42.819 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:11:42.832 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T05:11:42.833 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:11:42.833 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:11:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.961+0000 7f2495f9f640 1 Processor -- start 2024-05-09T05:11:42.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.961+0000 7f2495f9f640 1 -- start start 2024-05-09T05:11:42.966 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.962+0000 7f2495f9f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f249015c790 0x7f24901605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.962+0000 7f2495f9f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24900c8450 con 0x7f24900705e0 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.962+0000 7f2495f9f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f24900c8730 con 0x7f24900c5a50 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.962+0000 7f2495f9f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f249006fe50 con 0x7f249015c790 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.962+0000 7f2494f9d640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f249015c790 msgr2=0x7f24901605a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.962+0000 7f249579e640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f24900c5a50 legacy=0x7f2490167030 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.962+0000 7f2494f9d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f249015c790 0x7f24901605a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f24900705e0 0x7f24900c72a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44378/0 (socket says 172.21.15.1:44378) 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248ffff640 1 -- 172.21.15.1:0/3898548784 learned_addr learned my addr 172.21.15.1:0/3898548784 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 515897184 0 0) 0x7f24900c8450 con 0x7f24900705e0 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2480003520 con 0x7f24900705e0 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 428606079 0 0) 0x7f2480003520 con 0x7f24900705e0 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 >> v2:172.21.15.196:3300/0 conn(0x7f249015c790 msgr2=0x7f24901605a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:42.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 --2- 172.21.15.1:0/3898548784 >> v2:172.21.15.196:3300/0 conn(0x7f249015c790 0x7f24901605a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 >> v1:172.21.15.196:6789/0 conn(0x7f24900c5a50 legacy=0x7f2490167030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f249015fda0 con 0x7f24900705e0 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f247c0016f0 con 0x7f24900705e0 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 >> v1:172.21.15.1:6789/0 conn(0x7f24900705e0 legacy=0x7f24900c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2480003a20 0x7f2480003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2480004800 0x7f2480004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24800053f0 0x7f248000ba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f249006fe50 con 0x7f24800053f0 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f248000c350 con 0x7f2480003a20 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f248000c550 con 0x7f2480004800 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f2494f9d640 1 -- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2480003a20 msgr2=0x7f2480003e90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f2494f9d640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2480003a20 0x7f2480003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.963+0000 7f249579e640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24800053f0 0x7f248000ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.964+0000 7f248ffff640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2480004800 0x7f2480004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.968 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.964+0000 7f249579e640 1 -- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2480004800 msgr2=0x7f2480004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.964+0000 7f249579e640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2480004800 0x7f2480004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.964+0000 7f249579e640 1 -- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2480003a20 msgr2=0x7f2480003e90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.964+0000 7f249579e640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2480003a20 0x7f2480003e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.964+0000 7f249579e640 1 -- 172.21.15.1:0/3898548784 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2480003520 con 0x7f24800053f0 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.965+0000 7f249579e640 1 -- 172.21.15.1:0/3898548784 reap_dead start 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.965+0000 7f249579e640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24800053f0 0x7f248000ba70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2488009e30 tx=0x7f2480005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14a8d9b260359bf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.965+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2488009a30 con 0x7f24800053f0 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.965+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f248802c040 con 0x7f24800053f0 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.965+0000 7f248f7fe640 1 -- 172.21.15.1:0/3898548784 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2488005550 con 0x7f24800053f0 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.966+0000 7f2495f9f640 1 -- 172.21.15.1:0/3898548784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24800053f0 msgr2=0x7f248000ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.966+0000 7f2495f9f640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24800053f0 0x7f248000ba70 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f2488009e30 tx=0x7f2480005830 comp rx=0 tx=0).stop 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.966+0000 7f2495f9f640 1 -- 172.21.15.1:0/3898548784 shutdown_connections 2024-05-09T05:11:42.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.966+0000 7f2495f9f640 1 --2- 172.21.15.1:0/3898548784 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24800053f0 0x7f248000ba70 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.966+0000 7f2495f9f640 1 -- 172.21.15.1:0/3898548784 >> 172.21.15.1:0/3898548784 conn(0x7f2490073b70 msgr2=0x7f2490073f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:42.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.966+0000 7f2495f9f640 1 -- 172.21.15.1:0/3898548784 shutdown_connections 2024-05-09T05:11:42.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.966+0000 7f2495f9f640 1 -- 172.21.15.1:0/3898548784 wait complete. 2024-05-09T05:11:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.968+0000 7f2495f9f640 1 Processor -- start 2024-05-09T05:11:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.968+0000 7f2495f9f640 1 -- start start 2024-05-09T05:11:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2495f9f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24900ccea0 0x7f24901f6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2495f9f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24901e67b0 0x7f24900ca400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2495f9f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f249014a720 0x7f249014ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2495f9f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f24900c8a10 con 0x7f24900ccea0 2024-05-09T05:11:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2495f9f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2490160110 con 0x7f24901e67b0 2024-05-09T05:11:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2495f9f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2490156be0 con 0x7f249014a720 2024-05-09T05:11:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2494f9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24900ccea0 0x7f24901f6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2494f9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24900ccea0 0x7f24901f6910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38724/0 (socket says 172.21.15.1:38724) 2024-05-09T05:11:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2494f9d640 1 -- 172.21.15.1:0/657374222 learned_addr learned my addr 172.21.15.1:0/657374222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f248ffff640 1 -- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24901e67b0 msgr2=0x7f24900ca400 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f248ffff640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24901e67b0 0x7f24900ca400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f249579e640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f249014a720 0x7f249014ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2494f9d640 1 -- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f249014a720 msgr2=0x7f249014ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2494f9d640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f249014a720 0x7f249014ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2494f9d640 1 -- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24901e67b0 msgr2=0x7f24900ca400 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2494f9d640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24901e67b0 0x7f24900ca400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f2494f9d640 1 -- 172.21.15.1:0/657374222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24901e5f70 con 0x7f24900ccea0 2024-05-09T05:11:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.970+0000 7f249579e640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f249014a720 0x7f249014ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:11:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.971+0000 7f2494f9d640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24900ccea0 0x7f24901f6910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f248400b580 tx=0x7f248400ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.971+0000 7f248d7fa640 1 -- 172.21.15.1:0/657374222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2484013020 con 0x7f24900ccea0 2024-05-09T05:11:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.971+0000 7f248d7fa640 1 -- 172.21.15.1:0/657374222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2484003de0 con 0x7f24900ccea0 2024-05-09T05:11:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.971+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24901e5430 con 0x7f24900ccea0 2024-05-09T05:11:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.971+0000 7f248d7fa640 1 -- 172.21.15.1:0/657374222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24840052b0 con 0x7f24900ccea0 2024-05-09T05:11:42.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.971+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24901e56d0 con 0x7f24900ccea0 2024-05-09T05:11:42.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.972+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f24900c8730 con 0x7f24900ccea0 2024-05-09T05:11:42.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.973+0000 7f248d7fa640 1 -- 172.21.15.1:0/657374222 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2484016020 con 0x7f24900ccea0 2024-05-09T05:11:42.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.974+0000 7f248d7fa640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2474076280 0x7f2474078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:42.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.974+0000 7f248d7fa640 1 -- 172.21.15.1:0/657374222 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2484097420 con 0x7f24900ccea0 2024-05-09T05:11:42.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.974+0000 7f248ffff640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2474076280 0x7f2474078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:42.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.975+0000 7f248ffff640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2474076280 0x7f2474078790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24900ca1b0 tx=0x7f247c001e70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:42.980+0000 7f248d7fa640 1 -- 172.21.15.1:0/657374222 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f24900c8730 con 0x7f24900ccea0 2024-05-09T05:11:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.288+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f24900703c0 con 0x7f24900ccea0 2024-05-09T05:11:43.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.289+0000 7f248d7fa640 1 -- 172.21.15.1:0/657374222 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f248401d020 con 0x7f24900ccea0 2024-05-09T05:11:43.290 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:11:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2474076280 msgr2=0x7f2474078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2474076280 0x7f2474078790 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f24900ca1b0 tx=0x7f247c001e70 comp rx=0 tx=0).stop 2024-05-09T05:11:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24900ccea0 msgr2=0x7f24901f6910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24900ccea0 0x7f24901f6910 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f248400b580 tx=0x7f248400ba40 comp rx=0 tx=0).stop 2024-05-09T05:11:43.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 shutdown_connections 2024-05-09T05:11:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2474076280 0x7f2474078790 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f24901e67b0 0x7f24900ca400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f249014a720 0x7f249014ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 --2- 172.21.15.1:0/657374222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f24900ccea0 0x7f24901f6910 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 >> 172.21.15.1:0/657374222 conn(0x7f2490073b70 msgr2=0x7f249014d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 shutdown_connections 2024-05-09T05:11:43.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:43.291+0000 7f2495f9f640 1 -- 172.21.15.1:0/657374222 wait complete. 2024-05-09T05:11:46.304 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:11:46.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.425+0000 7f0fb8957640 1 Processor -- start 2024-05-09T05:11:46.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.425+0000 7f0fb8957640 1 -- start start 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb8957640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fb4068a50 0x7f0fb40cd350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb8957640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0fb40c77a0 0x7f0fb40c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb8957640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0fb406d8a0 con 0x7f0fb406c6f0 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb8957640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f0fb415a680 con 0x7f0fb4068a50 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb8957640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0fb406db40 con 0x7f0fb40c77a0 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb2d9d640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f0fb40c77a0 msgr2=0x7f0fb40c7bb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fb4068a50 0x7f0fb40cd350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb2d9d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0fb40c77a0 0x7f0fb40c7bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0fb4068a50 0x7f0fb40cd350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38736/0 (socket says 172.21.15.1:38736) 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb259c640 1 -- 172.21.15.1:0/2443948389 learned_addr learned my addr 172.21.15.1:0/2443948389 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fabfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f0fb406c6f0 0x7f0fb4151070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:44390/0 (socket says 172.21.15.1:44390) 2024-05-09T05:11:46.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb259c640 1 -- 172.21.15.1:0/2443948389 >> v2:172.21.15.196:3300/0 conn(0x7f0fb40c77a0 msgr2=0x7f0fb40c7bb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb259c640 1 --2- 172.21.15.1:0/2443948389 >> v2:172.21.15.196:3300/0 conn(0x7f0fb40c77a0 0x7f0fb40c7bb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb259c640 1 -- 172.21.15.1:0/2443948389 >> v1:172.21.15.1:6789/0 conn(0x7f0fb406c6f0 legacy=0x7f0fb4151070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb259c640 1 -- 172.21.15.1:0/2443948389 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb406de20 con 0x7f0fb4068a50 2024-05-09T05:11:46.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.427+0000 7f0fb1d9b640 1 -- 172.21.15.1:0/2443948389 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4272183082 0 0) 0x7f0fb406d8a0 con 0x7f0fb406c6f0 2024-05-09T05:11:46.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.428+0000 7f0fb259c640 1 --2- 172.21.15.1:0/2443948389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb4068a50 0x7f0fb40cd350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4074d40 tx=0x7f0fa402a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52936a444dff209a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:46.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.429+0000 7f0fb1d9b640 1 -- 172.21.15.1:0/2443948389 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4037070 con 0x7f0fb4068a50 2024-05-09T05:11:46.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.429+0000 7f0fb1d9b640 1 -- 172.21.15.1:0/2443948389 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fa4032440 con 0x7f0fb4068a50 2024-05-09T05:11:46.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.429+0000 7f0fb1d9b640 1 -- 172.21.15.1:0/2443948389 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fa4032810 con 0x7f0fb4068a50 2024-05-09T05:11:46.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.429+0000 7f0fb8957640 1 -- 172.21.15.1:0/2443948389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb4068a50 msgr2=0x7f0fb40cd350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:46.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.429+0000 7f0fb8957640 1 --2- 172.21.15.1:0/2443948389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb4068a50 0x7f0fb40cd350 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f0fb4074d40 tx=0x7f0fa402a210 comp rx=0 tx=0).stop 2024-05-09T05:11:46.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.429+0000 7f0fb8957640 1 -- 172.21.15.1:0/2443948389 shutdown_connections 2024-05-09T05:11:46.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.429+0000 7f0fb8957640 1 --2- 172.21.15.1:0/2443948389 >> v2:172.21.15.196:3300/0 conn(0x7f0fb40c77a0 0x7f0fb40c7bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.430+0000 7f0fb8957640 1 --2- 172.21.15.1:0/2443948389 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb4068a50 0x7f0fb40cd350 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.430+0000 7f0fb8957640 1 -- 172.21.15.1:0/2443948389 >> 172.21.15.1:0/2443948389 conn(0x7f0fb40ce110 msgr2=0x7f0fb40ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:46.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.430+0000 7f0fb8957640 1 -- 172.21.15.1:0/2443948389 shutdown_connections 2024-05-09T05:11:46.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.430+0000 7f0fb8957640 1 -- 172.21.15.1:0/2443948389 wait complete. 2024-05-09T05:11:46.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.432+0000 7f0fb8957640 1 Processor -- start 2024-05-09T05:11:46.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.432+0000 7f0fb8957640 1 -- start start 2024-05-09T05:11:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb8957640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fb4068a50 0x7f0fb4155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb8957640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0fb406c6f0 0x7f0fb4157540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb8957640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb40c77a0 0x7f0fb4157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb8957640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb40cc4b0 con 0x7f0fb40c77a0 2024-05-09T05:11:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb8957640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0fb40cc1d0 con 0x7f0fb4068a50 2024-05-09T05:11:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb8957640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0fb415ab80 con 0x7f0fb406c6f0 2024-05-09T05:11:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb40c77a0 0x7f0fb4157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:46.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb259c640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fb4068a50 msgr2=0x7f0fb4155210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb259c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fb4068a50 0x7f0fb4155210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb40c77a0 0x7f0fb4157a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:38744/0 (socket says 172.21.15.1:38744) 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fb2d9d640 1 -- 172.21.15.1:0/3503751506 learned_addr learned my addr 172.21.15.1:0/3503751506 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.433+0000 7f0fabfff640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0fb406c6f0 0x7f0fb4157540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0fb2d9d640 1 -- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0fb406c6f0 msgr2=0x7f0fb4157540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0fb2d9d640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0fb406c6f0 0x7f0fb4157540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0fb2d9d640 1 -- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fb4068a50 msgr2=0x7f0fb4155210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0fb2d9d640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fb4068a50 0x7f0fb4155210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0fb2d9d640 1 -- 172.21.15.1:0/3503751506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0fb4155750 con 0x7f0fb40c77a0 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0fb2d9d640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb40c77a0 0x7f0fb4157a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00be50 tx=0x7f0fac00bf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0faaffd640 1 -- 172.21.15.1:0/3503751506 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac016430 con 0x7f0fb40c77a0 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0fb4157fc0 con 0x7f0fb40c77a0 2024-05-09T05:11:46.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0faaffd640 1 -- 172.21.15.1:0/3503751506 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0fac010030 con 0x7f0fb40c77a0 2024-05-09T05:11:46.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0faaffd640 1 -- 172.21.15.1:0/3503751506 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0fac016830 con 0x7f0fb40c77a0 2024-05-09T05:11:46.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.434+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0fb415bea0 con 0x7f0fb40c77a0 2024-05-09T05:11:46.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.436+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0fb4158b90 con 0x7f0fb40c77a0 2024-05-09T05:11:46.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.436+0000 7f0faaffd640 1 -- 172.21.15.1:0/3503751506 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0fac0060d0 con 0x7f0fb40c77a0 2024-05-09T05:11:46.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.437+0000 7f0faaffd640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f94076280 0x7f0f94078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:46.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.437+0000 7f0faaffd640 1 -- 172.21.15.1:0/3503751506 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0fac097b10 con 0x7f0fb40c77a0 2024-05-09T05:11:46.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.437+0000 7f0fb259c640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f94076280 0x7f0f94078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:46.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.438+0000 7f0fb259c640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f94076280 0x7f0f94078790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4034000 tx=0x7f0fa402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:46.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.442+0000 7f0faaffd640 1 -- 172.21.15.1:0/3503751506 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0fac060c90 con 0x7f0fb40c77a0 2024-05-09T05:11:46.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.768+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0fb406cdd0 con 0x7f0fb40c77a0 2024-05-09T05:11:46.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.768+0000 7f0faaffd640 1 -- 172.21.15.1:0/3503751506 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0fac064f90 con 0x7f0fb40c77a0 2024-05-09T05:11:46.769 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":50,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:11:46.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f94076280 msgr2=0x7f0f94078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:46.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f94076280 0x7f0f94078790 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f0fa4034000 tx=0x7f0fa402e000 comp rx=0 tx=0).stop 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb40c77a0 msgr2=0x7f0fb4157a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb40c77a0 0x7f0fb4157a80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f0fac00be50 tx=0x7f0fac00bf40 comp rx=0 tx=0).stop 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 shutdown_connections 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f94076280 0x7f0f94078790 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0fb40c77a0 0x7f0fb4157a80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0fb406c6f0 0x7f0fb4157540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 --2- 172.21.15.1:0/3503751506 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0fb4068a50 0x7f0fb4155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 >> 172.21.15.1:0/3503751506 conn(0x7f0fb40ce110 msgr2=0x7f0fb40ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 shutdown_connections 2024-05-09T05:11:46.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:46.771+0000 7f0fb8957640 1 -- 172.21.15.1:0/3503751506 wait complete. 2024-05-09T05:11:49.783 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:11:49.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.913+0000 7fbb4f59e640 1 Processor -- start 2024-05-09T05:11:49.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.913+0000 7fbb4f59e640 1 -- start start 2024-05-09T05:11:49.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb4f59e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 0x7fbb500af940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb4f59e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb50073f90 con 0x7fbb5015c860 2024-05-09T05:11:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb4f59e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb500733c0 con 0x7fbb50156860 2024-05-09T05:11:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb4f59e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fbb50156c70 con 0x7fbb500af530 2024-05-09T05:11:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb4ed9d640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 msgr2=0x7fbb500af940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb47fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbb5015c860 0x7fbb5015cc70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46286/0 (socket says 172.21.15.1:46286) 2024-05-09T05:11:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 learned_addr learned my addr 172.21.15.1:0/2500420392 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb4e59c640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fbb50156860 legacy=0x7fbb500c9d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:49.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:49.915+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 0x7fbb500af940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:50.116+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 msgr2=0x7fbb500af940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:50.116+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> v1:172.21.15.196:6789/0 conn(0x7fbb50156860 legacy=0x7fbb500c9d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:50.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:50.116+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 0x7fbb500af940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:11:50.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:50.517+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> v1:172.21.15.196:6789/0 conn(0x7fbb50156860 legacy=0x7fbb500c9d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:50.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:50.517+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 msgr2=0x7fbb500af940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:50.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:50.517+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 0x7fbb500af940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:11:51.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:51.318+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> v1:172.21.15.196:6789/0 conn(0x7fbb50156860 legacy=0x7fbb500c9d40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:51.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:51.318+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 msgr2=0x7fbb500af940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:51.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:51.318+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 0x7fbb500af940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:11:52.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 msgr2=0x7fbb500af940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4cd99640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb500af530 0x7fbb500af940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 >> v1:172.21.15.196:6789/0 conn(0x7fbb50156860 legacy=0x7fbb500c9d40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 >> v1:172.21.15.1:6789/0 conn(0x7fbb5015c860 legacy=0x7fbb5015cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4cd99640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4cd99640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbb30008d10 con 0x7fbb300061a0 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbb30008ec0 con 0x7fbb300036c0 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:52.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.915+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:52.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.916+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:52.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.916+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:52.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.916+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:52.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:52.916+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:53.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.116+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:53.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.116+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:53.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.116+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:53.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.116+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:53.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.116+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:11:53.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.117+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:53.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.117+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:53.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.117+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:53.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.117+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:53.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.117+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:11:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.517+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.517+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.517+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.517+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.517+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:11:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.518+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.518+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.518+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.518+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:53.518+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:11:54.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.318+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:54.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.318+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:54.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.318+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:54.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.318+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:54.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.318+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:11:54.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.319+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:54.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.319+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:11:54.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.319+0000 7fbb4e59c640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:54.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.319+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:54.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:54.319+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 msgr2=0x7fbb30005b60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb300036c0 0x7fbb30005b60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 msgr2=0x7fbb30008690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb300061a0 0x7fbb30008690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 reap_dead start 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 0x7fbb3000d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbb50073660 con 0x7fbb30003b30 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbb30008d10 con 0x7fbb30007cd0 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4cd99640 1 -- 172.21.15.1:0/2500420392 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fbb30008ec0 con 0x7fbb3000b310 2024-05-09T05:11:55.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 msgr2=0x7fbb3000d730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.915+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 0x7fbb3000d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.916+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 msgr2=0x7fbb300080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:11:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.916+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 msgr2=0x7fbb300080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.916+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:55.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:55.916+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:56.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.117+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 msgr2=0x7fbb3000d730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:56.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.117+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:56.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.117+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 0x7fbb3000d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:11:56.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.117+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 msgr2=0x7fbb300080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:11:56.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.117+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 msgr2=0x7fbb300080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:56.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.117+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:56.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.117+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:11:56.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.518+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 msgr2=0x7fbb3000d730 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:56.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.518+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:56.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.518+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 0x7fbb3000d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:11:56.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.518+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 msgr2=0x7fbb300080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:11:56.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.518+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 msgr2=0x7fbb300080e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:11:56.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.518+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:11:56.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.518+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:11:56.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.931+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3770288370 0 0) 0x7fbb50073660 con 0x7fbb30003b30 2024-05-09T05:11:56.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.931+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fbb3c003520 con 0x7fbb30003b30 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3692571592 0 0) 0x7fbb3c003520 con 0x7fbb30003b30 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 msgr2=0x7fbb3000d730 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.196:3300/0 conn(0x7fbb3000b310 0x7fbb3000d730 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 msgr2=0x7fbb300080e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 --2- 172.21.15.1:0/2500420392 >> v2:172.21.15.1:3300/0 conn(0x7fbb30007cd0 0x7fbb300080e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb500733c0 con 0x7fbb30003b30 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fbb40009850 con 0x7fbb30003b30 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 >> v1:172.21.15.1:6789/0 conn(0x7fbb30003b30 legacy=0x7fbb300065b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb3c003a20 0x7fbb3c003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c004800 0x7fbb3c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb3c00ba20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30008d10 con 0x7fbb3c0053a0 2024-05-09T05:11:56.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbb30008ec0 con 0x7fbb3c004800 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbb3c00c380 con 0x7fbb3c003a20 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb47fff640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c004800 msgr2=0x7fbb3c004c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb3c00ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb47fff640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c004800 0x7fbb3c004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.938+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb3c003a20 0x7fbb3c003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.939+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb3c003a20 msgr2=0x7fbb3c003e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.939+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb3c003a20 0x7fbb3c003e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.939+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c004800 msgr2=0x7fbb3c004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.939+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb3c004800 0x7fbb3c004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.939+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb3c003520 con 0x7fbb3c0053a0 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.939+0000 7fbb4ed9d640 1 -- 172.21.15.1:0/2500420392 reap_dead start 2024-05-09T05:11:56.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.939+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb3c00ba20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbb500732b0 tx=0x7fbb3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bdbfae566fa17854 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:56.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.941+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480079a0 con 0x7fbb3c0053a0 2024-05-09T05:11:56.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.941+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb4802e070 con 0x7fbb3c0053a0 2024-05-09T05:11:56.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.941+0000 7fbb4dd9b640 1 -- 172.21.15.1:0/2500420392 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb480336b0 con 0x7fbb3c0053a0 2024-05-09T05:11:56.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.942+0000 7fbb4f59e640 1 -- 172.21.15.1:0/2500420392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 msgr2=0x7fbb3c00ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:56.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.942+0000 7fbb4f59e640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb3c00ba20 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbb500732b0 tx=0x7fbb3c005830 comp rx=0 tx=0).stop 2024-05-09T05:11:56.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.942+0000 7fbb4f59e640 1 -- 172.21.15.1:0/2500420392 shutdown_connections 2024-05-09T05:11:56.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.942+0000 7fbb4f59e640 1 --2- 172.21.15.1:0/2500420392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb3c00ba20 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:56.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.942+0000 7fbb4f59e640 1 -- 172.21.15.1:0/2500420392 >> 172.21.15.1:0/2500420392 conn(0x7fbb500ca860 msgr2=0x7fbb500cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:56.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.942+0000 7fbb4f59e640 1 -- 172.21.15.1:0/2500420392 shutdown_connections 2024-05-09T05:11:56.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.942+0000 7fbb4f59e640 1 -- 172.21.15.1:0/2500420392 wait complete. 2024-05-09T05:11:56.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.945+0000 7fbb4f59e640 1 Processor -- start 2024-05-09T05:11:56.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.945+0000 7fbb4f59e640 1 -- start start 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb500c7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4f59e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb5015b730 0x7fbb501f56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4f59e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb50154530 0x7fbb500b1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4f59e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbb500ca510 con 0x7fbb3c0053a0 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4f59e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbb50073c20 con 0x7fbb50154530 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4f59e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbb5006de50 con 0x7fbb5015b730 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb500c7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4ed9d640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb50154530 msgr2=0x7fbb500b1ad0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:56.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb500c7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57570/0 (socket says 172.21.15.1:57570) 2024-05-09T05:11:56.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4e59c640 1 -- 172.21.15.1:0/1354963458 learned_addr learned my addr 172.21.15.1:0/1354963458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:56.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.946+0000 7fbb4ed9d640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb50154530 0x7fbb500b1ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:56.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb47fff640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb5015b730 0x7fbb501f56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:56.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb4e59c640 1 -- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb5015b730 msgr2=0x7fbb501f56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:56.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb5015b730 0x7fbb501f56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:56.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb4e59c640 1 -- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb50154530 msgr2=0x7fbb500b1ad0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:56.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb50154530 0x7fbb500b1ad0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:56.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb4e59c640 1 -- 172.21.15.1:0/1354963458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb50153400 con 0x7fbb3c0053a0 2024-05-09T05:11:56.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb4e59c640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb500c7ef0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000bd60 tx=0x7fbb40005480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:56.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb46ffd640 1 -- 172.21.15.1:0/1354963458 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb4001c070 con 0x7fbb3c0053a0 2024-05-09T05:11:56.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb46ffd640 1 -- 172.21.15.1:0/1354963458 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbb40018650 con 0x7fbb3c0053a0 2024-05-09T05:11:56.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb46ffd640 1 -- 172.21.15.1:0/1354963458 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb40018a40 con 0x7fbb3c0053a0 2024-05-09T05:11:56.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb5015bee0 con 0x7fbb3c0053a0 2024-05-09T05:11:56.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.947+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb50153d30 con 0x7fbb3c0053a0 2024-05-09T05:11:56.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.948+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb50076f80 con 0x7fbb3c0053a0 2024-05-09T05:11:56.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.949+0000 7fbb46ffd640 1 -- 172.21.15.1:0/1354963458 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbb4001f070 con 0x7fbb3c0053a0 2024-05-09T05:11:56.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.950+0000 7fbb46ffd640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb300770a0 0x7fbb300795b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:56.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.950+0000 7fbb46ffd640 1 -- 172.21.15.1:0/1354963458 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbb40098bb0 con 0x7fbb3c0053a0 2024-05-09T05:11:56.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.950+0000 7fbb47fff640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb300770a0 0x7fbb300795b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:56.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.951+0000 7fbb47fff640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb300770a0 0x7fbb300795b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbb30006070 tx=0x7fbb3801c040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:56.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:56.955+0000 7fbb46ffd640 1 -- 172.21.15.1:0/1354963458 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbb40012040 con 0x7fbb3c0053a0 2024-05-09T05:11:57.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.278+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbb500b16d0 con 0x7fbb3c0053a0 2024-05-09T05:11:57.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.279+0000 7fbb46ffd640 1 -- 172.21.15.1:0/1354963458 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fbb40061da0 con 0x7fbb3c0053a0 2024-05-09T05:11:57.280 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb300770a0 msgr2=0x7fbb300795b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb300770a0 0x7fbb300795b0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fbb30006070 tx=0x7fbb3801c040 comp rx=0 tx=0).stop 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 msgr2=0x7fbb500c7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb500c7ef0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbb4000bd60 tx=0x7fbb40005480 comp rx=0 tx=0).stop 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 shutdown_connections 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbb300770a0 0x7fbb300795b0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbb50154530 0x7fbb500b1ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbb5015b730 0x7fbb501f56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 --2- 172.21.15.1:0/1354963458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbb3c0053a0 0x7fbb500c7ef0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 >> 172.21.15.1:0/1354963458 conn(0x7fbb500ca860 msgr2=0x7fbb500af940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 shutdown_connections 2024-05-09T05:11:57.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.281+0000 7fbb4f59e640 1 -- 172.21.15.1:0/1354963458 wait complete. 2024-05-09T05:11:57.294 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:11:57.294 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:11:57.295 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:11:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.420+0000 7f9aa83d5640 1 Processor -- start 2024-05-09T05:11:57.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.420+0000 7f9aa83d5640 1 -- start start 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.421+0000 7f9aa83d5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa00b4ec0 0x7f9aa00cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.421+0000 7f9aa83d5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa015c7b0 con 0x7f9aa00ce010 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa83d5640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9aa00cca50 con 0x7f9aa0160460 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa83d5640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa015d0c0 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa5970640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9aa00ce010 0x7f9aa015fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:46304/0 (socket says 172.21.15.1:46304) 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa5970640 1 -- 172.21.15.1:0/2807791332 learned_addr learned my addr 172.21.15.1:0/2807791332 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa6972640 1 -- 172.21.15.1:0/2807791332 >> v1:172.21.15.196:6789/0 conn(0x7f9aa0160460 legacy=0x7f9aa0167130 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa6171640 1 --2- 172.21.15.1:0/2807791332 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa00b4ec0 0x7f9aa00cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa516f640 1 -- 172.21.15.1:0/2807791332 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4244444013 0 0) 0x7f9aa015c7b0 con 0x7f9aa00ce010 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa516f640 1 -- 172.21.15.1:0/2807791332 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a8c003520 con 0x7f9aa00ce010 2024-05-09T05:11:57.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa6171640 1 -- 172.21.15.1:0/2807791332 >> v1:172.21.15.196:6789/0 conn(0x7f9aa0160460 legacy=0x7f9aa0167130 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa6171640 1 -- 172.21.15.1:0/2807791332 >> v1:172.21.15.1:6789/0 conn(0x7f9aa00ce010 legacy=0x7f9aa015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa516f640 1 -- 172.21.15.1:0/2807791332 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1155478969 0 0) 0x7f9a8c003520 con 0x7f9aa00ce010 2024-05-09T05:11:57.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.422+0000 7f9aa6171640 1 -- 172.21.15.1:0/2807791332 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa00ccd30 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.423+0000 7f9aa6171640 1 --2- 172.21.15.1:0/2807791332 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa00b4ec0 0x7f9aa00cf9d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0097d0 tx=0x7f9a9c02a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a88ed736d16dd357 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.423+0000 7f9aa516f640 1 -- 172.21.15.1:0/2807791332 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c037070 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.423+0000 7f9aa516f640 1 -- 172.21.15.1:0/2807791332 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9c02ac10 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.423+0000 7f9aa516f640 1 -- 172.21.15.1:0/2807791332 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a900016f0 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.424+0000 7f9aa83d5640 1 -- 172.21.15.1:0/2807791332 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa00b4ec0 msgr2=0x7f9aa00cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.424+0000 7f9aa83d5640 1 --2- 172.21.15.1:0/2807791332 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa00b4ec0 0x7f9aa00cf9d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0097d0 tx=0x7f9a9c02a180 comp rx=0 tx=0).stop 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.424+0000 7f9aa83d5640 1 -- 172.21.15.1:0/2807791332 shutdown_connections 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.424+0000 7f9aa83d5640 1 --2- 172.21.15.1:0/2807791332 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa00b4ec0 0x7f9aa00cf9d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.424+0000 7f9aa83d5640 1 -- 172.21.15.1:0/2807791332 >> 172.21.15.1:0/2807791332 conn(0x7f9aa00d2360 msgr2=0x7f9aa00ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:57.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.424+0000 7f9aa83d5640 1 -- 172.21.15.1:0/2807791332 shutdown_connections 2024-05-09T05:11:57.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.424+0000 7f9aa83d5640 1 -- 172.21.15.1:0/2807791332 wait complete. 2024-05-09T05:11:57.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.426+0000 7f9aa83d5640 1 Processor -- start 2024-05-09T05:11:57.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.426+0000 7f9aa83d5640 1 -- start start 2024-05-09T05:11:57.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa83d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00b4ec0 0x7f9aa01598f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa83d5640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9aa00ce010 0x7f9aa00c9d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa83d5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa0160460 0x7f9aa0153440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa83d5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa00cc750 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa83d5640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa015cd50 con 0x7f9aa00ce010 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa83d5640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9aa00cff10 con 0x7f9aa0160460 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa6171640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00b4ec0 0x7f9aa01598f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa5970640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9aa00ce010 msgr2=0x7f9aa00c9d30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa6171640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00b4ec0 0x7f9aa01598f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57582/0 (socket says 172.21.15.1:57582) 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.427+0000 7f9aa5970640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9aa00ce010 0x7f9aa00c9d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa6171640 1 -- 172.21.15.1:0/1156741170 learned_addr learned my addr 172.21.15.1:0/1156741170 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa6972640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa0160460 0x7f9aa0153440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa6171640 1 -- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa0160460 msgr2=0x7f9aa0153440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa6171640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9aa0160460 0x7f9aa0153440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa6171640 1 -- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9aa00ce010 msgr2=0x7f9aa00c9d30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:57.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa6171640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9aa00ce010 0x7f9aa00c9d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa6171640 1 -- 172.21.15.1:0/1156741170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9aa00ca270 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa6171640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00b4ec0 0x7f9aa01598f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0097d0 tx=0x7f9a9c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c042070 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a9c03e460 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9aa00ca9b0 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a9c037040 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.428+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9aa01f3460 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.429+0000 7f9aa83d5640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c8f70 0x7f9aa00cac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.429+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa00ca740 con 0x7f9aa00c8f70 2024-05-09T05:11:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.430+0000 7f9aa5970640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c8f70 0x7f9aa00cac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.430+0000 7f9aa5970640 1 -- 172.21.15.1:0/1156741170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9aa0050870 con 0x7f9aa00c8f70 2024-05-09T05:11:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.430+0000 7f9aa5970640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c8f70 0x7f9aa00cac20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000c680 tx=0x7f9a9000cb40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.430+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90017070 con 0x7f9aa00c8f70 2024-05-09T05:11:57.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.431+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f9a9000ce80 con 0x7f9aa00c8f70 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.431+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c8f70 msgr2=0x7f9aa00cac20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.431+0000 7f9a96ffd640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c8f70 0x7f9aa00cac20 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000c680 tx=0x7f9a9000cb40 comp rx=0 tx=0).stop 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.431+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c8f70 msgr2=0x7f9aa00cac20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.431+0000 7f9a96ffd640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c8f70 0x7f9aa00cac20 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.431+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9a9c045070 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.432+0000 7f9a96ffd640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a84075db0 0x7f9a840782c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.432+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9a9c0b77b0 con 0x7f9aa00b4ec0 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.432+0000 7f9aa5970640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a84075db0 0x7f9a840782c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.432+0000 7f9aa5970640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a84075db0 0x7f9a840782c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000a480 tx=0x7f9a90011000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.438+0000 7f9aa83d5640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c9380 0x7f9aa00cb160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.438+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9aa0156320 con 0x7f9aa00c9380 2024-05-09T05:11:57.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.438+0000 7f9aa6972640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c9380 0x7f9aa00cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9aa6972640 1 -- 172.21.15.1:0/1156741170 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f9aa00cd2b0 con 0x7f9aa00c9380 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9aa6972640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c9380 0x7f9aa00cb160 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00b6470 tx=0x7f9a98007660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a980054e0 con 0x7f9aa00c9380 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4882 (secure 0 0 0) 0x7f9a98007e20 con 0x7f9aa00c9380 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c9380 msgr2=0x7f9aa00cb160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9a96ffd640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c9380 0x7f9aa00cb160 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00b6470 tx=0x7f9a98007660 comp rx=0 tx=0).stop 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9a96ffd640 1 -- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c9380 msgr2=0x7f9aa00cb160 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9a96ffd640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00c9380 0x7f9aa00cb160 secure :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f9aa00b6470 tx=0x7f9a98007660 comp rx=0 tx=0).stop 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:57.440 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 54, 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 106033, 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:57.441 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:57.442 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:57.443 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:57.444 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:57.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:57.446 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:57.448 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:57.449 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.450 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.451 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:57.452 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:57.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a84075db0 msgr2=0x7f9a840782c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9aa83d5640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a84075db0 0x7f9a840782c0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f9a9000a480 tx=0x7f9a90011000 comp rx=0 tx=0).stop 2024-05-09T05:11:57.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00b4ec0 msgr2=0x7f9aa01598f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9aa6972640 1 -- 172.21.15.1:0/1156741170 reap_dead start 2024-05-09T05:11:57.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.439+0000 7f9aa83d5640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00b4ec0 0x7f9aa01598f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f9a9c0097d0 tx=0x7f9a9c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.440+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 shutdown_connections 2024-05-09T05:11:57.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.440+0000 7f9aa83d5640 1 --2- 172.21.15.1:0/1156741170 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9aa00b4ec0 0x7f9aa01598f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.440+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 >> 172.21.15.1:0/1156741170 conn(0x7f9aa00d2360 msgr2=0x7f9aa006d9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:57.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.440+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 shutdown_connections 2024-05-09T05:11:57.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.440+0000 7f9aa83d5640 1 -- 172.21.15.1:0/1156741170 wait complete. 2024-05-09T05:11:57.454 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:11:57.454 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:11:57.454 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:11:57.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.580+0000 7f632f66a640 1 Processor -- start 2024-05-09T05:11:57.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.580+0000 7f632f66a640 1 -- start start 2024-05-09T05:11:57.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632f66a640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f63280c5600 0x7f632814ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632f66a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63280cc1f0 0x7f63280cc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632f66a640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6328153ba0 con 0x7f63280c8210 2024-05-09T05:11:57.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632f66a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6328159980 con 0x7f63280cc1f0 2024-05-09T05:11:57.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632f66a640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f63280c5a10 con 0x7f63280c5600 2024-05-09T05:11:57.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632cc05640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f63280c5600 msgr2=0x7f632814ff20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632d406640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f63280c8210 legacy=0x7f6328159e10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632dc07640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63280cc1f0 0x7f63280cc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632cc05640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f63280c5600 0x7f632814ff20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632dc07640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f63280cc1f0 0x7f63280cc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57600/0 (socket says 172.21.15.1:57600) 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.582+0000 7f632dc07640 1 -- 172.21.15.1:0/3637323250 learned_addr learned my addr 172.21.15.1:0/3637323250 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.583+0000 7f632dc07640 1 -- 172.21.15.1:0/3637323250 >> v2:172.21.15.196:3300/0 conn(0x7f63280c5600 msgr2=0x7f632814ff20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.583+0000 7f632dc07640 1 --2- 172.21.15.1:0/3637323250 >> v2:172.21.15.196:3300/0 conn(0x7f63280c5600 0x7f632814ff20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.583+0000 7f632dc07640 1 -- 172.21.15.1:0/3637323250 >> v1:172.21.15.196:6789/0 conn(0x7f63280c8210 legacy=0x7f6328159e10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:11:57.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.583+0000 7f632dc07640 1 -- 172.21.15.1:0/3637323250 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63280c62b0 con 0x7f63280cc1f0 2024-05-09T05:11:57.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.584+0000 7f632dc07640 1 --2- 172.21.15.1:0/3637323250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc1f0 0x7f63280cc600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6328159b00 tx=0x7f632002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7f0817a7e0230a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.584+0000 7f631ffff640 1 -- 172.21.15.1:0/3637323250 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6320036070 con 0x7f63280cc1f0 2024-05-09T05:11:57.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.584+0000 7f631ffff640 1 -- 172.21.15.1:0/3637323250 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6320031600 con 0x7f63280cc1f0 2024-05-09T05:11:57.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.584+0000 7f631ffff640 1 -- 172.21.15.1:0/3637323250 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63200319d0 con 0x7f63280cc1f0 2024-05-09T05:11:57.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.585+0000 7f632f66a640 1 -- 172.21.15.1:0/3637323250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc1f0 msgr2=0x7f63280cc600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.585+0000 7f632f66a640 1 --2- 172.21.15.1:0/3637323250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc1f0 0x7f63280cc600 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f6328159b00 tx=0x7f632002a6a0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.585+0000 7f632f66a640 1 -- 172.21.15.1:0/3637323250 shutdown_connections 2024-05-09T05:11:57.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.585+0000 7f632f66a640 1 --2- 172.21.15.1:0/3637323250 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280cc1f0 0x7f63280cc600 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.585+0000 7f632f66a640 1 --2- 172.21.15.1:0/3637323250 >> v2:172.21.15.196:3300/0 conn(0x7f63280c5600 0x7f632814ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.585+0000 7f632f66a640 1 -- 172.21.15.1:0/3637323250 >> 172.21.15.1:0/3637323250 conn(0x7f63280ce180 msgr2=0x7f63280ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:57.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.585+0000 7f632f66a640 1 -- 172.21.15.1:0/3637323250 shutdown_connections 2024-05-09T05:11:57.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.585+0000 7f632f66a640 1 -- 172.21.15.1:0/3637323250 wait complete. 2024-05-09T05:11:57.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.587+0000 7f632f66a640 1 Processor -- start 2024-05-09T05:11:57.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.587+0000 7f632f66a640 1 -- start start 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632f66a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280c5600 0x7f63280c9500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632f66a640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63280c8210 0x7f632815d120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632f66a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280cc1f0 0x7f632815c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632f66a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6328159610 con 0x7f63280c5600 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632f66a640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f63280c6870 con 0x7f63280c8210 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632f66a640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f63280c5f50 con 0x7f63280cc1f0 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632d406640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280c5600 0x7f63280c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632d406640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280c5600 0x7f63280c9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57612/0 (socket says 172.21.15.1:57612) 2024-05-09T05:11:57.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632d406640 1 -- 172.21.15.1:0/1888177645 learned_addr learned my addr 172.21.15.1:0/1888177645 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632cc05640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63280c8210 msgr2=0x7f632815d120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632cc05640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63280c8210 0x7f632815d120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632dc07640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280cc1f0 0x7f632815c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632d406640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280cc1f0 msgr2=0x7f632815c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632d406640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280cc1f0 0x7f632815c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632d406640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63280c8210 msgr2=0x7f632815d120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632d406640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f63280c8210 0x7f632815d120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.588+0000 7f632d406640 1 -- 172.21.15.1:0/1888177645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63280c9a40 con 0x7f63280c5600 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.589+0000 7f632d406640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280c5600 0x7f63280c9500 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f632400b550 tx=0x7f632400ba10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.589+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324003af0 con 0x7f63280c5600 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.589+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6324015070 con 0x7f63280c5600 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.589+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63280ca940 con 0x7f63280c5600 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.589+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6324005070 con 0x7f63280c5600 2024-05-09T05:11:57.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.589+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63280ca660 con 0x7f63280c5600 2024-05-09T05:11:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.590+0000 7f632f66a640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280c87f0 0x7f63281f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.590+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f63280c8ed0 con 0x7f63280c87f0 2024-05-09T05:11:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.591+0000 7f632cc05640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280c87f0 0x7f63281f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.591+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6324005210 con 0x7f63280c5600 2024-05-09T05:11:57.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.591+0000 7f632cc05640 1 -- 172.21.15.1:0/1888177645 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6328050870 con 0x7f63280c87f0 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.591+0000 7f632cc05640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280c87f0 0x7f63281f5e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6328147c00 tx=0x7f631800b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f631dffb640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f630c076280 0x7f630c078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6324096ff0 con 0x7f63280c5600 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f632dc07640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f630c076280 0x7f630c078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6318003e10 con 0x7f63280c87f0 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6318015070 con 0x7f63280c87f0 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280c87f0 msgr2=0x7f63281f5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f631dffb640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280c87f0 0x7f63281f5e50 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6328147c00 tx=0x7f631800b8c0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280c87f0 msgr2=0x7f63281f5e50 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.592+0000 7f631dffb640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63280c87f0 0x7f63281f5e50 secure :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6328147c00 tx=0x7f631800b8c0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.593+0000 7f632dc07640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f630c076280 0x7f630c078790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6320033000 tx=0x7f6320009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.600+0000 7f632f66a640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63281f6390 0x7f63281f67a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:11:57.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.600+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6328146d30 con 0x7f63281f6390 2024-05-09T05:11:57.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.600+0000 7f632cc05640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63281f6390 0x7f63281f67a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:11:57.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.600+0000 7f632cc05640 1 -- 172.21.15.1:0/1888177645 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f63280ccd50 con 0x7f63281f6390 2024-05-09T05:11:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.600+0000 7f632cc05640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63281f6390 0x7f63281f67a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6318014000 tx=0x7f6318005a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:11:57.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f631801ae20 con 0x7f63281f6390 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5050 (secure 0 0 0) 0x7f6318005800 con 0x7f63281f6390 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63281f6390 msgr2=0x7f63281f67a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f631dffb640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63281f6390 0x7f63281f67a0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6318014000 tx=0x7f6318005a20 comp rx=0 tx=0).stop 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f631dffb640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63281f6390 msgr2=0x7f63281f67a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f631dffb640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f63281f6390 0x7f63281f67a0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:57.602 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 54, 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 275599, 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:57.603 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:57.604 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:11:57.605 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:11:57.606 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:57.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:57.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:57.609 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:11:57.610 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.611 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.612 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:11:57.613 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:11:57.614 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f630c076280 msgr2=0x7f630c078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f632f66a640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f630c076280 0x7f630c078790 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f6320033000 tx=0x7f6320009000 comp rx=0 tx=0).stop 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280c5600 msgr2=0x7f63280c9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f632f66a640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280c5600 0x7f63280c9500 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f632400b550 tx=0x7f632400ba10 comp rx=0 tx=0).stop 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f632dc07640 1 -- 172.21.15.1:0/1888177645 reap_dead start 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 shutdown_connections 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f632f66a640 1 --2- 172.21.15.1:0/1888177645 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f63280c5600 0x7f63280c9500 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.601+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 >> 172.21.15.1:0/1888177645 conn(0x7f63280ce180 msgr2=0x7f6328075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.602+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 shutdown_connections 2024-05-09T05:11:57.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:11:57.602+0000 7f632f66a640 1 -- 172.21.15.1:0/1888177645 wait complete. 2024-05-09T05:11:57.616 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:11:57.616 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:12:02.819 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:12:02.821 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:12:17.616 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:12:17.617 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-05-09T05:12:17.617 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T05:12:17.617 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T05:12:17.620 INFO:tasks.ceph.mon.b:Started 2024-05-09T05:12:17.620 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:12:17.621 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:12:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.741+0000 7f95c6f9d640 1 Processor -- start 2024-05-09T05:12:17.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.741+0000 7f95c6f9d640 1 -- start start 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c6f9d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 0x7f95c01548f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c6f9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c006c6f0 0x7f95c015e1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c6f9d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f95c00c9000 con 0x7f95c00c9d80 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c6f9d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f95c00c8a60 con 0x7f95c00689a0 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c6f9d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f95c006dc90 con 0x7f95c006c6f0 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c553a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c006c6f0 0x7f95c015e1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95bffff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 0x7f95c01548f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c4d39640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f95c00c9d80 0x7f95c00c7710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:47704/0 (socket says 172.21.15.1:47704) 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c4d39640 1 -- 172.21.15.1:0/3876692298 learned_addr learned my addr 172.21.15.1:0/3876692298 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95bffff640 1 -- 172.21.15.1:0/3876692298 >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 msgr2=0x7f95c01548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:12:17.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95bffff640 1 -- 172.21.15.1:0/3876692298 >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 msgr2=0x7f95c01548f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:17.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95bffff640 1 --2- 172.21.15.1:0/3876692298 >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 0x7f95c01548f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:17.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95bffff640 1 --2- 172.21.15.1:0/3876692298 >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 0x7f95c01548f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:17.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c553a640 1 -- 172.21.15.1:0/3876692298 >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 msgr2=0x7f95c01548f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:17.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c553a640 1 --2- 172.21.15.1:0/3876692298 >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 0x7f95c01548f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:17.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c553a640 1 -- 172.21.15.1:0/3876692298 >> v1:172.21.15.196:6789/0 conn(0x7f95c00c9d80 legacy=0x7f95c00c7710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:17.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.743+0000 7f95c553a640 1 -- 172.21.15.1:0/3876692298 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c006df70 con 0x7f95c006c6f0 2024-05-09T05:12:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.744+0000 7f95c553a640 1 --2- 172.21.15.1:0/3876692298 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c006c6f0 0x7f95c015e1e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009d90 tx=0x7f95b802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=57e1a0aa5c711187 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.744+0000 7f95bf7fe640 1 -- 172.21.15.1:0/3876692298 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b8007740 con 0x7f95c006c6f0 2024-05-09T05:12:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.744+0000 7f95bf7fe640 1 -- 172.21.15.1:0/3876692298 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95b80078e0 con 0x7f95c006c6f0 2024-05-09T05:12:17.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.744+0000 7f95bf7fe640 1 -- 172.21.15.1:0/3876692298 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95b8007cb0 con 0x7f95c006c6f0 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.745+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3876692298 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c006c6f0 msgr2=0x7f95c015e1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.745+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3876692298 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c006c6f0 0x7f95c015e1e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f95b8009d90 tx=0x7f95b802a660 comp rx=0 tx=0).stop 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.745+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3876692298 shutdown_connections 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.745+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3876692298 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c006c6f0 0x7f95c015e1e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.745+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3876692298 >> v2:172.21.15.196:3300/0 conn(0x7f95c00689a0 0x7f95c01548f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.745+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3876692298 >> 172.21.15.1:0/3876692298 conn(0x7f95c00ce0e0 msgr2=0x7f95c00ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.745+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3876692298 shutdown_connections 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.745+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3876692298 wait complete. 2024-05-09T05:12:17.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.747+0000 7f95c6f9d640 1 Processor -- start 2024-05-09T05:12:17.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.747+0000 7f95c6f9d640 1 -- start start 2024-05-09T05:12:17.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95c6f9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c00689a0 0x7f95c01e3870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:17.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95c6f9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95c006c6f0 0x7f95c015d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:17.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95c6f9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 0x7f95c01574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95c6f9d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f95c006cb00 con 0x7f95c006c6f0 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95c6f9d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f95c015dcc0 con 0x7f95c00c9d80 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95c6f9d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f95c00c7e20 con 0x7f95c00689a0 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95bffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95c006c6f0 0x7f95c015d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95bffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95c006c6f0 0x7f95c015d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35526/0 (socket says 172.21.15.1:35526) 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95bffff640 1 -- 172.21.15.1:0/3847450208 learned_addr learned my addr 172.21.15.1:0/3847450208 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95c4d39640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c00689a0 0x7f95c01e3870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.749+0000 7f95c553a640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 0x7f95c01574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95c553a640 1 -- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 msgr2=0x7f95c01574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95c553a640 1 -- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 msgr2=0x7f95c01574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95c553a640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 0x7f95c01574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95c553a640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 0x7f95c01574c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bffff640 1 -- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c00689a0 msgr2=0x7f95c01e3870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:17.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bffff640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c00689a0 0x7f95c01e3870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bffff640 1 -- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 msgr2=0x7f95c01574c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bffff640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 0x7f95c01574c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bffff640 1 -- 172.21.15.1:0/3847450208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f95c01e3db0 con 0x7f95c006c6f0 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bffff640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95c006c6f0 0x7f95c015d2d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00d800 tx=0x7f95ac00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bd7fa640 1 -- 172.21.15.1:0/3847450208 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac00b6c0 con 0x7f95c006c6f0 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bd7fa640 1 -- 172.21.15.1:0/3847450208 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f95ac019070 con 0x7f95c006c6f0 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95c0149b10 con 0x7f95c006c6f0 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.750+0000 7f95bd7fa640 1 -- 172.21.15.1:0/3847450208 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95ac00bac0 con 0x7f95c006c6f0 2024-05-09T05:12:17.753 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.751+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95c01f1580 con 0x7f95c006c6f0 2024-05-09T05:12:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.752+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f95c01e53f0 con 0x7f95c006c6f0 2024-05-09T05:12:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.752+0000 7f95bd7fa640 1 -- 172.21.15.1:0/3847450208 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f95ac00bc60 con 0x7f95c006c6f0 2024-05-09T05:12:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.753+0000 7f95bd7fa640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95a4076300 0x7f95a4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.753+0000 7f95bd7fa640 1 -- 172.21.15.1:0/3847450208 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f95ac097b20 con 0x7f95c006c6f0 2024-05-09T05:12:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.753+0000 7f95c4d39640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95a4076300 0x7f95a4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:17.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.754+0000 7f95c4d39640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95a4076300 0x7f95a4078810 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f95b4000f60 tx=0x7f95b4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:17.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:17.759+0000 7f95bd7fa640 1 -- 172.21.15.1:0/3847450208 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f95ac061370 con 0x7f95c006c6f0 2024-05-09T05:12:18.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.060+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f95c0050870 con 0x7f95c006c6f0 2024-05-09T05:12:18.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.061+0000 7f95bd7fa640 1 -- 172.21.15.1:0/3847450208 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f95ac060c70 con 0x7f95c006c6f0 2024-05-09T05:12:18.063 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:12:18.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.063+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95a4076300 msgr2=0x7f95a4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:18.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95a4076300 0x7f95a4078810 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f95b4000f60 tx=0x7f95b4009040 comp rx=0 tx=0).stop 2024-05-09T05:12:18.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95c006c6f0 msgr2=0x7f95c015d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95c006c6f0 0x7f95c015d2d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f95ac00d800 tx=0x7f95ac00dcc0 comp rx=0 tx=0).stop 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 shutdown_connections 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f95a4076300 0x7f95a4078810 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f95c00c9d80 0x7f95c01574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f95c006c6f0 0x7f95c015d2d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 --2- 172.21.15.1:0/3847450208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f95c00689a0 0x7f95c01e3870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 >> 172.21.15.1:0/3847450208 conn(0x7f95c00ce0e0 msgr2=0x7f95c0151f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 shutdown_connections 2024-05-09T05:12:18.065 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:18.064+0000 7f95c6f9d640 1 -- 172.21.15.1:0/3847450208 wait complete. 2024-05-09T05:12:21.077 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:12:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.210+0000 7fca1f91f640 1 Processor -- start 2024-05-09T05:12:21.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.210+0000 7fca1f91f640 1 -- start start 2024-05-09T05:12:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1f91f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca180d16b0 0x7fca180d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1f91f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 0x7fca180ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1f91f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca1806f6d0 con 0x7fca1815ca90 2024-05-09T05:12:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1f91f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fca180cd0e0 con 0x7fca180d16b0 2024-05-09T05:12:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1f91f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fca180cc4d0 con 0x7fca1806f1b0 2024-05-09T05:12:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1d6bb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fca180d16b0 0x7fca180d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1debc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fca1815ca90 0x7fca1815cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35340/0 (socket says 172.21.15.1:35340) 2024-05-09T05:12:21.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1debc640 1 -- 172.21.15.1:0/3101864643 learned_addr learned my addr 172.21.15.1:0/3101864643 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1ceba640 1 --2- 172.21.15.1:0/3101864643 >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 0x7fca180ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1ceba640 1 -- 172.21.15.1:0/3101864643 >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 msgr2=0x7fca180ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1ceba640 1 -- 172.21.15.1:0/3101864643 >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 msgr2=0x7fca180ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1ceba640 1 --2- 172.21.15.1:0/3101864643 >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 0x7fca180ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1d6bb640 1 -- 172.21.15.1:0/3101864643 >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 msgr2=0x7fca180ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1ceba640 1 --2- 172.21.15.1:0/3101864643 >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 0x7fca180ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca0ffff640 1 -- 172.21.15.1:0/3101864643 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1788296736 0 0) 0x7fca1806f6d0 con 0x7fca1815ca90 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1d6bb640 1 --2- 172.21.15.1:0/3101864643 >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 0x7fca180ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1d6bb640 1 -- 172.21.15.1:0/3101864643 >> v1:172.21.15.1:6789/0 conn(0x7fca1815ca90 legacy=0x7fca1815cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:21.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.212+0000 7fca1d6bb640 1 -- 172.21.15.1:0/3101864643 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca180cc7b0 con 0x7fca180d16b0 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.213+0000 7fca1d6bb640 1 --2- 172.21.15.1:0/3101864643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca180d16b0 0x7fca180d03d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca1806d0d0 tx=0x7fca1402a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba5f06247f9f4069 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.213+0000 7fca0ffff640 1 -- 172.21.15.1:0/3101864643 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14037070 con 0x7fca180d16b0 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.213+0000 7fca0ffff640 1 -- 172.21.15.1:0/3101864643 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca14032440 con 0x7fca180d16b0 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.213+0000 7fca0ffff640 1 -- 172.21.15.1:0/3101864643 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14032810 con 0x7fca180d16b0 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.214+0000 7fca1f91f640 1 -- 172.21.15.1:0/3101864643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca180d16b0 msgr2=0x7fca180d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.214+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3101864643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca180d16b0 0x7fca180d03d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fca1806d0d0 tx=0x7fca1402a230 comp rx=0 tx=0).stop 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.214+0000 7fca1f91f640 1 -- 172.21.15.1:0/3101864643 shutdown_connections 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.214+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3101864643 >> v2:172.21.15.196:3300/0 conn(0x7fca1806f1b0 0x7fca180ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.214+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3101864643 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca180d16b0 0x7fca180d03d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.214+0000 7fca1f91f640 1 -- 172.21.15.1:0/3101864643 >> 172.21.15.1:0/3101864643 conn(0x7fca180b3660 msgr2=0x7fca180b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.214+0000 7fca1f91f640 1 -- 172.21.15.1:0/3101864643 shutdown_connections 2024-05-09T05:12:21.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.214+0000 7fca1f91f640 1 -- 172.21.15.1:0/3101864643 wait complete. 2024-05-09T05:12:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.217+0000 7fca1f91f640 1 Processor -- start 2024-05-09T05:12:21.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.217+0000 7fca1f91f640 1 -- start start 2024-05-09T05:12:21.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1f91f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca1806f1b0 0x7fca181e9110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1f91f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 0x7fca1814eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1f91f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca1815ca90 0x7fca1814f530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1f91f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fca180d0910 con 0x7fca1806f1b0 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1f91f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fca180ccd70 con 0x7fca180d16b0 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1f91f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fca180cec80 con 0x7fca1815ca90 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca1806f1b0 0x7fca181e9110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca1806f1b0 0x7fca181e9110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58268/0 (socket says 172.21.15.1:58268) 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 -- 172.21.15.1:0/3008457540 learned_addr learned my addr 172.21.15.1:0/3008457540 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1ceba640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 0x7fca1814eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1debc640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca1815ca90 0x7fca1814f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1ceba640 1 -- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 msgr2=0x7fca1814eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1ceba640 1 -- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 msgr2=0x7fca1814eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1ceba640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 0x7fca1814eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:21.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1ceba640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 0x7fca1814eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 -- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca1815ca90 msgr2=0x7fca1814f530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca1815ca90 0x7fca1814f530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 -- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 msgr2=0x7fca1814eff0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 0x7fca1814eff0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 -- 172.21.15.1:0/3008457540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca181f2360 con 0x7fca1806f1b0 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.219+0000 7fca1d6bb640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca1806f1b0 0x7fca181e9110 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fca14000c00 tx=0x7fca140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.220+0000 7fca0dffb640 1 -- 172.21.15.1:0/3008457540 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14037050 con 0x7fca1806f1b0 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.220+0000 7fca0dffb640 1 -- 172.21.15.1:0/3008457540 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca14038040 con 0x7fca1806f1b0 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.220+0000 7fca0dffb640 1 -- 172.21.15.1:0/3008457540 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca14006e10 con 0x7fca1806f1b0 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.220+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca180b60b0 con 0x7fca1806f1b0 2024-05-09T05:12:21.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.220+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca181f25e0 con 0x7fca1806f1b0 2024-05-09T05:12:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.221+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca18050870 con 0x7fca1806f1b0 2024-05-09T05:12:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.222+0000 7fca0dffb640 1 -- 172.21.15.1:0/3008457540 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fca14030980 con 0x7fca1806f1b0 2024-05-09T05:12:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.223+0000 7fca0dffb640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9fc076300 0x7fc9fc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.223+0000 7fca0dffb640 1 -- 172.21.15.1:0/3008457540 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fca140b7f00 con 0x7fca1806f1b0 2024-05-09T05:12:21.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.223+0000 7fca1ceba640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9fc076300 0x7fc9fc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:21.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.223+0000 7fca1ceba640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9fc076300 0x7fc9fc078810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fca180c87a0 tx=0x7fca08008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:21.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.227+0000 7fca0dffb640 1 -- 172.21.15.1:0/3008457540 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fca181f2360 con 0x7fca1806f1b0 2024-05-09T05:12:21.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.540+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fca18155bd0 con 0x7fca1806f1b0 2024-05-09T05:12:21.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.540+0000 7fca0dffb640 1 -- 172.21.15.1:0/3008457540 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fca140817f0 con 0x7fca1806f1b0 2024-05-09T05:12:21.541 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":54,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:12:21.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9fc076300 msgr2=0x7fc9fc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9fc076300 0x7fc9fc078810 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fca180c87a0 tx=0x7fca08008040 comp rx=0 tx=0).stop 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca1806f1b0 msgr2=0x7fca181e9110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca1806f1b0 0x7fca181e9110 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fca14000c00 tx=0x7fca140046c0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 shutdown_connections 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9fc076300 0x7fc9fc078810 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca180d16b0 0x7fca1814eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca1815ca90 0x7fca1814f530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 --2- 172.21.15.1:0/3008457540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca1806f1b0 0x7fca181e9110 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 >> 172.21.15.1:0/3008457540 conn(0x7fca180b3660 msgr2=0x7fca180cff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 shutdown_connections 2024-05-09T05:12:21.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:21.543+0000 7fca1f91f640 1 -- 172.21.15.1:0/3008457540 wait complete. 2024-05-09T05:12:24.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:12:24.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.679+0000 7f52a8827640 1 Processor -- start 2024-05-09T05:12:24.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.679+0000 7f52a8827640 1 -- start start 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a8827640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f52a40c4eb0 0x7f52a4070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a8827640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a415b110 con 0x7f52a4072120 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a8827640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f52a415b3b0 con 0x7f52a40ca340 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a8827640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f52a40c55b0 con 0x7f52a40c4eb0 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a1d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f52a4072120 0x7f52a40c62e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35348/0 (socket says 172.21.15.1:35348) 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a1d9b640 1 -- 172.21.15.1:0/1084357565 learned_addr learned my addr 172.21.15.1:0/1084357565 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a2d9d640 1 --2- 172.21.15.1:0/1084357565 >> v2:172.21.15.196:3300/0 conn(0x7f52a40c4eb0 0x7f52a4070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a2d9d640 1 -- 172.21.15.1:0/1084357565 >> v2:172.21.15.196:3300/0 conn(0x7f52a40c4eb0 msgr2=0x7f52a4070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a2d9d640 1 -- 172.21.15.1:0/1084357565 >> v2:172.21.15.196:3300/0 conn(0x7f52a40c4eb0 msgr2=0x7f52a4070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:24.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a2d9d640 1 --2- 172.21.15.1:0/1084357565 >> v2:172.21.15.196:3300/0 conn(0x7f52a40c4eb0 0x7f52a4070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a2d9d640 1 --2- 172.21.15.1:0/1084357565 >> v2:172.21.15.196:3300/0 conn(0x7f52a40c4eb0 0x7f52a4070e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2943897550 0 0) 0x7f52a415b110 con 0x7f52a4072120 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f528c003520 con 0x7f52a4072120 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1015742905 0 0) 0x7f528c003520 con 0x7f52a4072120 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 >> v2:172.21.15.196:3300/0 conn(0x7f52a40c4eb0 msgr2=0x7f52a4070e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a159a640 1 --2- 172.21.15.1:0/1084357565 >> v2:172.21.15.196:3300/0 conn(0x7f52a40c4eb0 0x7f52a4070e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.681+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 >> v1:172.21.15.196:6789/0 conn(0x7f52a40ca340 legacy=0x7f52a41547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a40c5800 con 0x7f52a4072120 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f52980016f0 con 0x7f52a4072120 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 >> v1:172.21.15.1:6789/0 conn(0x7f52a4072120 legacy=0x7f52a40c62e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f528c003ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f528c004800 0x7f528c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:24.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f528c0053f0 0x7f528c00ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f52a40c55b0 con 0x7f528c003a50 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f528c00c360 con 0x7f528c004800 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f528c00c560 con 0x7f528c0053f0 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a1d9b640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f528c004800 0x7f528c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a259c640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f528c003ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a2d9d640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f528c0053f0 0x7f528c00ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a259c640 1 -- 172.21.15.1:0/1084357565 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f528c0053f0 msgr2=0x7f528c00ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a259c640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f528c0053f0 0x7f528c00ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a259c640 1 -- 172.21.15.1:0/1084357565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f528c004800 msgr2=0x7f528c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a259c640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f528c004800 0x7f528c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a259c640 1 -- 172.21.15.1:0/1084357565 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f528c003520 con 0x7f528c003a50 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.682+0000 7f52a2d9d640 1 -- 172.21.15.1:0/1084357565 reap_dead start 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.683+0000 7f52a259c640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f528c003ee0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5294001790 tx=0x7f528c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40459bf532eebc58 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:24.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.683+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f52940013b0 con 0x7f528c003a50 2024-05-09T05:12:24.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.683+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f52940052d0 con 0x7f528c003a50 2024-05-09T05:12:24.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.684+0000 7f52a159a640 1 -- 172.21.15.1:0/1084357565 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529402aae0 con 0x7f528c003a50 2024-05-09T05:12:24.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.685+0000 7f52a8827640 1 -- 172.21.15.1:0/1084357565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 msgr2=0x7f528c003ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:24.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.685+0000 7f52a8827640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f528c003ee0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5294001790 tx=0x7f528c005830 comp rx=0 tx=0).stop 2024-05-09T05:12:24.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.685+0000 7f52a8827640 1 -- 172.21.15.1:0/1084357565 shutdown_connections 2024-05-09T05:12:24.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.685+0000 7f52a8827640 1 --2- 172.21.15.1:0/1084357565 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f528c003ee0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:24.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.685+0000 7f52a8827640 1 -- 172.21.15.1:0/1084357565 >> 172.21.15.1:0/1084357565 conn(0x7f52a40731d0 msgr2=0x7f52a40cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:24.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.685+0000 7f52a8827640 1 -- 172.21.15.1:0/1084357565 shutdown_connections 2024-05-09T05:12:24.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.685+0000 7f52a8827640 1 -- 172.21.15.1:0/1084357565 wait complete. 2024-05-09T05:12:24.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.688+0000 7f52a8827640 1 Processor -- start 2024-05-09T05:12:24.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.688+0000 7f52a8827640 1 -- start start 2024-05-09T05:12:24.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a8827640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f52a41f3d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a8827640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52a4154ed0 0x7f52a415e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a8827640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52a40717a0 0x7f52a4151430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a8827640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f52a40c52c0 con 0x7f528c003a50 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a8827640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f52a40c5dc0 con 0x7f52a40717a0 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a8827640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f52a40c4860 con 0x7f52a4154ed0 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f52a41f3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f52a41f3d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58272/0 (socket says 172.21.15.1:58272) 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 -- 172.21.15.1:0/1118498320 learned_addr learned my addr 172.21.15.1:0/1118498320 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a1d9b640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52a4154ed0 0x7f52a415e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a2d9d640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52a40717a0 0x7f52a4151430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 -- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52a4154ed0 msgr2=0x7f52a415e1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52a4154ed0 0x7f52a415e1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:24.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 -- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52a40717a0 msgr2=0x7f52a4151430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:24.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52a40717a0 0x7f52a4151430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:24.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 -- 172.21.15.1:0/1118498320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f52a41f4290 con 0x7f528c003a50 2024-05-09T05:12:24.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.690+0000 7f52a259c640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f52a41f3d50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5294001790 tx=0x7f5294001120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:24.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.691+0000 7f5292ffd640 1 -- 172.21.15.1:0/1118498320 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f529403f020 con 0x7f528c003a50 2024-05-09T05:12:24.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.691+0000 7f5292ffd640 1 -- 172.21.15.1:0/1118498320 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5294005480 con 0x7f528c003a50 2024-05-09T05:12:24.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.691+0000 7f5292ffd640 1 -- 172.21.15.1:0/1118498320 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5294037040 con 0x7f528c003a50 2024-05-09T05:12:24.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.691+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f52a415d350 con 0x7f528c003a50 2024-05-09T05:12:24.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.691+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f52a415d8f0 con 0x7f528c003a50 2024-05-09T05:12:24.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.692+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f52a4050870 con 0x7f528c003a50 2024-05-09T05:12:24.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.693+0000 7f5292ffd640 1 -- 172.21.15.1:0/1118498320 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5294042020 con 0x7f528c003a50 2024-05-09T05:12:24.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.694+0000 7f5292ffd640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5284076300 0x7f5284078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:24.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.694+0000 7f5292ffd640 1 -- 172.21.15.1:0/1118498320 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f52940b7040 con 0x7f528c003a50 2024-05-09T05:12:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.694+0000 7f52a1d9b640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5284076300 0x7f5284078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:24.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.694+0000 7f52a1d9b640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5284076300 0x7f5284078810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f52a4153e20 tx=0x7f5298001e70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:24.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:24.699+0000 7f5292ffd640 1 -- 172.21.15.1:0/1118498320 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5294049030 con 0x7f528c003a50 2024-05-09T05:12:25.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.025+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f52a40c9fc0 con 0x7f528c003a50 2024-05-09T05:12:25.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.026+0000 7f5292ffd640 1 -- 172.21.15.1:0/1118498320 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f52940802d0 con 0x7f528c003a50 2024-05-09T05:12:25.026 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:12:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5284076300 msgr2=0x7f5284078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5284076300 0x7f5284078810 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f52a4153e20 tx=0x7f5298001e70 comp rx=0 tx=0).stop 2024-05-09T05:12:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 msgr2=0x7f52a41f3d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f52a41f3d50 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5294001790 tx=0x7f5294001120 comp rx=0 tx=0).stop 2024-05-09T05:12:25.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 shutdown_connections 2024-05-09T05:12:25.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5284076300 0x7f5284078810 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f52a40717a0 0x7f52a4151430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f52a4154ed0 0x7f52a415e1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 --2- 172.21.15.1:0/1118498320 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f528c003a50 0x7f52a41f3d50 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 >> 172.21.15.1:0/1118498320 conn(0x7f52a40731d0 msgr2=0x7f52a4075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 shutdown_connections 2024-05-09T05:12:25.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.028+0000 7f52a8827640 1 -- 172.21.15.1:0/1118498320 wait complete. 2024-05-09T05:12:25.040 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:12:25.041 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:12:25.041 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:12:25.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.178+0000 7f97dbb00640 1 Processor -- start 2024-05-09T05:12:25.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.178+0000 7f97dbb00640 1 -- start start 2024-05-09T05:12:25.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97dbb00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d40c5670 0x7f97d4156a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97dbb00640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f97d406d3a0 0x7f97d41672a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97dbb00640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f97d415d770 con 0x7f97d406cf90 2024-05-09T05:12:25.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97dbb00640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f97d40cbfd0 con 0x7f97d406d3a0 2024-05-09T05:12:25.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97dbb00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f97d406cb00 con 0x7f97d40c5670 2024-05-09T05:12:25.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97da09d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f97d406d3a0 0x7f97d41672a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97d989c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d40c5670 0x7f97d4156a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97da09d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f97d406d3a0 0x7f97d41672a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:34322/0 (socket says 172.21.15.1:34322) 2024-05-09T05:12:25.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.180+0000 7f97da09d640 1 -- 172.21.15.1:0/2231410198 learned_addr learned my addr 172.21.15.1:0/2231410198 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:25.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.181+0000 7f97da09d640 1 -- 172.21.15.1:0/2231410198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d40c5670 msgr2=0x7f97d4156a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:25.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.181+0000 7f97da09d640 1 --2- 172.21.15.1:0/2231410198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d40c5670 0x7f97d4156a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.181+0000 7f97da09d640 1 -- 172.21.15.1:0/2231410198 >> v1:172.21.15.196:6789/0 conn(0x7f97d406cf90 legacy=0x7f97d40c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.181+0000 7f97da09d640 1 -- 172.21.15.1:0/2231410198 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d40cc220 con 0x7f97d406d3a0 2024-05-09T05:12:25.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.181+0000 7f97d989c640 1 --2- 172.21.15.1:0/2231410198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d40c5670 0x7f97d4156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:12:25.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.181+0000 7f97da09d640 1 --2- 172.21.15.1:0/2231410198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 0x7f97d41672a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97cc009e50 tx=0x7f97cc02a6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ad707f56b3b9cb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.181+0000 7f97d889a640 1 -- 172.21.15.1:0/2231410198 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc036070 con 0x7f97d406d3a0 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97d889a640 1 -- 172.21.15.1:0/2231410198 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97cc031600 con 0x7f97d406d3a0 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97d889a640 1 -- 172.21.15.1:0/2231410198 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc0319d0 con 0x7f97d406d3a0 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97dbb00640 1 -- 172.21.15.1:0/2231410198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 msgr2=0x7f97d41672a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97dbb00640 1 --2- 172.21.15.1:0/2231410198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 0x7f97d41672a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f97cc009e50 tx=0x7f97cc02a6a0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97dbb00640 1 -- 172.21.15.1:0/2231410198 shutdown_connections 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97dbb00640 1 --2- 172.21.15.1:0/2231410198 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 0x7f97d41672a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97dbb00640 1 --2- 172.21.15.1:0/2231410198 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d40c5670 0x7f97d4156a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97dbb00640 1 -- 172.21.15.1:0/2231410198 >> 172.21.15.1:0/2231410198 conn(0x7f97d40cac70 msgr2=0x7f97d40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97dbb00640 1 -- 172.21.15.1:0/2231410198 shutdown_connections 2024-05-09T05:12:25.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.182+0000 7f97dbb00640 1 -- 172.21.15.1:0/2231410198 wait complete. 2024-05-09T05:12:25.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.184+0000 7f97dbb00640 1 Processor -- start 2024-05-09T05:12:25.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.184+0000 7f97dbb00640 1 -- start start 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97dbb00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d406cf90 0x7f97d415e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97dbb00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 0x7f97d415b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97dbb00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c5670 0x7f97d40c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97dbb00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f97d4073730 con 0x7f97d40c5670 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97dbb00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f97d40cc500 con 0x7f97d406d3a0 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97dbb00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f97d4156fd0 con 0x7f97d406cf90 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97da09d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c5670 0x7f97d40c9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97da09d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c5670 0x7f97d40c9c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58284/0 (socket says 172.21.15.1:58284) 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97da09d640 1 -- 172.21.15.1:0/2131229067 learned_addr learned my addr 172.21.15.1:0/2131229067 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:25.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97d989c640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d406cf90 0x7f97d415e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97d909b640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 0x7f97d415b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97da09d640 1 -- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d406cf90 msgr2=0x7f97d415e340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97da09d640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f97d406cf90 0x7f97d415e340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97da09d640 1 -- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 msgr2=0x7f97d415b600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97da09d640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 0x7f97d415b600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97da09d640 1 -- 172.21.15.1:0/2131229067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97d40ca1b0 con 0x7f97d40c5670 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.186+0000 7f97d909b640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f97d406d3a0 0x7f97d415b600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.187+0000 7f97da09d640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c5670 0x7f97d40c9c70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f97cc034040 tx=0x7f97cc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.187+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc036050 con 0x7f97d40c5670 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.187+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f97cc042070 con 0x7f97d40c5670 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.187+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97d415b320 con 0x7f97d40c5670 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.187+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97cc009510 con 0x7f97d40c5670 2024-05-09T05:12:25.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.187+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97d4153cf0 con 0x7f97d40c5670 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.188+0000 7f97dbb00640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40ca380 0x7f97d414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.188+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f97d4153ec0 con 0x7f97d40ca380 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.188+0000 7f97d989c640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40ca380 0x7f97d414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.188+0000 7f97d989c640 1 -- 172.21.15.1:0/2131229067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f97d4050870 con 0x7f97d40ca380 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.189+0000 7f97d989c640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40ca380 0x7f97d414c2c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f97d0011860 tx=0x7f97d0011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.189+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97d000faa0 con 0x7f97d40ca380 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.189+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f97d000fc40 con 0x7f97d40ca380 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.189+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40ca380 msgr2=0x7f97d414c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.189+0000 7f97ca7fc640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40ca380 0x7f97d414c2c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f97d0011860 tx=0x7f97d0011d20 comp rx=0 tx=0).stop 2024-05-09T05:12:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.189+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40ca380 msgr2=0x7f97d414c2c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.189+0000 7f97ca7fc640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40ca380 0x7f97d414c2c0 secure :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f97d0011860 tx=0x7f97d0011d20 comp rx=0 tx=0).stop 2024-05-09T05:12:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.189+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f97cc007bf0 con 0x7f97d40c5670 2024-05-09T05:12:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.190+0000 7f97ca7fc640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f97b8075e20 0x7f97b8078330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.190+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f97cc0b8300 con 0x7f97d40c5670 2024-05-09T05:12:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.190+0000 7f97d989c640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f97b8075e20 0x7f97b8078330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.191+0000 7f97d989c640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f97b8075e20 0x7f97b8078330 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97d0000c00 tx=0x7f97d0008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.197+0000 7f97dbb00640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c8820 0x7f97d4148bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.197+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f97d415b0d0 con 0x7f97d40c8820 2024-05-09T05:12:25.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.197+0000 7f97d909b640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c8820 0x7f97d4148bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.198+0000 7f97d909b640 1 -- 172.21.15.1:0/2131229067 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f97d415c4b0 con 0x7f97d40c8820 2024-05-09T05:12:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.198+0000 7f97d909b640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c8820 0x7f97d4148bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f97d40cac40 tx=0x7f97c400fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.198+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f97c40052e0 con 0x7f97d40c8820 2024-05-09T05:12:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.198+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4893 (secure 0 0 0) 0x7f97c40052e0 con 0x7f97d40c8820 2024-05-09T05:12:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.198+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c8820 msgr2=0x7f97d4148bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.198+0000 7f97ca7fc640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c8820 0x7f97d4148bb0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f97d40cac40 tx=0x7f97c400fd20 comp rx=0 tx=0).stop 2024-05-09T05:12:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.198+0000 7f97ca7fc640 1 -- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c8820 msgr2=0x7f97d4148bb0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:25.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.198+0000 7f97ca7fc640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c8820 0x7f97d4148bb0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.200 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:12:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:12:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-05-09T05:12:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 133793, 2024-05-09T05:12:25.200 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:12:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:12:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:12:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:12:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:12:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:12:25.201 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:12:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.202 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.203 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:12:25.204 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.205 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.206 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:12:25.207 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:25.208 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:12:25.209 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.211 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.212 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:12:25.213 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.214 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.214 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.214 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:12:25.214 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f97b8075e20 msgr2=0x7f97b8078330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f97b8075e20 0x7f97b8078330 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f97d0000c00 tx=0x7f97d0008030 comp rx=0 tx=0).stop 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c5670 msgr2=0x7f97d40c9c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c5670 0x7f97d40c9c70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f97cc034040 tx=0x7f97cc004a40 comp rx=0 tx=0).stop 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97da09d640 1 -- 172.21.15.1:0/2131229067 reap_dead start 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 shutdown_connections 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 --2- 172.21.15.1:0/2131229067 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f97d40c5670 0x7f97d40c9c70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 >> 172.21.15.1:0/2131229067 conn(0x7f97d40cac70 msgr2=0x7f97d40c7160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 shutdown_connections 2024-05-09T05:12:25.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.199+0000 7f97dbb00640 1 -- 172.21.15.1:0/2131229067 wait complete. 2024-05-09T05:12:25.219 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:12:25.219 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:12:25.219 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:12:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.343+0000 7f3c45a61640 1 Processor -- start 2024-05-09T05:12:25.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.343+0000 7f3c45a61640 1 -- start start 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.345+0000 7f3c45a61640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c4015d130 0x7f3c401569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.345+0000 7f3c45a61640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c400c3ce0 con 0x7f3c400c48c0 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.345+0000 7f3c45a61640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3c400c3fc0 con 0x7f3c4006c6f0 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.345+0000 7f3c45a61640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f3c4015d540 con 0x7f3c4015d130 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.345+0000 7f3c3effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f3c4015d130 0x7f3c401569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.345+0000 7f3c3f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3c400c48c0 0x7f3c400c4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35358/0 (socket says 172.21.15.1:35358) 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.345+0000 7f3c3f7fe640 1 -- 172.21.15.1:0/2330300337 learned_addr learned my addr 172.21.15.1:0/2330300337 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.345+0000 7f3c3effd640 1 --2- 172.21.15.1:0/2330300337 >> v2:172.21.15.1:3300/0 conn(0x7f3c4015d130 0x7f3c401569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58304/0 (socket says 172.21.15.1:58304) 2024-05-09T05:12:25.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.346+0000 7f3c3effd640 1 -- 172.21.15.1:0/2330300337 >> v1:172.21.15.196:6789/0 conn(0x7f3c4006c6f0 legacy=0x7f3c400716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.346+0000 7f3c3effd640 1 -- 172.21.15.1:0/2330300337 >> v1:172.21.15.1:6789/0 conn(0x7f3c400c48c0 legacy=0x7f3c400c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.346+0000 7f3c3effd640 1 -- 172.21.15.1:0/2330300337 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4006e4a0 con 0x7f3c4015d130 2024-05-09T05:12:25.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.346+0000 7f3c3dffb640 1 -- 172.21.15.1:0/2330300337 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1547722581 0 0) 0x7f3c400c3ce0 con 0x7f3c400c48c0 2024-05-09T05:12:25.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.346+0000 7f3c3effd640 1 --2- 172.21.15.1:0/2330300337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c4015d130 0x7f3c401569c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3c300097a0 tx=0x7f3c3002a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b3e592927a601b91 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.347+0000 7f3c3dffb640 1 -- 172.21.15.1:0/2330300337 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30037070 con 0x7f3c4015d130 2024-05-09T05:12:25.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.347+0000 7f3c3dffb640 1 -- 172.21.15.1:0/2330300337 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c30032440 con 0x7f3c4015d130 2024-05-09T05:12:25.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.347+0000 7f3c3dffb640 1 -- 172.21.15.1:0/2330300337 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30032810 con 0x7f3c4015d130 2024-05-09T05:12:25.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.347+0000 7f3c45a61640 1 -- 172.21.15.1:0/2330300337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c4015d130 msgr2=0x7f3c401569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.347+0000 7f3c45a61640 1 --2- 172.21.15.1:0/2330300337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c4015d130 0x7f3c401569c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3c300097a0 tx=0x7f3c3002a230 comp rx=0 tx=0).stop 2024-05-09T05:12:25.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.347+0000 7f3c45a61640 1 -- 172.21.15.1:0/2330300337 shutdown_connections 2024-05-09T05:12:25.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.347+0000 7f3c45a61640 1 --2- 172.21.15.1:0/2330300337 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c4015d130 0x7f3c401569c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.347+0000 7f3c45a61640 1 -- 172.21.15.1:0/2330300337 >> 172.21.15.1:0/2330300337 conn(0x7f3c400747b0 msgr2=0x7f3c400ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.348+0000 7f3c45a61640 1 -- 172.21.15.1:0/2330300337 shutdown_connections 2024-05-09T05:12:25.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.348+0000 7f3c45a61640 1 -- 172.21.15.1:0/2330300337 wait complete. 2024-05-09T05:12:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.350+0000 7f3c45a61640 1 Processor -- start 2024-05-09T05:12:25.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.350+0000 7f3c45a61640 1 -- start start 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c45a61640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4006c6f0 0x7f3c4014de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c45a61640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c400c48c0 0x7f3c401f3270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c45a61640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3c4015d130 0x7f3c40154e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c45a61640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4006cb00 con 0x7f3c400c48c0 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c45a61640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4006e1a0 con 0x7f3c4006c6f0 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c45a61640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3c4006cc80 con 0x7f3c4015d130 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c400c48c0 0x7f3c401f3270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3e7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c400c48c0 0x7f3c401f3270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58310/0 (socket says 172.21.15.1:58310) 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3e7fc640 1 -- 172.21.15.1:0/3089947364 learned_addr learned my addr 172.21.15.1:0/3089947364 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3f7fe640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3c4015d130 0x7f3c40154e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3effd640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4006c6f0 0x7f3c4014de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3e7fc640 1 -- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3c4015d130 msgr2=0x7f3c40154e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3e7fc640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3c4015d130 0x7f3c40154e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3e7fc640 1 -- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4006c6f0 msgr2=0x7f3c4014de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3e7fc640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4006c6f0 0x7f3c4014de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.352+0000 7f3c3e7fc640 1 -- 172.21.15.1:0/3089947364 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4015a480 con 0x7f3c400c48c0 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.353+0000 7f3c3e7fc640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c400c48c0 0x7f3c401f3270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c3a0 tx=0x7f3c3400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.353+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34017070 con 0x7f3c400c48c0 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.353+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3c340052d0 con 0x7f3c400c48c0 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.353+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c401f37b0 con 0x7f3c400c48c0 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.353+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c340037f0 con 0x7f3c400c48c0 2024-05-09T05:12:25.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.353+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c4014ee50 con 0x7f3c400c48c0 2024-05-09T05:12:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.354+0000 7f3c45a61640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c401f2960 0x7f3c4014c320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.354+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4015a690 con 0x7f3c401f2960 2024-05-09T05:12:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.354+0000 7f3c3effd640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c401f2960 0x7f3c4014c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.355+0000 7f3c3effd640 1 -- 172.21.15.1:0/3089947364 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3c40050870 con 0x7f3c401f2960 2024-05-09T05:12:25.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.355+0000 7f3c3effd640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c401f2960 0x7f3c4014c320 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3c400c43d0 tx=0x7f3c300046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.355+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3c3401a070 con 0x7f3c400c48c0 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.355+0000 7f3c27fff640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3c2c0762d0 0x7f3c2c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3c340968a0 con 0x7f3c400c48c0 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c3f7fe640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3c2c0762d0 0x7f3c2c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30041070 con 0x7f3c401f2960 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3c30003df0 con 0x7f3c401f2960 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c401f2960 msgr2=0x7f3c4014c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c27fff640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c401f2960 0x7f3c4014c320 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3c400c43d0 tx=0x7f3c300046c0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c401f2960 msgr2=0x7f3c4014c320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c27fff640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c401f2960 0x7f3c4014c320 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.356+0000 7f3c3f7fe640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3c2c0762d0 0x7f3c2c0787e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3c38007fd0 tx=0x7f3c38005b70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.365+0000 7f3c45a61640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4014d700 0x7f3c4014c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.365+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3c400cb100 con 0x7f3c4014d700 2024-05-09T05:12:25.365 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.365+0000 7f3c3effd640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4014d700 0x7f3c4014c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.365+0000 7f3c3effd640 1 -- 172.21.15.1:0/3089947364 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3c400ca4a0 con 0x7f3c4014d700 2024-05-09T05:12:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.365+0000 7f3c3effd640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4014d700 0x7f3c4014c860 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c30000c00 tx=0x7f3c30008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c30053020 con 0x7f3c4014d700 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f3c30032b50 con 0x7f3c4014d700 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4014d700 msgr2=0x7f3c4014c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c27fff640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4014d700 0x7f3c4014c860 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f3c30000c00 tx=0x7f3c30008030 comp rx=0 tx=0).stop 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c27fff640 1 -- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4014d700 msgr2=0x7f3c4014c860 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c27fff640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3c4014d700 0x7f3c4014c860 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:12:25.367 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 7644, 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.368 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.369 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:12:25.370 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.371 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:12:25.372 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.373 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.374 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.375 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.376 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.377 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.378 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.379 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:12:25.379 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:12:25.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3c2c0762d0 msgr2=0x7f3c2c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c45a61640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3c2c0762d0 0x7f3c2c0787e0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f3c38007fd0 tx=0x7f3c38005b70 comp rx=0 tx=0).stop 2024-05-09T05:12:25.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c400c48c0 msgr2=0x7f3c401f3270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c45a61640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c400c48c0 0x7f3c401f3270 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f3c3400c3a0 tx=0x7f3c3400c860 comp rx=0 tx=0).stop 2024-05-09T05:12:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c3f7fe640 1 -- 172.21.15.1:0/3089947364 reap_dead start 2024-05-09T05:12:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 shutdown_connections 2024-05-09T05:12:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c45a61640 1 --2- 172.21.15.1:0/3089947364 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3c400c48c0 0x7f3c401f3270 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.366+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 >> 172.21.15.1:0/3089947364 conn(0x7f3c400747b0 msgr2=0x7f3c40071e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.367+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 shutdown_connections 2024-05-09T05:12:25.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.367+0000 7f3c45a61640 1 -- 172.21.15.1:0/3089947364 wait complete. 2024-05-09T05:12:25.381 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:12:25.381 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:12:25.382 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:12:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.514+0000 7f2a5c043640 1 Processor -- start 2024-05-09T05:12:25.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.515+0000 7f2a5c043640 1 -- start start 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5c043640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f2a54072400 0x7f2a54154530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5c043640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 0x7f2a54071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5c043640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a540cae10 0x7f2a540cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5c043640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2a540cb760 con 0x7f2a540cae10 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5c043640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f2a540ca3c0 con 0x7f2a54072400 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5c043640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2a540cbf90 con 0x7f2a54070f00 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5a5e0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a540cae10 0x7f2a540cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5a5e0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2a540cae10 0x7f2a540cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58312/0 (socket says 172.21.15.1:58312) 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5a5e0640 1 -- 172.21.15.1:0/2694346740 learned_addr learned my addr 172.21.15.1:0/2694346740 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a595de640 1 --2- 172.21.15.1:0/2694346740 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 0x7f2a54071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a59ddf640 1 --2- 172.21.15.1:0/2694346740 >> v2:172.21.15.196:3300/0 conn(0x7f2a54072400 0x7f2a54154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5a5e0640 1 -- 172.21.15.1:0/2694346740 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 msgr2=0x7f2a54071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:25.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5a5e0640 1 --2- 172.21.15.1:0/2694346740 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 0x7f2a54071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5a5e0640 1 -- 172.21.15.1:0/2694346740 >> v2:172.21.15.196:3300/0 conn(0x7f2a54072400 msgr2=0x7f2a54154530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:25.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5a5e0640 1 --2- 172.21.15.1:0/2694346740 >> v2:172.21.15.196:3300/0 conn(0x7f2a54072400 0x7f2a54154530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.517+0000 7f2a5a5e0640 1 -- 172.21.15.1:0/2694346740 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a5415a780 con 0x7f2a540cae10 2024-05-09T05:12:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.518+0000 7f2a5a5e0640 1 --2- 172.21.15.1:0/2694346740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a540cae10 0x7f2a540cb220 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2a540cd070 tx=0x7f2a4c02a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4404e0f0294c21 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.518+0000 7f2a58ddd640 1 -- 172.21.15.1:0/2694346740 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c036070 con 0x7f2a540cae10 2024-05-09T05:12:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.518+0000 7f2a58ddd640 1 -- 172.21.15.1:0/2694346740 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a4c031600 con 0x7f2a540cae10 2024-05-09T05:12:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.518+0000 7f2a58ddd640 1 -- 172.21.15.1:0/2694346740 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4c0319d0 con 0x7f2a540cae10 2024-05-09T05:12:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 -- 172.21.15.1:0/2694346740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a540cae10 msgr2=0x7f2a540cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 --2- 172.21.15.1:0/2694346740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a540cae10 0x7f2a540cb220 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f2a540cd070 tx=0x7f2a4c02a6a0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 -- 172.21.15.1:0/2694346740 shutdown_connections 2024-05-09T05:12:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 --2- 172.21.15.1:0/2694346740 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a540cae10 0x7f2a540cb220 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 --2- 172.21.15.1:0/2694346740 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 0x7f2a54071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 --2- 172.21.15.1:0/2694346740 >> v2:172.21.15.196:3300/0 conn(0x7f2a54072400 0x7f2a54154530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 -- 172.21.15.1:0/2694346740 >> 172.21.15.1:0/2694346740 conn(0x7f2a540d0470 msgr2=0x7f2a540cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 -- 172.21.15.1:0/2694346740 shutdown_connections 2024-05-09T05:12:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.519+0000 7f2a5c043640 1 -- 172.21.15.1:0/2694346740 wait complete. 2024-05-09T05:12:25.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.521+0000 7f2a5c043640 1 Processor -- start 2024-05-09T05:12:25.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.521+0000 7f2a5c043640 1 -- start start 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.522+0000 7f2a5c043640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 0x7f2a541594f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.522+0000 7f2a5c043640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a54072400 0x7f2a541515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.522+0000 7f2a5c043640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a540cae10 0x7f2a541491e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.522+0000 7f2a5c043640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540b2630 con 0x7f2a54072400 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.522+0000 7f2a5c043640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2a540ca8f0 con 0x7f2a540cae10 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.522+0000 7f2a5c043640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2a540b27b0 con 0x7f2a54070f00 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a54072400 0x7f2a541515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a54072400 0x7f2a541515b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58320/0 (socket says 172.21.15.1:58320) 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 -- 172.21.15.1:0/1155335132 learned_addr learned my addr 172.21.15.1:0/1155335132 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:25.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a59ddf640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 0x7f2a541594f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a5a5e0640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a540cae10 0x7f2a541491e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 -- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 msgr2=0x7f2a541594f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a54070f00 0x7f2a541594f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 -- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a540cae10 msgr2=0x7f2a541491e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2a540cae10 0x7f2a541491e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 -- 172.21.15.1:0/1155335132 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a54158850 con 0x7f2a54072400 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a595de640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a54072400 0x7f2a541515b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400e870 tx=0x7f2a4400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44017070 con 0x7f2a54072400 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2a4400ee60 con 0x7f2a54072400 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a541f3880 con 0x7f2a54072400 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.523+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4400cb80 con 0x7f2a54072400 2024-05-09T05:12:25.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.524+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a541f04c0 con 0x7f2a54072400 2024-05-09T05:12:25.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.524+0000 7f2a5c043640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a541f4360 0x7f2a54149720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.524+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2a541f3700 con 0x7f2a541f4360 2024-05-09T05:12:25.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.525+0000 7f2a59ddf640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a541f4360 0x7f2a54149720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.525+0000 7f2a59ddf640 1 -- 172.21.15.1:0/1155335132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2a54158b30 con 0x7f2a541f4360 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.525+0000 7f2a59ddf640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a541f4360 0x7f2a54149720 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2a50011890 tx=0x7f2a50011d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.526+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a50019070 con 0x7f2a541f4360 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.526+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2a4401a070 con 0x7f2a54072400 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.526+0000 7f2a4a7fc640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a38076220 0x7f2a38078730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.526+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2a440976f0 con 0x7f2a54072400 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.527+0000 7f2a5a5e0640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a38076220 0x7f2a38078730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.527+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2a50004240 con 0x7f2a541f4360 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.527+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a541f4360 msgr2=0x7f2a54149720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.527+0000 7f2a4a7fc640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a541f4360 0x7f2a54149720 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f2a50011890 tx=0x7f2a50011d50 comp rx=0 tx=0).stop 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.527+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a541f4360 msgr2=0x7f2a54149720 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.527+0000 7f2a4a7fc640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a541f4360 0x7f2a54149720 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.527+0000 7f2a5a5e0640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a38076220 0x7f2a38078730 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c033000 tx=0x7f2a4c009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.537+0000 7f2a5c043640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a5415a070 0x7f2a540b3ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.537+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2a541f0780 con 0x7f2a5415a070 2024-05-09T05:12:25.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.537+0000 7f2a59ddf640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a5415a070 0x7f2a540b3ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.538+0000 7f2a59ddf640 1 -- 172.21.15.1:0/1155335132 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2a54050870 con 0x7f2a5415a070 2024-05-09T05:12:25.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.538+0000 7f2a59ddf640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a5415a070 0x7f2a540b3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2a50011890 tx=0x7f2a500046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.538+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a5001e450 con 0x7f2a5415a070 2024-05-09T05:12:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5061 (secure 0 0 0) 0x7f2a500283f0 con 0x7f2a5415a070 2024-05-09T05:12:25.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a5415a070 msgr2=0x7f2a540b3ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.539 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 303535, 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:25.540 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.541 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.542 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:12:25.543 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:25.544 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.545 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:12:25.546 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.547 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:25.548 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.549 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.550 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.551 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:25.552 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:12:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:25.553 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:12:25.553 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:12:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a4a7fc640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a5415a070 0x7f2a540b3ef0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2a50011890 tx=0x7f2a500046c0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a4a7fc640 1 -- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a5415a070 msgr2=0x7f2a540b3ef0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:25.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a4a7fc640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2a5415a070 0x7f2a540b3ef0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a38076220 msgr2=0x7f2a38078730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5c043640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2a38076220 0x7f2a38078730 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2a4c033000 tx=0x7f2a4c009000 comp rx=0 tx=0).stop 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a54072400 msgr2=0x7f2a541515b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5c043640 1 --2- 172.21.15.1:0/1155335132 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2a54072400 0x7f2a541515b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f2a4400e870 tx=0x7f2a4400ed30 comp rx=0 tx=0).stop 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5a5e0640 1 -- 172.21.15.1:0/1155335132 reap_dead start 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5a5e0640 1 -- 172.21.15.1:0/1155335132 reap_dead start 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 shutdown_connections 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 >> 172.21.15.1:0/1155335132 conn(0x7f2a540d0470 msgr2=0x7f2a54075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 shutdown_connections 2024-05-09T05:12:25.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.539+0000 7f2a5c043640 1 -- 172.21.15.1:0/1155335132 wait complete. 2024-05-09T05:12:25.555 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:12:25.556 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:12:25.556 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:12:25.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.673+0000 7f8da3c05640 1 Processor -- start 2024-05-09T05:12:25.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.673+0000 7f8da3c05640 1 -- start start 2024-05-09T05:12:25.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da3c05640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8d9c0723b0 0x7f8d9c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da3c05640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d9c15cae0 con 0x7f8d9c0c6890 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da3c05640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8d9c15cdc0 con 0x7f8d9c0ca680 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da3c05640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8d9c06dd40 con 0x7f8d9c0723b0 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da19a1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f8d9c0c6890 0x7f8d9c070f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35360/0 (socket says 172.21.15.1:35360) 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da19a1640 1 -- 172.21.15.1:0/1968807549 learned_addr learned my addr 172.21.15.1:0/1968807549 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2358159494 0 0) 0x7f8d9c15cae0 con 0x7f8d9c0c6890 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8d88003520 con 0x7f8d9c0c6890 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.675+0000 7f8da11a0640 1 --2- 172.21.15.1:0/1968807549 >> v2:172.21.15.196:3300/0 conn(0x7f8d9c0723b0 0x7f8d9c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3864423328 0 0) 0x7f8d88003520 con 0x7f8d9c0c6890 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 >> v2:172.21.15.196:3300/0 conn(0x7f8d9c0723b0 msgr2=0x7f8d9c06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 --2- 172.21.15.1:0/1968807549 >> v2:172.21.15.196:3300/0 conn(0x7f8d9c0723b0 0x7f8d9c06d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 >> v1:172.21.15.196:6789/0 conn(0x7f8d9c0ca680 legacy=0x7f8d9c0caa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c15d0a0 con 0x7f8d9c0c6890 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f8d980016f0 con 0x7f8d9c0c6890 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 >> v1:172.21.15.1:6789/0 conn(0x7f8d9c0c6890 legacy=0x7f8d9c070f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d88003a50 0x7f8d88003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88004800 0x7f8d88004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d88005300 0x7f8d8800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c06dd40 con 0x7f8d88005300 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8d8800c290 con 0x7f8d88003a50 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8d8800c490 con 0x7f8d88004800 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.676+0000 7f8da21a2640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d88005300 0x7f8d8800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da11a0640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88004800 0x7f8d88004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da21a2640 1 -- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88004800 msgr2=0x7f8d88004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:25.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da19a1640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d88003a50 0x7f8d88003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da21a2640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88004800 0x7f8d88004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da21a2640 1 -- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d88003a50 msgr2=0x7f8d88003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da21a2640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d88003a50 0x7f8d88003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da21a2640 1 -- 172.21.15.1:0/1968807549 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d88003520 con 0x7f8d88005300 2024-05-09T05:12:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da11a0640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88004800 0x7f8d88004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:12:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da19a1640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d88003a50 0x7f8d88003ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:12:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da21a2640 1 -- 172.21.15.1:0/1968807549 reap_dead start 2024-05-09T05:12:25.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.677+0000 7f8da21a2640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d88005300 0x7f8d8800b9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8d94001e60 tx=0x7f8d88005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc1b5ebe6167acb5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.678+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94001ab0 con 0x7f8d88005300 2024-05-09T05:12:25.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.678+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d9402ae00 con 0x7f8d88005300 2024-05-09T05:12:25.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.678+0000 7f8da099f640 1 -- 172.21.15.1:0/1968807549 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d94003b70 con 0x7f8d88005300 2024-05-09T05:12:25.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.679+0000 7f8da3c05640 1 -- 172.21.15.1:0/1968807549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d88005300 msgr2=0x7f8d8800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.679+0000 7f8da3c05640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d88005300 0x7f8d8800b9c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f8d94001e60 tx=0x7f8d88005830 comp rx=0 tx=0).stop 2024-05-09T05:12:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.679+0000 7f8da3c05640 1 -- 172.21.15.1:0/1968807549 shutdown_connections 2024-05-09T05:12:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.679+0000 7f8da3c05640 1 --2- 172.21.15.1:0/1968807549 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d88005300 0x7f8d8800b9c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.679+0000 7f8da3c05640 1 -- 172.21.15.1:0/1968807549 >> 172.21.15.1:0/1968807549 conn(0x7f8d9c0734d0 msgr2=0x7f8d9c0cfa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.679+0000 7f8da3c05640 1 -- 172.21.15.1:0/1968807549 shutdown_connections 2024-05-09T05:12:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.679+0000 7f8da3c05640 1 -- 172.21.15.1:0/1968807549 wait complete. 2024-05-09T05:12:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.682+0000 7f8da3c05640 1 Processor -- start 2024-05-09T05:12:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.682+0000 7f8da3c05640 1 -- start start 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.683+0000 7f8da3c05640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88005300 0x7f8d9c1f2ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.683+0000 7f8da3c05640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d9c0cfae0 0x7f8d9c1596d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.683+0000 7f8da3c05640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d9c14c4b0 0x7f8d9c14c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.683+0000 7f8da3c05640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c15c790 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.683+0000 7f8da3c05640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8d9c15d660 con 0x7f8d9c14c4b0 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.683+0000 7f8da3c05640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8d9c156cf0 con 0x7f8d88005300 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d9c0cfae0 0x7f8d9c1596d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d9c0cfae0 0x7f8d9c1596d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:58342/0 (socket says 172.21.15.1:58342) 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 -- 172.21.15.1:0/242496888 learned_addr learned my addr 172.21.15.1:0/242496888 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da19a1640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88005300 0x7f8d9c1f2ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da21a2640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d9c14c4b0 0x7f8d9c14c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 -- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88005300 msgr2=0x7f8d9c1f2ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88005300 0x7f8d9c1f2ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 -- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d9c14c4b0 msgr2=0x7f8d9c14c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d9c14c4b0 0x7f8d9c14c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 -- 172.21.15.1:0/242496888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d9c14ce00 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da11a0640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d9c0cfae0 0x7f8d9c1596d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00e7e0 tx=0x7f8d8c00eca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8d927fc640 1 -- 172.21.15.1:0/242496888 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c017070 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8d927fc640 1 -- 172.21.15.1:0/242496888 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8d8c00c6d0 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.684+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d9c1f3020 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.685+0000 7f8d927fc640 1 -- 172.21.15.1:0/242496888 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d8c00f040 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.685+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d9c06e180 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.686+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d9c050870 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.686+0000 7f8d927fc640 1 -- 172.21.15.1:0/242496888 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8d8c01a070 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.687+0000 7f8d927fc640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d800762d0 0x7f8d800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:25.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.687+0000 7f8d927fc640 1 -- 172.21.15.1:0/242496888 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8d8c0972c0 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.688+0000 7f8da19a1640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d800762d0 0x7f8d800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:25.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.688+0000 7f8da19a1640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d800762d0 0x7f8d800787e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8d98000f60 tx=0x7f8d98004820 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:25.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.692+0000 7f8d927fc640 1 -- 172.21.15.1:0/242496888 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8d8c0604c0 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.984+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f8d9c0cbbd0 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.985+0000 7f8d927fc640 1 -- 172.21.15.1:0/242496888 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f8d8c0647c0 con 0x7f8d9c0cfae0 2024-05-09T05:12:25.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d800762d0 msgr2=0x7f8d800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d800762d0 0x7f8d800787e0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8d98000f60 tx=0x7f8d98004820 comp rx=0 tx=0).stop 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d9c0cfae0 msgr2=0x7f8d9c1596d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d9c0cfae0 0x7f8d9c1596d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f8d8c00e7e0 tx=0x7f8d8c00eca0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 shutdown_connections 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d800762d0 0x7f8d800787e0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8d9c14c4b0 0x7f8d9c14c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8d9c0cfae0 0x7f8d9c1596d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.988+0000 7f8da3c05640 1 --2- 172.21.15.1:0/242496888 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8d88005300 0x7f8d9c1f2ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.989+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 >> 172.21.15.1:0/242496888 conn(0x7f8d9c0734d0 msgr2=0x7f8d9c156380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.989+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 shutdown_connections 2024-05-09T05:12:25.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:25.989+0000 7f8da3c05640 1 -- 172.21.15.1:0/242496888 wait complete. 2024-05-09T05:12:26.002 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:12:33.069 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:12:33.072 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:12:36.003 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:12:36.003 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:12:36.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.129+0000 7fe24359d640 1 Processor -- start 2024-05-09T05:12:36.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.129+0000 7fe24359d640 1 -- start start 2024-05-09T05:12:36.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe24359d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 0x7fe23c15aea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe24359d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe23c0cac80 0x7fe23c0c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe24359d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe23c0cf7b0 0x7fe23c0cfbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe24359d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe23c154960 con 0x7fe23c0cf7b0 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe24359d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe23c0c48d0 con 0x7fe23c0cac80 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe24359d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe23c0cbf70 con 0x7fe23c154550 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe242d9c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe23c0cf7b0 0x7fe23c0cfbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe242d9c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe23c0cf7b0 0x7fe23c0cfbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34312/0 (socket says 172.21.15.1:34312) 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.131+0000 7fe242d9c640 1 -- 172.21.15.1:0/1876694526 learned_addr learned my addr 172.21.15.1:0/1876694526 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.132+0000 7fe24259b640 1 --2- 172.21.15.1:0/1876694526 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 0x7fe23c15aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.132+0000 7fe241d9a640 1 --2- 172.21.15.1:0/1876694526 >> v2:172.21.15.196:3300/0 conn(0x7fe23c0cac80 0x7fe23c0c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.132+0000 7fe242d9c640 1 -- 172.21.15.1:0/1876694526 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 msgr2=0x7fe23c15aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:36.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.132+0000 7fe242d9c640 1 --2- 172.21.15.1:0/1876694526 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 0x7fe23c15aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.132+0000 7fe242d9c640 1 -- 172.21.15.1:0/1876694526 >> v2:172.21.15.196:3300/0 conn(0x7fe23c0cac80 msgr2=0x7fe23c0c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:36.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.132+0000 7fe242d9c640 1 --2- 172.21.15.1:0/1876694526 >> v2:172.21.15.196:3300/0 conn(0x7fe23c0cac80 0x7fe23c0c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.132+0000 7fe242d9c640 1 -- 172.21.15.1:0/1876694526 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe23c15b3e0 con 0x7fe23c0cf7b0 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.133+0000 7fe242d9c640 1 --2- 172.21.15.1:0/1876694526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cf7b0 0x7fe23c0cfbc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe23c0d0380 tx=0x7fe23402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fdf72905e373c460 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.133+0000 7fe241599640 1 -- 172.21.15.1:0/1876694526 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe234036070 con 0x7fe23c0cf7b0 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.133+0000 7fe241599640 1 -- 172.21.15.1:0/1876694526 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe234031600 con 0x7fe23c0cf7b0 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.133+0000 7fe241599640 1 -- 172.21.15.1:0/1876694526 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2340319d0 con 0x7fe23c0cf7b0 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 -- 172.21.15.1:0/1876694526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cf7b0 msgr2=0x7fe23c0cfbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 --2- 172.21.15.1:0/1876694526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cf7b0 0x7fe23c0cfbc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe23c0d0380 tx=0x7fe23402a730 comp rx=0 tx=0).stop 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 -- 172.21.15.1:0/1876694526 shutdown_connections 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 --2- 172.21.15.1:0/1876694526 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cf7b0 0x7fe23c0cfbc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 --2- 172.21.15.1:0/1876694526 >> v2:172.21.15.196:3300/0 conn(0x7fe23c0cac80 0x7fe23c0c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 --2- 172.21.15.1:0/1876694526 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 0x7fe23c15aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 -- 172.21.15.1:0/1876694526 >> 172.21.15.1:0/1876694526 conn(0x7fe23c068650 msgr2=0x7fe23c068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:36.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 -- 172.21.15.1:0/1876694526 shutdown_connections 2024-05-09T05:12:36.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.134+0000 7fe24359d640 1 -- 172.21.15.1:0/1876694526 wait complete. 2024-05-09T05:12:36.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.135+0000 7fe24359d640 1 Processor -- start 2024-05-09T05:12:36.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.136+0000 7fe24359d640 1 -- start start 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24359d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cac80 0x7fe23c06f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24359d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe23c0cf7b0 0x7fe23c1f5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24359d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 0x7fe23c0c8f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24359d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe23c0cc4a0 con 0x7fe23c0cac80 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24359d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe23c0cc1c0 con 0x7fe23c0cf7b0 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24359d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe23c0cc780 con 0x7fe23c154550 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cac80 0x7fe23c06f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cac80 0x7fe23c06f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34328/0 (socket says 172.21.15.1:34328) 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 -- 172.21.15.1:0/433487951 learned_addr learned my addr 172.21.15.1:0/433487951 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe241d9a640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe23c0cf7b0 0x7fe23c1f5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe242d9c640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 0x7fe23c0c8f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 -- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 msgr2=0x7fe23c0c8f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 0x7fe23c0c8f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 -- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe23c0cf7b0 msgr2=0x7fe23c1f5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe23c0cf7b0 0x7fe23c1f5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 -- 172.21.15.1:0/433487951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe23c06f770 con 0x7fe23c0cac80 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.137+0000 7fe24259b640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cac80 0x7fe23c06f150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe23800d5f0 tx=0x7fe23800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.138+0000 7fe232ffd640 1 -- 172.21.15.1:0/433487951 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2380125f0 con 0x7fe23c0cac80 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.138+0000 7fe232ffd640 1 -- 172.21.15.1:0/433487951 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe238019070 con 0x7fe23c0cac80 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.138+0000 7fe232ffd640 1 -- 172.21.15.1:0/433487951 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe2380129d0 con 0x7fe23c0cac80 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.138+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe23c14db80 con 0x7fe23c0cac80 2024-05-09T05:12:36.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.138+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe23c14e140 con 0x7fe23c0cac80 2024-05-09T05:12:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.139+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe23c076f80 con 0x7fe23c0cac80 2024-05-09T05:12:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.140+0000 7fe232ffd640 1 -- 172.21.15.1:0/433487951 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe238012b70 con 0x7fe23c0cac80 2024-05-09T05:12:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.140+0000 7fe232ffd640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe220076300 0x7fe220078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.141+0000 7fe232ffd640 1 -- 172.21.15.1:0/433487951 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe238098340 con 0x7fe23c0cac80 2024-05-09T05:12:36.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.141+0000 7fe241d9a640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe220076300 0x7fe220078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.141+0000 7fe241d9a640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe220076300 0x7fe220078810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe23c0cb270 tx=0x7fe22c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.146+0000 7fe232ffd640 1 -- 172.21.15.1:0/433487951 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe2380616b0 con 0x7fe23c0cac80 2024-05-09T05:12:36.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.450+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe23c0c9c90 con 0x7fe23c0cac80 2024-05-09T05:12:36.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.450+0000 7fe232ffd640 1 -- 172.21.15.1:0/433487951 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe2380659b0 con 0x7fe23c0cac80 2024-05-09T05:12:36.451 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":56,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe220076300 msgr2=0x7fe220078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe220076300 0x7fe220078810 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe23c0cb270 tx=0x7fe22c008040 comp rx=0 tx=0).stop 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cac80 msgr2=0x7fe23c06f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cac80 0x7fe23c06f150 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe23800d5f0 tx=0x7fe23800dab0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 shutdown_connections 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe220076300 0x7fe220078810 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe23c0cf7b0 0x7fe23c1f5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe23c154550 0x7fe23c0c8f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 --2- 172.21.15.1:0/433487951 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe23c0cac80 0x7fe23c06f150 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 >> 172.21.15.1:0/433487951 conn(0x7fe23c068650 msgr2=0x7fe23c0c9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 shutdown_connections 2024-05-09T05:12:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.453+0000 7fe24359d640 1 -- 172.21.15.1:0/433487951 wait complete. 2024-05-09T05:12:36.466 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:12:36.466 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:12:36.466 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:12:36.466 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:12:36.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.608+0000 7f502b501640 1 Processor -- start 2024-05-09T05:12:36.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.609+0000 7f502b501640 1 -- start start 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f502b501640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 0x7f50240cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f502b501640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50240cc370 0x7f50240cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f502b501640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50240c7970 con 0x7f50240747f0 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f502b501640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f50240c5600 con 0x7f50240cc370 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f502b501640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f50240c7be0 con 0x7f50240cbf60 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f5029a9e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50240cc370 0x7f50240cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f5029a9e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f50240cc370 0x7f50240cda00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34340/0 (socket says 172.21.15.1:34340) 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f502929d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f50240747f0 0x7f5024152c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:53720/0 (socket says 172.21.15.1:53720) 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f5029a9e640 1 -- 172.21.15.1:0/477807696 learned_addr learned my addr 172.21.15.1:0/477807696 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f5028a9c640 1 --2- 172.21.15.1:0/477807696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 0x7f50240cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f501bfff640 1 -- 172.21.15.1:0/477807696 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3963261241 0 0) 0x7f50240c7970 con 0x7f50240747f0 2024-05-09T05:12:36.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f5029a9e640 1 -- 172.21.15.1:0/477807696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 msgr2=0x7f50240cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f5029a9e640 1 --2- 172.21.15.1:0/477807696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 0x7f50240cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f5029a9e640 1 -- 172.21.15.1:0/477807696 >> v1:172.21.15.196:6789/0 conn(0x7f50240747f0 legacy=0x7f5024152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.611+0000 7f5029a9e640 1 -- 172.21.15.1:0/477807696 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50240c7ec0 con 0x7f50240cc370 2024-05-09T05:12:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.612+0000 7f5029a9e640 1 --2- 172.21.15.1:0/477807696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 0x7f50240cda00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f501c009e50 tx=0x7f501c02a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6698a58bc51579f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.612+0000 7f501bfff640 1 -- 172.21.15.1:0/477807696 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c036070 con 0x7f50240cc370 2024-05-09T05:12:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.613+0000 7f501bfff640 1 -- 172.21.15.1:0/477807696 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501c031600 con 0x7f50240cc370 2024-05-09T05:12:36.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.613+0000 7f501bfff640 1 -- 172.21.15.1:0/477807696 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c0319d0 con 0x7f50240cc370 2024-05-09T05:12:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.614+0000 7f502b501640 1 -- 172.21.15.1:0/477807696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 msgr2=0x7f50240cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.614+0000 7f502b501640 1 --2- 172.21.15.1:0/477807696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 0x7f50240cda00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f501c009e50 tx=0x7f501c02a680 comp rx=0 tx=0).stop 2024-05-09T05:12:36.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.614+0000 7f502b501640 1 -- 172.21.15.1:0/477807696 shutdown_connections 2024-05-09T05:12:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.614+0000 7f502b501640 1 --2- 172.21.15.1:0/477807696 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 0x7f50240cda00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.614+0000 7f502b501640 1 --2- 172.21.15.1:0/477807696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 0x7f50240cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.614+0000 7f502b501640 1 -- 172.21.15.1:0/477807696 >> 172.21.15.1:0/477807696 conn(0x7f50240ce180 msgr2=0x7f50240ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.614+0000 7f502b501640 1 -- 172.21.15.1:0/477807696 shutdown_connections 2024-05-09T05:12:36.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.614+0000 7f502b501640 1 -- 172.21.15.1:0/477807696 wait complete. 2024-05-09T05:12:36.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.616+0000 7f502b501640 1 Processor -- start 2024-05-09T05:12:36.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.616+0000 7f502b501640 1 -- start start 2024-05-09T05:12:36.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.617+0000 7f502b501640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240747f0 0x7f5024157170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.617+0000 7f502b501640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 0x7f502406c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.617+0000 7f502b501640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 0x7f5024159380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.617+0000 7f502b501640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f50240740c0 con 0x7f50240cc370 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.617+0000 7f502b501640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f50240c8480 con 0x7f50240747f0 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.617+0000 7f502b501640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f50240c5a90 con 0x7f50240cbf60 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 0x7f5024159380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 0x7f5024159380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34352/0 (socket says 172.21.15.1:34352) 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 -- 172.21.15.1:0/1228652384 learned_addr learned my addr 172.21.15.1:0/1228652384 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f502929d640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240747f0 0x7f5024157170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5028a9c640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 0x7f502406c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 -- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 msgr2=0x7f502406c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50240cbf60 0x7f502406c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 -- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240747f0 msgr2=0x7f5024157170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240747f0 0x7f5024157170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 -- 172.21.15.1:0/1228652384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50241f4300 con 0x7f50240cc370 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5029a9e640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 0x7f5024159380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f501c009e50 tx=0x7f501c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c041020 con 0x7f50240cc370 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501c037040 con 0x7f50240cc370 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.618+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50241576b0 con 0x7f50240cc370 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.619+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c036050 con 0x7f50240cc370 2024-05-09T05:12:36.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.619+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50241f3d40 con 0x7f50240cc370 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.620+0000 7f502b501640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024156b70 0x7f50241449d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.620+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f50241f37d0 con 0x7f5024156b70 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.620+0000 7f502929d640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024156b70 0x7f50241449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.621+0000 7f502929d640 1 -- 172.21.15.1:0/1228652384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5024050870 con 0x7f5024156b70 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.621+0000 7f502929d640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024156b70 0x7f50241449d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5020000c00 tx=0x7f50200018c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.621+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5020019070 con 0x7f5024156b70 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.621+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f501c007c50 con 0x7f50240cc370 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5019ffb640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5008076200 0x7f5008078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f501c0b7be0 con 0x7f50240cc370 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5020011580 con 0x7f5024156b70 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5028a9c640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5008076200 0x7f5008078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024156b70 msgr2=0x7f50241449d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5019ffb640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024156b70 0x7f50241449d0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5020000c00 tx=0x7f50200018c0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024156b70 msgr2=0x7f50241449d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:36.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5019ffb640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024156b70 0x7f50241449d0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.622+0000 7f5028a9c640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5008076200 0x7f5008078710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f502406d1a0 tx=0x7f5014009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.628+0000 7f502b501640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024157880 0x7f5024144f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.628+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f502406e2e0 con 0x7f5024157880 2024-05-09T05:12:36.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.629+0000 7f502929d640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024157880 0x7f5024144f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.629+0000 7f502929d640 1 -- 172.21.15.1:0/1228652384 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f50240c5190 con 0x7f5024157880 2024-05-09T05:12:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.629+0000 7f502929d640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024157880 0x7f5024144f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5020000c00 tx=0x7f5020007f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.629+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f502001de60 con 0x7f5024157880 2024-05-09T05:12:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4894 (secure 0 0 0) 0x7f5020014450 con 0x7f5024157880 2024-05-09T05:12:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024157880 msgr2=0x7f5024144f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f5019ffb640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024157880 0x7f5024144f10 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5020000c00 tx=0x7f5020007f90 comp rx=0 tx=0).stop 2024-05-09T05:12:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f5019ffb640 1 -- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024157880 msgr2=0x7f5024144f10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:36.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f5019ffb640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5024157880 0x7f5024144f10 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f5020000c00 tx=0x7f5020007f90 comp rx=0 tx=0).stop 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 145224, 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:12:36.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 13, 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:36.632 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:36.633 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:12:36.634 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.635 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:36.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:36.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:36.638 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:12:36.639 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.639 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.639 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.639 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.673 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:12:36.673 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:12:36.673 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.675 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:36.676 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:36.677 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:12:36.678 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.678 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.678 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:12:36.678 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:12:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5008076200 msgr2=0x7f5008078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f502b501640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5008076200 0x7f5008078710 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f502406d1a0 tx=0x7f5014009190 comp rx=0 tx=0).stop 2024-05-09T05:12:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 msgr2=0x7f5024159380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f502b501640 1 --2- 172.21.15.1:0/1228652384 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240cc370 0x7f5024159380 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f501c009e50 tx=0x7f501c004a40 comp rx=0 tx=0).stop 2024-05-09T05:12:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f5029a9e640 1 -- 172.21.15.1:0/1228652384 reap_dead start 2024-05-09T05:12:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f5029a9e640 1 -- 172.21.15.1:0/1228652384 reap_dead start 2024-05-09T05:12:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 shutdown_connections 2024-05-09T05:12:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 >> 172.21.15.1:0/1228652384 conn(0x7f50240ce180 msgr2=0x7f50240ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 shutdown_connections 2024-05-09T05:12:36.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.630+0000 7f502b501640 1 -- 172.21.15.1:0/1228652384 wait complete. 2024-05-09T05:12:36.680 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:12:36.680 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:12:36.680 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:12:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.809+0000 7fcf6e670640 1 Processor -- start 2024-05-09T05:12:36.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.809+0000 7fcf6e670640 1 -- start start 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6e670640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680c6970 0x7fcf68156a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6e670640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcf680cb080 0x7fcf680cc750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6e670640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf680ccc90 0x7fcf6815fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6e670640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcf6815ca00 con 0x7fcf680ccc90 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6e670640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fcf6815cce0 con 0x7fcf680cb080 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6e670640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcf6815cfc0 con 0x7fcf680c6970 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6cc0d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf680ccc90 0x7fcf6815fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6cc0d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf680ccc90 0x7fcf6815fee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34370/0 (socket says 172.21.15.1:34370) 2024-05-09T05:12:36.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.810+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/1965789649 learned_addr learned my addr 172.21.15.1:0/1965789649 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.811+0000 7fcf677fe640 1 --2- 172.21.15.1:0/1965789649 >> v2:172.21.15.196:3300/0 conn(0x7fcf680cb080 0x7fcf680cc750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.811+0000 7fcf67fff640 1 --2- 172.21.15.1:0/1965789649 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680c6970 0x7fcf68156a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.811+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/1965789649 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680c6970 msgr2=0x7fcf68156a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.811+0000 7fcf6cc0d640 1 --2- 172.21.15.1:0/1965789649 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680c6970 0x7fcf68156a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.811+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/1965789649 >> v2:172.21.15.196:3300/0 conn(0x7fcf680cb080 msgr2=0x7fcf680cc750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.811+0000 7fcf6cc0d640 1 --2- 172.21.15.1:0/1965789649 >> v2:172.21.15.196:3300/0 conn(0x7fcf680cb080 0x7fcf680cc750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.811+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/1965789649 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf6815d2a0 con 0x7fcf680ccc90 2024-05-09T05:12:36.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.812+0000 7fcf6cc0d640 1 --2- 172.21.15.1:0/1965789649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680ccc90 0x7fcf6815fee0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcf680d0aa0 tx=0x7fcf6002a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3092c60b25f9e5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.812+0000 7fcf66ffd640 1 -- 172.21.15.1:0/1965789649 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60036070 con 0x7fcf680ccc90 2024-05-09T05:12:36.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.812+0000 7fcf66ffd640 1 -- 172.21.15.1:0/1965789649 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf60031600 con 0x7fcf680ccc90 2024-05-09T05:12:36.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.812+0000 7fcf66ffd640 1 -- 172.21.15.1:0/1965789649 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf600319d0 con 0x7fcf680ccc90 2024-05-09T05:12:36.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 -- 172.21.15.1:0/1965789649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680ccc90 msgr2=0x7fcf6815fee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 --2- 172.21.15.1:0/1965789649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680ccc90 0x7fcf6815fee0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fcf680d0aa0 tx=0x7fcf6002a6a0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 -- 172.21.15.1:0/1965789649 shutdown_connections 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 --2- 172.21.15.1:0/1965789649 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680ccc90 0x7fcf6815fee0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 --2- 172.21.15.1:0/1965789649 >> v2:172.21.15.196:3300/0 conn(0x7fcf680cb080 0x7fcf680cc750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 --2- 172.21.15.1:0/1965789649 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680c6970 0x7fcf68156a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 -- 172.21.15.1:0/1965789649 >> 172.21.15.1:0/1965789649 conn(0x7fcf68068830 msgr2=0x7fcf68068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 -- 172.21.15.1:0/1965789649 shutdown_connections 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.813+0000 7fcf6e670640 1 -- 172.21.15.1:0/1965789649 wait complete. 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.815+0000 7fcf6e670640 1 Processor -- start 2024-05-09T05:12:36.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.815+0000 7fcf6e670640 1 -- start start 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6e670640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf680cb080 0x7fcf681583c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6e670640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680ccc90 0x7fcf680cdeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6e670640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680cee20 0x7fcf6814e2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6e670640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6806d800 con 0x7fcf680cee20 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6e670640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcf68156f40 con 0x7fcf680cb080 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6e670640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcf6806d980 con 0x7fcf680ccc90 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6cc0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680cee20 0x7fcf6814e2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6cc0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680cee20 0x7fcf6814e2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34374/0 (socket says 172.21.15.1:34374) 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/3661875628 learned_addr learned my addr 172.21.15.1:0/3661875628 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:36.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf677fe640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680ccc90 0x7fcf680cdeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.816+0000 7fcf67fff640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf680cb080 0x7fcf681583c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680ccc90 msgr2=0x7fcf680cdeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf6cc0d640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf680ccc90 0x7fcf680cdeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf680cb080 msgr2=0x7fcf681583c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf6cc0d640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf680cb080 0x7fcf681583c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/3661875628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf680d0810 con 0x7fcf680cee20 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf6cc0d640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680cee20 0x7fcf6814e2c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009f40 tx=0x7fcf60004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60036050 con 0x7fcf680cee20 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf60042070 con 0x7fcf680cee20 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf680ce3f0 con 0x7fcf680cee20 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf60009510 con 0x7fcf680cee20 2024-05-09T05:12:36.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.817+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf680d0530 con 0x7fcf680cee20 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.818+0000 7fcf6e670640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf6814e800 0x7fcf681571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.818+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcf680d0040 con 0x7fcf6814e800 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.819+0000 7fcf67fff640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf6814e800 0x7fcf681571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.819+0000 7fcf67fff640 1 -- 172.21.15.1:0/3661875628 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcf68158900 con 0x7fcf6814e800 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.819+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcf60007bd0 con 0x7fcf680cee20 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.819+0000 7fcf67fff640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf6814e800 0x7fcf681571e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcf58011890 tx=0x7fcf58011d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf64ff9640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf54076300 0x7fcf54078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcf600b82c0 con 0x7fcf680cee20 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf677fe640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf54076300 0x7fcf54078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58019070 con 0x7fcf6814e800 2024-05-09T05:12:36.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcf58004240 con 0x7fcf6814e800 2024-05-09T05:12:36.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf6814e800 msgr2=0x7fcf681571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf64ff9640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf6814e800 0x7fcf681571e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcf58011890 tx=0x7fcf58011d50 comp rx=0 tx=0).stop 2024-05-09T05:12:36.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf6814e800 msgr2=0x7fcf681571e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:36.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf64ff9640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf6814e800 0x7fcf681571e0 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcf58011890 tx=0x7fcf58011d50 comp rx=0 tx=0).stop 2024-05-09T05:12:36.822 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.820+0000 7fcf677fe640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf54076300 0x7fcf54078810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcf680ca6a0 tx=0x7fcf5c009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.829+0000 7fcf6e670640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf68157720 0x7fcf681591b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.829+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcf6814ec10 con 0x7fcf68157720 2024-05-09T05:12:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.829+0000 7fcf67fff640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf68157720 0x7fcf681591b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.829+0000 7fcf67fff640 1 -- 172.21.15.1:0/3661875628 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcf68050870 con 0x7fcf68157720 2024-05-09T05:12:36.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.829+0000 7fcf67fff640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf68157720 0x7fcf681591b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcf58011890 tx=0x7fcf580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf58019040 con 0x7fcf68157720 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7fcf5800fe90 con 0x7fcf68157720 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf68157720 msgr2=0x7fcf681591b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf64ff9640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf68157720 0x7fcf681591b0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fcf58011890 tx=0x7fcf580046c0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf64ff9640 1 -- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf68157720 msgr2=0x7fcf681591b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf64ff9640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf68157720 0x7fcf681591b0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-05-09T05:12:36.831 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19108, 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 13, 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.832 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.833 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:36.834 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.835 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:36.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:12:36.840 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:12:36.841 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:12:36.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf54076300 msgr2=0x7fcf54078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcf54076300 0x7fcf54078810 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcf680ca6a0 tx=0x7fcf5c009190 comp rx=0 tx=0).stop 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680cee20 msgr2=0x7fcf6814e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680cee20 0x7fcf6814e2c0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fcf60009f40 tx=0x7fcf60004a40 comp rx=0 tx=0).stop 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6cc0d640 1 -- 172.21.15.1:0/3661875628 reap_dead start 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 shutdown_connections 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 --2- 172.21.15.1:0/3661875628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf680cee20 0x7fcf6814e2c0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 >> 172.21.15.1:0/3661875628 conn(0x7fcf68068830 msgr2=0x7fcf6815a6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 shutdown_connections 2024-05-09T05:12:36.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.830+0000 7fcf6e670640 1 -- 172.21.15.1:0/3661875628 wait complete. 2024-05-09T05:12:36.843 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:12:36.843 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:12:36.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:12:36.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.972+0000 7f5423fe3640 1 Processor -- start 2024-05-09T05:12:36.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.972+0000 7f5423fe3640 1 -- start start 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.973+0000 7f5423fe3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 0x7f541c0cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.973+0000 7f5423fe3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f541c160460 0x7f541c160870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.973+0000 7f5423fe3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f541c15c7b0 con 0x7f541c0ce010 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.973+0000 7f5423fe3640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f541c15d0c0 con 0x7f541c160460 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.973+0000 7f5423fe3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f541c0cca50 con 0x7f541c0b4ec0 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f542157e640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f541c0ce010 0x7f541c15fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35228/0 (socket says 172.21.15.1:35228) 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f542157e640 1 -- 172.21.15.1:0/1518357149 learned_addr learned my addr 172.21.15.1:0/1518357149 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f5422580640 1 --2- 172.21.15.1:0/1518357149 >> v2:172.21.15.1:3300/0 conn(0x7f541c160460 0x7f541c160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f5421d7f640 1 --2- 172.21.15.1:0/1518357149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 0x7f541c0cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f5420d7d640 1 -- 172.21.15.1:0/1518357149 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1378014314 0 0) 0x7f541c15c7b0 con 0x7f541c0ce010 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f5420d7d640 1 -- 172.21.15.1:0/1518357149 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5408003520 con 0x7f541c0ce010 2024-05-09T05:12:36.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f5422580640 1 -- 172.21.15.1:0/1518357149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 msgr2=0x7f541c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:36.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f5422580640 1 --2- 172.21.15.1:0/1518357149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 0x7f541c0cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f5422580640 1 -- 172.21.15.1:0/1518357149 >> v1:172.21.15.1:6789/0 conn(0x7f541c0ce010 legacy=0x7f541c15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.974+0000 7f5422580640 1 -- 172.21.15.1:0/1518357149 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c0ccd30 con 0x7f541c160460 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.975+0000 7f5422580640 1 --2- 172.21.15.1:0/1518357149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c160460 0x7f541c160870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f541c15d240 tx=0x7f541402a6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=876320d85c785cda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.975+0000 7f5420d7d640 1 -- 172.21.15.1:0/1518357149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414007740 con 0x7f541c160460 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.975+0000 7f5420d7d640 1 -- 172.21.15.1:0/1518357149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f54140078e0 con 0x7f541c160460 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.975+0000 7f5420d7d640 1 -- 172.21.15.1:0/1518357149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5414007cb0 con 0x7f541c160460 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.976+0000 7f5423fe3640 1 -- 172.21.15.1:0/1518357149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c160460 msgr2=0x7f541c160870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.976+0000 7f5423fe3640 1 --2- 172.21.15.1:0/1518357149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c160460 0x7f541c160870 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f541c15d240 tx=0x7f541402a6d0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.976+0000 7f5423fe3640 1 -- 172.21.15.1:0/1518357149 shutdown_connections 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.976+0000 7f5423fe3640 1 --2- 172.21.15.1:0/1518357149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c160460 0x7f541c160870 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.976+0000 7f5423fe3640 1 --2- 172.21.15.1:0/1518357149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 0x7f541c0cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.976+0000 7f5423fe3640 1 -- 172.21.15.1:0/1518357149 >> 172.21.15.1:0/1518357149 conn(0x7f541c0d2360 msgr2=0x7f541c0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:36.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.976+0000 7f5423fe3640 1 -- 172.21.15.1:0/1518357149 shutdown_connections 2024-05-09T05:12:36.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.976+0000 7f5423fe3640 1 -- 172.21.15.1:0/1518357149 wait complete. 2024-05-09T05:12:36.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.978+0000 7f5423fe3640 1 Processor -- start 2024-05-09T05:12:36.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.978+0000 7f5423fe3640 1 -- start start 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f5423fe3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 0x7f541c1f6230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f5423fe3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0ce010 0x7f541c15d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f5423fe3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c160460 0x7f541c0d0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f5423fe3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f541c0cc750 con 0x7f541c0ce010 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f5423fe3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f541c15cd50 con 0x7f541c160460 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f5423fe3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f541c0cff10 con 0x7f541c0b4ec0 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f542157e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0ce010 0x7f541c15d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f542157e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0ce010 0x7f541c15d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34388/0 (socket says 172.21.15.1:34388) 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f542157e640 1 -- 172.21.15.1:0/3195541189 learned_addr learned my addr 172.21.15.1:0/3195541189 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f5421d7f640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 0x7f541c1f6230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f5422580640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c160460 0x7f541c0d0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f542157e640 1 -- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 msgr2=0x7f541c1f6230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f542157e640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c0b4ec0 0x7f541c1f6230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f542157e640 1 -- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c160460 msgr2=0x7f541c0d0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f542157e640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f541c160460 0x7f541c0d0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.980+0000 7f542157e640 1 -- 172.21.15.1:0/3195541189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c1f38c0 con 0x7f541c0ce010 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.981+0000 7f542157e640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0ce010 0x7f541c15d970 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f540c00c440 tx=0x7f540c00c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.981+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c017070 con 0x7f541c0ce010 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.981+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f540c005ce0 con 0x7f541c0ce010 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.981+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c1f6770 con 0x7f541c0ce010 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.981+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c007270 con 0x7f541c0ce010 2024-05-09T05:12:36.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.981+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c1f3020 con 0x7f541c0ce010 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.982+0000 7f5423fe3640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15c230 0x7f541c1f31f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.982+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f541c06f440 con 0x7f541c15c230 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.982+0000 7f5421d7f640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15c230 0x7f541c1f31f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.983+0000 7f5421d7f640 1 -- 172.21.15.1:0/3195541189 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f541c050870 con 0x7f541c15c230 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.983+0000 7f5421d7f640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15c230 0x7f541c1f31f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5418011890 tx=0x7f5418011d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.983+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f540c01a070 con 0x7f541c0ce010 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f54127fc640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f54000762d0 0x7f54000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f540c096790 con 0x7f541c0ce010 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f5422580640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f54000762d0 0x7f54000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418019070 con 0x7f541c15c230 2024-05-09T05:12:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5418004240 con 0x7f541c15c230 2024-05-09T05:12:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15c230 msgr2=0x7f541c1f31f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f54127fc640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15c230 0x7f541c1f31f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5418011890 tx=0x7f5418011d50 comp rx=0 tx=0).stop 2024-05-09T05:12:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15c230 msgr2=0x7f541c1f31f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.984+0000 7f54127fc640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c15c230 0x7f541c1f31f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.985+0000 7f5422580640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f54000762d0 0x7f54000787e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5414034000 tx=0x7f541402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.992+0000 7f5423fe3640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c14f4b0 0x7f541c14f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.992+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f541c1f3730 con 0x7f541c14f4b0 2024-05-09T05:12:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.993+0000 7f5421d7f640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c14f4b0 0x7f541c14f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:36.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.993+0000 7f5421d7f640 1 -- 172.21.15.1:0/3195541189 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f541c1f2ad0 con 0x7f541c14f4b0 2024-05-09T05:12:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.993+0000 7f5421d7f640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c14f4b0 0x7f541c14f8c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5418011890 tx=0x7f54180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.993+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5418019040 con 0x7f541c14f4b0 2024-05-09T05:12:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f5418003da0 con 0x7f541c14f4b0 2024-05-09T05:12:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c14f4b0 msgr2=0x7f541c14f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:36.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f54127fc640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c14f4b0 0x7f541c14f8c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5418011890 tx=0x7f54180046c0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f54127fc640 1 -- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c14f4b0 msgr2=0x7f541c14f8c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f54127fc640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f541c14f4b0 0x7f541c14f8c0 secure :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f5418011890 tx=0x7f54180046c0 comp rx=0 tx=0).stop 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 56, 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 314991, 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:12:36.995 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 13, 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:36.996 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:36.997 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:12:36.998 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:36.999 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:37.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:37.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:37.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.003 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:37.004 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.005 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:37.006 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:12:37.007 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:12:37.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f54000762d0 msgr2=0x7f54000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5423fe3640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f54000762d0 0x7f54000787e0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5414034000 tx=0x7f541402f000 comp rx=0 tx=0).stop 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0ce010 msgr2=0x7f541c15d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5423fe3640 1 --2- 172.21.15.1:0/3195541189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f541c0ce010 0x7f541c15d970 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f540c00c440 tx=0x7f540c00c900 comp rx=0 tx=0).stop 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5422580640 1 -- 172.21.15.1:0/3195541189 reap_dead start 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5422580640 1 -- 172.21.15.1:0/3195541189 reap_dead start 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 shutdown_connections 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 >> 172.21.15.1:0/3195541189 conn(0x7f541c0d2360 msgr2=0x7f541c155960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 shutdown_connections 2024-05-09T05:12:37.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:36.994+0000 7f5423fe3640 1 -- 172.21.15.1:0/3195541189 wait complete. 2024-05-09T05:12:37.009 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:12:37.010 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-05-09T05:12:37.010 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['c'] 2024-05-09T05:12:37.010 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-05-09T05:12:37.010 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-05-09T05:12:37.010 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T05:12:37.010 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:12:37.023 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T05:12:37.023 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-05-09T05:12:37.023 INFO:tasks.ceph.mon.c:Sent signal 19 2024-05-09T05:12:37.023 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-05-09T05:12:52.024 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-05-09T05:12:52.062 INFO:tasks.ceph.mon.c:Sent signal 18 2024-05-09T05:12:52.062 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:12:52.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:12:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.189+0000 7fcd6e8bd640 1 Processor -- start 2024-05-09T05:12:52.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.190+0000 7fcd6e8bd640 1 -- start start 2024-05-09T05:12:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6e8bd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 0x7fcd680c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6e8bd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6e8bd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd6806de20 con 0x7fcd6806efa0 2024-05-09T05:12:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6e8bd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcd68156d10 con 0x7fcd68156900 2024-05-09T05:12:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6e8bd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcd6815c9a0 con 0x7fcd680c7870 2024-05-09T05:12:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd67fff640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7fcd6806efa0 legacy=0x7fcd680cc560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:12:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd677fe640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 msgr2=0x7fcd680c7330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:52.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd677fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 0x7fcd680c7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6ce5a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6ce5a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:52800/0 (socket says 172.21.15.1:52800) 2024-05-09T05:12:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 learned_addr learned my addr 172.21.15.1:0/3082013048 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:12:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:52.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.191+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.392+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 msgr2=0x7fcd680c7330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.392+0000 7fcd67fff640 1 -- 172.21.15.1:0/3082013048 >> v1:172.21.15.1:6789/0 conn(0x7fcd6806efa0 legacy=0x7fcd680cc560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:12:52.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.392+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 0x7fcd680c7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:12:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.392+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.392+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:12:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.392+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.392+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:52.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.392+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:12:52.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.793+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 msgr2=0x7fcd680c7330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:52.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.793+0000 7fcd67fff640 1 -- 172.21.15.1:0/3082013048 >> v1:172.21.15.1:6789/0 conn(0x7fcd6806efa0 legacy=0x7fcd680cc560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:12:52.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.793+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 0x7fcd680c7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:12:52.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.793+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:52.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.793+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:12:52.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.793+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:52.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.793+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:52.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:52.793+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:12:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:53.594+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 msgr2=0x7fcd680c7330 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:53.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:53.594+0000 7fcd67fff640 1 -- 172.21.15.1:0/3082013048 >> v1:172.21.15.1:6789/0 conn(0x7fcd6806efa0 legacy=0x7fcd680cc560 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:12:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:53.594+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 0x7fcd680c7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:12:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:53.594+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:53.595+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:12:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:53.595+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:53.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:53.595+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:53.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:53.595+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:12:55.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.191+0000 7fcd65ffb640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:55.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.191+0000 7fcd65ffb640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd680c7870 0x7fcd680c7c80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:55.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.191+0000 7fcd65ffb640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 msgr2=0x7fcd680c7330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:55.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.191+0000 7fcd65ffb640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.1:3300/0 conn(0x7fcd68156900 0x7fcd680c7330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:55.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.191+0000 7fcd65ffb640 1 -- 172.21.15.1:0/3082013048 >> v1:172.21.15.1:6789/0 conn(0x7fcd6806efa0 legacy=0x7fcd680cc560 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:55.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.191+0000 7fcd65ffb640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:55.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.191+0000 7fcd65ffb640 1 -- 172.21.15.1:0/3082013048 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd680b51a0 con 0x7fcd540036a0 2024-05-09T05:12:55.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.191+0000 7fcd65ffb640 1 -- 172.21.15.1:0/3082013048 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fcd68156d10 con 0x7fcd54007430 2024-05-09T05:12:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.192+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.192+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:12:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.192+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.192+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:55.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.192+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.393+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.393+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:12:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.393+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.393+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:55.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.393+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:12:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.794+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.794+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:12:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.794+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.794+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:55.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:55.794+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:12:56.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:56.595+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:56.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:56.595+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:12:56.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:56.595+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:12:56.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:56.595+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:12:56.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:56.595+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:12:57.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.074+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2393404600 0 0) 0x7fcd680b51a0 con 0x7fcd540036a0 2024-05-09T05:12:57.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.074+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcd50003520 con 0x7fcd540036a0 2024-05-09T05:12:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1509921 0 0) 0x7fcd50003520 con 0x7fcd540036a0 2024-05-09T05:12:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 msgr2=0x7fcd54009940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 --2- 172.21.15.1:0/3082013048 >> v2:172.21.15.196:3300/0 conn(0x7fcd54007430 0x7fcd54009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd6806de20 con 0x7fcd540036a0 2024-05-09T05:12:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fcd58004050 con 0x7fcd540036a0 2024-05-09T05:12:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 >> v1:172.21.15.196:6789/0 conn(0x7fcd540036a0 legacy=0x7fcd54005b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 reap_dead start 2024-05-09T05:12:57.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd500039b0 0x7fcd50003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd50004800 0x7fcd50004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50005380 0x7fcd5000ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcd68156d10 con 0x7fcd50005380 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcd5000c320 con 0x7fcd500039b0 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcd5000c520 con 0x7fcd50004800 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50005380 msgr2=0x7fcd5000ba40 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd6ce5a640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50005380 0x7fcd5000ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd67fff640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd500039b0 0x7fcd50003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd50004800 0x7fcd50004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd500039b0 msgr2=0x7fcd50003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd500039b0 0x7fcd50003e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50005380 msgr2=0x7fcd5000ba40 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:57.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50005380 0x7fcd5000ba40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd677fe640 1 -- 172.21.15.1:0/3082013048 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd50003520 con 0x7fcd50004800 2024-05-09T05:12:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.077+0000 7fcd67fff640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd500039b0 0x7fcd50003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:12:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.078+0000 7fcd677fe640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd50004800 0x7fcd50004c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcd6808fd00 tx=0x7fcd50005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc27b4feb51b9ce7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.079+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c031070 con 0x7fcd50004800 2024-05-09T05:12:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.079+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd5c02be30 con 0x7fcd50004800 2024-05-09T05:12:57.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.079+0000 7fcd66ffd640 1 -- 172.21.15.1:0/3082013048 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd5c009b40 con 0x7fcd50004800 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd50004800 msgr2=0x7fcd50004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd50004800 0x7fcd50004c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcd6808fd00 tx=0x7fcd50005830 comp rx=0 tx=0).stop 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/3082013048 shutdown_connections 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50005380 0x7fcd5000ba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd50004800 0x7fcd50004c10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/3082013048 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd500039b0 0x7fcd50003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/3082013048 >> 172.21.15.1:0/3082013048 conn(0x7fcd68068d90 msgr2=0x7fcd680d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/3082013048 shutdown_connections 2024-05-09T05:12:57.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.080+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/3082013048 wait complete. 2024-05-09T05:12:57.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.082+0000 7fcd6e8bd640 1 Processor -- start 2024-05-09T05:12:57.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.083+0000 7fcd6e8bd640 1 -- start start 2024-05-09T05:12:57.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6e8bd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd500039b0 0x7fcd681f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6e8bd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50004800 0x7fcd681e6b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6e8bd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd50005380 0x7fcd680cd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6e8bd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6815cc40 con 0x7fcd50004800 2024-05-09T05:12:57.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6e8bd640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcd6806db40 con 0x7fcd50005380 2024-05-09T05:12:57.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6e8bd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcd6815cf20 con 0x7fcd500039b0 2024-05-09T05:12:57.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd677fe640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50004800 msgr2=0x7fcd681e6b30 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:57.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd677fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50004800 0x7fcd681e6b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6ce5a640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd50005380 0x7fcd680cd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd67fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd500039b0 0x7fcd681f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6ce5a640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd50005380 0x7fcd680cd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43728/0 (socket says 172.21.15.1:43728) 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd67fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd500039b0 0x7fcd681f31a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:52850/0 (socket says 172.21.15.1:52850) 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.084+0000 7fcd6ce5a640 1 -- 172.21.15.1:0/115215187 learned_addr learned my addr 172.21.15.1:0/115215187 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.085+0000 7fcd67fff640 1 -- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd50005380 msgr2=0x7fcd680cd460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.085+0000 7fcd67fff640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd50005380 0x7fcd680cd460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.085+0000 7fcd67fff640 1 -- 172.21.15.1:0/115215187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50004800 msgr2=0x7fcd681e6b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.085+0000 7fcd67fff640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50004800 0x7fcd681e6b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.085+0000 7fcd67fff640 1 -- 172.21.15.1:0/115215187 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd681e96d0 con 0x7fcd500039b0 2024-05-09T05:12:57.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.085+0000 7fcd67fff640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd500039b0 0x7fcd681f31a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800cad0 tx=0x7fcd5800cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.088+0000 7fcd64ff9640 1 -- 172.21.15.1:0/115215187 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58004e60 con 0x7fcd500039b0 2024-05-09T05:12:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.088+0000 7fcd64ff9640 1 -- 172.21.15.1:0/115215187 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd5800ae20 con 0x7fcd500039b0 2024-05-09T05:12:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.088+0000 7fcd64ff9640 1 -- 172.21.15.1:0/115215187 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd58016670 con 0x7fcd500039b0 2024-05-09T05:12:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.088+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd68157540 con 0x7fcd500039b0 2024-05-09T05:12:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.088+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd681e8bf0 con 0x7fcd500039b0 2024-05-09T05:12:57.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.089+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd68050870 con 0x7fcd500039b0 2024-05-09T05:12:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.090+0000 7fcd64ff9640 1 -- 172.21.15.1:0/115215187 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcd5801d020 con 0x7fcd500039b0 2024-05-09T05:12:57.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.090+0000 7fcd64ff9640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd54076990 0x7fcd54078ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.091+0000 7fcd64ff9640 1 -- 172.21.15.1:0/115215187 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcd580978d0 con 0x7fcd500039b0 2024-05-09T05:12:57.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.091+0000 7fcd677fe640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd54076990 0x7fcd54078ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.091+0000 7fcd677fe640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd54076990 0x7fcd54078ea0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c035000 tx=0x7fcd5c00f040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.096+0000 7fcd64ff9640 1 -- 172.21.15.1:0/115215187 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcd580612a0 con 0x7fcd500039b0 2024-05-09T05:12:57.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.372+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcd680c8cd0 con 0x7fcd500039b0 2024-05-09T05:12:57.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.373+0000 7fcd64ff9640 1 -- 172.21.15.1:0/115215187 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fcd58007020 con 0x7fcd500039b0 2024-05-09T05:12:57.373 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":58,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:12:57.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd54076990 msgr2=0x7fcd54078ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd54076990 0x7fcd54078ea0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fcd5c035000 tx=0x7fcd5c00f040 comp rx=0 tx=0).stop 2024-05-09T05:12:57.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd500039b0 msgr2=0x7fcd681f31a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd500039b0 0x7fcd681f31a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fcd5800cad0 tx=0x7fcd5800cf90 comp rx=0 tx=0).stop 2024-05-09T05:12:57.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 shutdown_connections 2024-05-09T05:12:57.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd54076990 0x7fcd54078ea0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd50005380 0x7fcd680cd460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd50004800 0x7fcd681e6b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 --2- 172.21.15.1:0/115215187 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd500039b0 0x7fcd681f31a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 >> 172.21.15.1:0/115215187 conn(0x7fcd68068d90 msgr2=0x7fcd680c7c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:57.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 shutdown_connections 2024-05-09T05:12:57.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.375+0000 7fcd6e8bd640 1 -- 172.21.15.1:0/115215187 wait complete. 2024-05-09T05:12:57.387 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:12:57.387 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:12:57.387 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:12:57.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.502+0000 7f05a673f640 1 Processor -- start 2024-05-09T05:12:57.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.502+0000 7f05a673f640 1 -- start start 2024-05-09T05:12:57.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.503+0000 7f05a673f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f05a00b4ec0 0x7f05a00cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.503+0000 7f05a673f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a00ce010 0x7f05a015fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.503+0000 7f05a673f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f05a015c7b0 con 0x7f05a0160290 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.503+0000 7f05a673f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f05a015d0c0 con 0x7f05a00b4ec0 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.503+0000 7f05a673f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f05a00cca50 con 0x7f05a00ce010 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.503+0000 7f059ffff640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7f05a00b4ec0 msgr2=0x7f05a00cf9d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.503+0000 7f059ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f05a00b4ec0 0x7f05a00cf9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f05a4cdc640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f05a0160290 0x7f05a01606a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:40342/0 (socket says 172.21.15.1:40342) 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f05a4cdc640 1 -- 172.21.15.1:0/67368745 learned_addr learned my addr 172.21.15.1:0/67368745 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f059f7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a00ce010 0x7f05a015fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f059effd640 1 -- 172.21.15.1:0/67368745 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 427369273 0 0) 0x7f05a015c7b0 con 0x7f05a0160290 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f059effd640 1 -- 172.21.15.1:0/67368745 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0588003520 con 0x7f05a0160290 2024-05-09T05:12:57.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f059f7fe640 1 -- 172.21.15.1:0/67368745 >> v2:172.21.15.1:3300/0 conn(0x7f05a00b4ec0 msgr2=0x7f05a00cf9d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:57.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f059f7fe640 1 --2- 172.21.15.1:0/67368745 >> v2:172.21.15.1:3300/0 conn(0x7f05a00b4ec0 0x7f05a00cf9d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f059f7fe640 1 -- 172.21.15.1:0/67368745 >> v1:172.21.15.196:6789/0 conn(0x7f05a0160290 legacy=0x7f05a01606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.504+0000 7f059f7fe640 1 -- 172.21.15.1:0/67368745 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a00ccd30 con 0x7f05a00ce010 2024-05-09T05:12:57.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.505+0000 7f059f7fe640 1 --2- 172.21.15.1:0/67368745 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a00ce010 0x7f05a015fd50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f05940097a0 tx=0x7f059402a230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3e703832f9331b16 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.505+0000 7f059effd640 1 -- 172.21.15.1:0/67368745 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0594037070 con 0x7f05a00ce010 2024-05-09T05:12:57.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.505+0000 7f059effd640 1 -- 172.21.15.1:0/67368745 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0594032440 con 0x7f05a00ce010 2024-05-09T05:12:57.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.505+0000 7f059effd640 1 -- 172.21.15.1:0/67368745 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05940327f0 con 0x7f05a00ce010 2024-05-09T05:12:57.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.506+0000 7f05a673f640 1 -- 172.21.15.1:0/67368745 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a00ce010 msgr2=0x7f05a015fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.506+0000 7f05a673f640 1 --2- 172.21.15.1:0/67368745 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a00ce010 0x7f05a015fd50 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f05940097a0 tx=0x7f059402a230 comp rx=0 tx=0).stop 2024-05-09T05:12:57.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.506+0000 7f05a673f640 1 -- 172.21.15.1:0/67368745 shutdown_connections 2024-05-09T05:12:57.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.506+0000 7f05a673f640 1 --2- 172.21.15.1:0/67368745 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a00ce010 0x7f05a015fd50 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.506+0000 7f05a673f640 1 --2- 172.21.15.1:0/67368745 >> v2:172.21.15.1:3300/0 conn(0x7f05a00b4ec0 0x7f05a00cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.506+0000 7f05a673f640 1 -- 172.21.15.1:0/67368745 >> 172.21.15.1:0/67368745 conn(0x7f05a00d2360 msgr2=0x7f05a00ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:57.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.506+0000 7f05a673f640 1 -- 172.21.15.1:0/67368745 shutdown_connections 2024-05-09T05:12:57.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.506+0000 7f05a673f640 1 -- 172.21.15.1:0/67368745 wait complete. 2024-05-09T05:12:57.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.507+0000 7f05a673f640 1 Processor -- start 2024-05-09T05:12:57.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.508+0000 7f05a673f640 1 -- start start 2024-05-09T05:12:57.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f05a673f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a00b4ec0 0x7f05a015ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f05a673f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a00ce010 0x7f05a015f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f05a673f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a0160290 0x7f05a006de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f05a673f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f05a00cc750 con 0x7f05a00ce010 2024-05-09T05:12:57.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f05a673f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f05a015cd50 con 0x7f05a00b4ec0 2024-05-09T05:12:57.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f05a673f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f05a00cff10 con 0x7f05a0160290 2024-05-09T05:12:57.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f059f7fe640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a00ce010 msgr2=0x7f05a015f910 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f059f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a00ce010 0x7f05a015f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f059ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a00b4ec0 0x7f05a015ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f059ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a00b4ec0 0x7f05a015ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43730/0 (socket says 172.21.15.1:43730) 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f059ffff640 1 -- 172.21.15.1:0/963642161 learned_addr learned my addr 172.21.15.1:0/963642161 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.509+0000 7f05a4cdc640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a0160290 0x7f05a006de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f05a4cdc640 1 -- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a00b4ec0 msgr2=0x7f05a015ac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f05a4cdc640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a00b4ec0 0x7f05a015ac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f05a4cdc640 1 -- 172.21.15.1:0/963642161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a00ce010 msgr2=0x7f05a015f910 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f05a4cdc640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f05a00ce010 0x7f05a015f910 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f05a4cdc640 1 -- 172.21.15.1:0/963642161 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05a015b1b0 con 0x7f05a0160290 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f059ffff640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a00b4ec0 0x7f05a015ac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f05a4cdc640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a0160290 0x7f05a006de70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f05a006f5b0 tx=0x7f059800ce40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f059800ab10 con 0x7f05a0160290 2024-05-09T05:12:57.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f059800acb0 con 0x7f05a0160290 2024-05-09T05:12:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0598012620 con 0x7f05a0160290 2024-05-09T05:12:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05a0152700 con 0x7f05a0160290 2024-05-09T05:12:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.510+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f05a006d6d0 con 0x7f05a0160290 2024-05-09T05:12:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.511+0000 7f05a673f640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006f020 0x7f05a006e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.511+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f05a015b680 con 0x7f05a006f020 2024-05-09T05:12:57.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.511+0000 7f059ffff640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006f020 0x7f05a006e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.511+0000 7f059ffff640 1 -- 172.21.15.1:0/963642161 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f05a006f700 con 0x7f05a006f020 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.512+0000 7f059ffff640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006f020 0x7f05a006e3b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f05900053c0 tx=0x7f0590005880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.512+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0590016070 con 0x7f05a006f020 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.512+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f05980127c0 con 0x7f05a0160290 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.512+0000 7f059cff9640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f058c0761d0 0x7f058c0786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.513+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0598097a80 con 0x7f05a0160290 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.513+0000 7f059f7fe640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f058c0761d0 0x7f058c0786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.513+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f059000eaf0 con 0x7f05a006f020 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.513+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006f020 msgr2=0x7f05a006e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.513+0000 7f059cff9640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006f020 0x7f05a006e3b0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f05900053c0 tx=0x7f0590005880 comp rx=0 tx=0).stop 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.513+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006f020 msgr2=0x7f05a006e3b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.513+0000 7f059cff9640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006f020 0x7f05a006e3b0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.513+0000 7f059f7fe640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f058c0761d0 0x7f058c0786e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0594034000 tx=0x7f059402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.519+0000 7f05a673f640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006d8a0 0x7f05a01f62b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.519+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f05a006dcb0 con 0x7f05a006d8a0 2024-05-09T05:12:57.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.520+0000 7f059ffff640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006d8a0 0x7f05a01f62b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.520+0000 7f059ffff640 1 -- 172.21.15.1:0/963642161 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f05a00b6230 con 0x7f05a006d8a0 2024-05-09T05:12:57.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.520+0000 7f059ffff640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006d8a0 0x7f05a01f62b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f05900053c0 tx=0x7f05900059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.520+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05900128c0 con 0x7f05a006d8a0 2024-05-09T05:12:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4541 (secure 0 0 0) 0x7f05900128c0 con 0x7f05a006d8a0 2024-05-09T05:12:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006d8a0 msgr2=0x7f05a01f62b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f059cff9640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006d8a0 0x7f05a01f62b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f05900053c0 tx=0x7f05900059d0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f059cff9640 1 -- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006d8a0 msgr2=0x7f05a01f62b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:57.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f059cff9640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f05a006d8a0 0x7f05a01f62b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 58, 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 39800, 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:12:57.522 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:57.523 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:57.524 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:12:57.525 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:57.526 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:57.527 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.528 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:57.529 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:57.530 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.531 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.532 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.533 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.534 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:12:57.534 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:12:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f058c0761d0 msgr2=0x7f058c0786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a673f640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f058c0761d0 0x7f058c0786e0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f0594034000 tx=0x7f059402e000 comp rx=0 tx=0).stop 2024-05-09T05:12:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a0160290 msgr2=0x7f05a006de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a673f640 1 --2- 172.21.15.1:0/963642161 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f05a0160290 0x7f05a006de70 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f05a006f5b0 tx=0x7f059800ce40 comp rx=0 tx=0).stop 2024-05-09T05:12:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a4cdc640 1 -- 172.21.15.1:0/963642161 reap_dead start 2024-05-09T05:12:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a4cdc640 1 -- 172.21.15.1:0/963642161 reap_dead start 2024-05-09T05:12:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 shutdown_connections 2024-05-09T05:12:57.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 >> 172.21.15.1:0/963642161 conn(0x7f05a00d2360 msgr2=0x7f05a00d0120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:57.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 shutdown_connections 2024-05-09T05:12:57.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.521+0000 7f05a673f640 1 -- 172.21.15.1:0/963642161 wait complete. 2024-05-09T05:12:57.536 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:12:57.536 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:12:57.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:12:57.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.651+0000 7eff28661640 1 Processor -- start 2024-05-09T05:12:57.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.651+0000 7eff28661640 1 -- start start 2024-05-09T05:12:57.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff28661640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7eff200c6990 0x7eff20156c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff28661640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff200cb010 0x7eff200cc840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff28661640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200cb420 0x7eff200ccd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff28661640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7eff2015d570 con 0x7eff200cb010 2024-05-09T05:12:57.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff28661640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7eff2015c9a0 con 0x7eff200c6990 2024-05-09T05:12:57.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff28661640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7eff2015cc40 con 0x7eff200cb420 2024-05-09T05:12:57.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff25bfc640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7eff200cb010 msgr2=0x7eff200cc840 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:57.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff25bfc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7eff200cb010 0x7eff200cc840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff263fd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7eff200c6990 0x7eff20156c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff26bfe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200cb420 0x7eff200ccd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff263fd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7eff200c6990 0x7eff20156c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:43754/0 (socket says 172.21.15.1:43754) 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.653+0000 7eff263fd640 1 -- 172.21.15.1:0/695778803 learned_addr learned my addr 172.21.15.1:0/695778803 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.654+0000 7eff263fd640 1 -- 172.21.15.1:0/695778803 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200cb420 msgr2=0x7eff200ccd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.654+0000 7eff263fd640 1 --2- 172.21.15.1:0/695778803 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200cb420 0x7eff200ccd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.654+0000 7eff263fd640 1 -- 172.21.15.1:0/695778803 >> v2:172.21.15.1:3300/0 conn(0x7eff200cb010 msgr2=0x7eff200cc840 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.654+0000 7eff263fd640 1 --2- 172.21.15.1:0/695778803 >> v2:172.21.15.1:3300/0 conn(0x7eff200cb010 0x7eff200cc840 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.654+0000 7eff263fd640 1 -- 172.21.15.1:0/695778803 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2015cf20 con 0x7eff200c6990 2024-05-09T05:12:57.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.654+0000 7eff263fd640 1 --2- 172.21.15.1:0/695778803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff200c6990 0x7eff20156c30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff200747f0 tx=0x7eff1002a750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b500916b9c97142 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.654+0000 7eff253fb640 1 -- 172.21.15.1:0/695778803 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff10036070 con 0x7eff200c6990 2024-05-09T05:12:57.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff253fb640 1 -- 172.21.15.1:0/695778803 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff10031600 con 0x7eff200c6990 2024-05-09T05:12:57.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff253fb640 1 -- 172.21.15.1:0/695778803 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff100319d0 con 0x7eff200c6990 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 -- 172.21.15.1:0/695778803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff200c6990 msgr2=0x7eff20156c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 --2- 172.21.15.1:0/695778803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff200c6990 0x7eff20156c30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7eff200747f0 tx=0x7eff1002a750 comp rx=0 tx=0).stop 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 -- 172.21.15.1:0/695778803 shutdown_connections 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 --2- 172.21.15.1:0/695778803 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200cb420 0x7eff200ccd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 --2- 172.21.15.1:0/695778803 >> v2:172.21.15.1:3300/0 conn(0x7eff200cb010 0x7eff200cc840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 --2- 172.21.15.1:0/695778803 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff200c6990 0x7eff20156c30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 -- 172.21.15.1:0/695778803 >> 172.21.15.1:0/695778803 conn(0x7eff20068830 msgr2=0x7eff20068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 -- 172.21.15.1:0/695778803 shutdown_connections 2024-05-09T05:12:57.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.655+0000 7eff28661640 1 -- 172.21.15.1:0/695778803 wait complete. 2024-05-09T05:12:57.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.657+0000 7eff28661640 1 Processor -- start 2024-05-09T05:12:57.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.657+0000 7eff28661640 1 -- start start 2024-05-09T05:12:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff28661640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c6990 0x7eff200c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff28661640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff200cb010 0x7eff200ceb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff28661640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff200cb420 0x7eff2014d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff28661640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff2006d870 con 0x7eff200cb010 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff28661640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7eff200cb980 con 0x7eff200cb420 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff28661640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7eff2006d9f0 con 0x7eff200c6990 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff25bfc640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff200cb010 msgr2=0x7eff200ceb80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff25bfc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff200cb010 0x7eff200ceb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.658+0000 7eff263fd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c6990 0x7eff200c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff263fd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c6990 0x7eff200c6da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:52908/0 (socket says 172.21.15.1:52908) 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff263fd640 1 -- 172.21.15.1:0/2375286368 learned_addr learned my addr 172.21.15.1:0/2375286368 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff26bfe640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff200cb420 0x7eff2014d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff263fd640 1 -- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff200cb420 msgr2=0x7eff2014d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff263fd640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff200cb420 0x7eff2014d3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff263fd640 1 -- 172.21.15.1:0/2375286368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff200cb010 msgr2=0x7eff200ceb80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:12:57.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff263fd640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff200cb010 0x7eff200ceb80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff263fd640 1 -- 172.21.15.1:0/2375286368 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff2015ad50 con 0x7eff200c6990 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff263fd640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c6990 0x7eff200c6da0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff10034040 tx=0x7eff10004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff10041020 con 0x7eff200c6990 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff100094d0 con 0x7eff200c6990 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff2015aff0 con 0x7eff200c6990 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff10036040 con 0x7eff200c6990 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.659+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7eff2015b590 con 0x7eff200c6990 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.660+0000 7eff28661640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c9b50 0x7eff2014d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.660+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7eff200c7370 con 0x7eff200c9b50 2024-05-09T05:12:57.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.660+0000 7eff25bfc640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c9b50 0x7eff2014d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.661+0000 7eff25bfc640 1 -- 172.21.15.1:0/2375286368 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff20154d00 con 0x7eff200c9b50 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.661+0000 7eff25bfc640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c9b50 0x7eff2014d920 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff14003980 tx=0x7eff1400b8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.661+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14016070 con 0x7eff200c9b50 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.661+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7eff10044020 con 0x7eff200c6990 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.662+0000 7eff1effd640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff0c0761d0 0x7eff0c0786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.662+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7eff100b8180 con 0x7eff200c6990 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.662+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7eff14003dc0 con 0x7eff200c9b50 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.662+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c9b50 msgr2=0x7eff2014d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.662+0000 7eff1effd640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c9b50 0x7eff2014d920 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff14003980 tx=0x7eff1400b8e0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.662+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c9b50 msgr2=0x7eff2014d920 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.662+0000 7eff1effd640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c9b50 0x7eff2014d920 secure :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7eff14003980 tx=0x7eff1400b8e0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.662+0000 7eff26bfe640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff0c0761d0 0x7eff0c0786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.663+0000 7eff26bfe640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff0c0761d0 0x7eff0c0786e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7eff200caa40 tx=0x7eff18009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.669+0000 7eff28661640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff2014df90 0x7eff201f5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:12:57.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.669+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7eff20050870 con 0x7eff2014df90 2024-05-09T05:12:57.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.669+0000 7eff25bfc640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff2014df90 0x7eff201f5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:12:57.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.669+0000 7eff25bfc640 1 -- 172.21.15.1:0/2375286368 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7eff200cbbe0 con 0x7eff2014df90 2024-05-09T05:12:57.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff25bfc640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff2014df90 0x7eff201f5530 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7eff14003980 tx=0x7eff140059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:12:57.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff14019b50 con 0x7eff2014df90 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5050 (secure 0 0 0) 0x7eff14019b50 con 0x7eff2014df90 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff2014df90 msgr2=0x7eff201f5530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff1effd640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff2014df90 0x7eff201f5530 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7eff14003980 tx=0x7eff140059d0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff1effd640 1 -- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff2014df90 msgr2=0x7eff201f5530 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff1effd640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff2014df90 0x7eff201f5530 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:12:57.671 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 58, 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 335668, 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:57.672 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:57.673 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:57.674 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:12:57.675 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:57.676 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:12:57.677 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.678 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:12:57.679 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.680 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:12:57.681 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.682 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 7 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.683 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:12:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:12:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff0c0761d0 msgr2=0x7eff0c0786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff28661640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff0c0761d0 0x7eff0c0786e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7eff200caa40 tx=0x7eff18009190 comp rx=0 tx=0).stop 2024-05-09T05:12:57.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c6990 msgr2=0x7eff200c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:12:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff28661640 1 --2- 172.21.15.1:0/2375286368 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff200c6990 0x7eff200c6da0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7eff10034040 tx=0x7eff10004a40 comp rx=0 tx=0).stop 2024-05-09T05:12:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff26bfe640 1 -- 172.21.15.1:0/2375286368 reap_dead start 2024-05-09T05:12:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff26bfe640 1 -- 172.21.15.1:0/2375286368 reap_dead start 2024-05-09T05:12:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 shutdown_connections 2024-05-09T05:12:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 >> 172.21.15.1:0/2375286368 conn(0x7eff20068830 msgr2=0x7eff20155a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:12:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 shutdown_connections 2024-05-09T05:12:57.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:12:57.670+0000 7eff28661640 1 -- 172.21.15.1:0/2375286368 wait complete. 2024-05-09T05:12:57.686 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:12:57.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:13:03.146 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:13:03.148 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:13:17.687 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-05-09T05:13:17.687 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-05-09T05:13:17.687 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T05:13:17.687 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T05:13:17.690 INFO:tasks.ceph.mon.a:Started 2024-05-09T05:13:17.690 INFO:tasks.mon_thrash:Sending STOP to mon c 2024-05-09T05:13:17.691 INFO:tasks.ceph.mon.c:Sent signal 19 2024-05-09T05:13:17.691 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-05-09T05:13:32.691 INFO:tasks.mon_thrash:Sending CONT to mon c 2024-05-09T05:13:32.692 INFO:tasks.ceph.mon.c:Sent signal 18 2024-05-09T05:13:32.692 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:13:32.692 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:13:32.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.823+0000 7f511ef72640 1 Processor -- start 2024-05-09T05:13:32.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.823+0000 7f511ef72640 1 -- start start 2024-05-09T05:13:32.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.824+0000 7f511ef72640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f511806f440 0x7f5118156b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:32.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.824+0000 7f511ef72640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51180ce4b0 0x7f51180ce8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.824+0000 7f511ef72640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f511815ca30 0x7f511815ce40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.824+0000 7f511ef72640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f5118157070 con 0x7f51180ce4b0 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.824+0000 7f511ef72640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f51180cd180 con 0x7f511815ca30 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.824+0000 7f511ef72640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f51180cee00 con 0x7f511806f440 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.824+0000 7f5117fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51180ce4b0 0x7f51180ce8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f5117fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f51180ce4b0 0x7f51180ce8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33006/0 (socket says 172.21.15.1:33006) 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f5117fff640 1 -- 172.21.15.1:0/4262841669 learned_addr learned my addr 172.21.15.1:0/4262841669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f511d50f640 1 --2- 172.21.15.1:0/4262841669 >> v2:172.21.15.196:3300/0 conn(0x7f511815ca30 0x7f511815ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f511cd0e640 1 --2- 172.21.15.1:0/4262841669 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f511806f440 0x7f5118156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f5117fff640 1 -- 172.21.15.1:0/4262841669 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f511806f440 msgr2=0x7f5118156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:32.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f5117fff640 1 --2- 172.21.15.1:0/4262841669 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f511806f440 0x7f5118156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:32.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f5117fff640 1 -- 172.21.15.1:0/4262841669 >> v2:172.21.15.196:3300/0 conn(0x7f511815ca30 msgr2=0x7f511815ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:32.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f5117fff640 1 --2- 172.21.15.1:0/4262841669 >> v2:172.21.15.196:3300/0 conn(0x7f511815ca30 0x7f511815ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:32.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.825+0000 7f5117fff640 1 -- 172.21.15.1:0/4262841669 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f511815ff70 con 0x7f51180ce4b0 2024-05-09T05:13:32.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.826+0000 7f5117fff640 1 --2- 172.21.15.1:0/4262841669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51180ce4b0 0x7f51180ce8c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f51180cefb0 tx=0x7f510402a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99112817067bc433 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.826+0000 7f51177fe640 1 -- 172.21.15.1:0/4262841669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104004030 con 0x7f51180ce4b0 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.826+0000 7f51177fe640 1 -- 172.21.15.1:0/4262841669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f51040041d0 con 0x7f51180ce4b0 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.826+0000 7f51177fe640 1 -- 172.21.15.1:0/4262841669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5104003610 con 0x7f51180ce4b0 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 -- 172.21.15.1:0/4262841669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51180ce4b0 msgr2=0x7f51180ce8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 --2- 172.21.15.1:0/4262841669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51180ce4b0 0x7f51180ce8c0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f51180cefb0 tx=0x7f510402a1e0 comp rx=0 tx=0).stop 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 -- 172.21.15.1:0/4262841669 shutdown_connections 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 --2- 172.21.15.1:0/4262841669 >> v2:172.21.15.196:3300/0 conn(0x7f511815ca30 0x7f511815ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 --2- 172.21.15.1:0/4262841669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f51180ce4b0 0x7f51180ce8c0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:32.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 --2- 172.21.15.1:0/4262841669 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f511806f440 0x7f5118156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 -- 172.21.15.1:0/4262841669 >> 172.21.15.1:0/4262841669 conn(0x7f51180b3670 msgr2=0x7f51180b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 -- 172.21.15.1:0/4262841669 shutdown_connections 2024-05-09T05:13:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.827+0000 7f511ef72640 1 -- 172.21.15.1:0/4262841669 wait complete. 2024-05-09T05:13:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.828+0000 7f511ef72640 1 Processor -- start 2024-05-09T05:13:32.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.828+0000 7f511ef72640 1 -- start start 2024-05-09T05:13:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511ef72640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f511806f440 0x7f5118151210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511ef72640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51180ce4b0 0x7f51181519a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511ef72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815ca30 0x7f51180b4740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:32.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511ef72640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5118160810 con 0x7f511815ca30 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511ef72640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5118160530 con 0x7f511806f440 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511ef72640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f511815d380 con 0x7f51180ce4b0 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511d50f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815ca30 0x7f51180b4740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511d50f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815ca30 0x7f51180b4740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33008/0 (socket says 172.21.15.1:33008) 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511d50f640 1 -- 172.21.15.1:0/1034917327 learned_addr learned my addr 172.21.15.1:0/1034917327 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511cd0e640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f511806f440 0x7f5118151210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f5117fff640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51180ce4b0 0x7f51181519a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511d50f640 1 -- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51180ce4b0 msgr2=0x7f51181519a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511d50f640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51180ce4b0 0x7f51181519a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511d50f640 1 -- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f511806f440 msgr2=0x7f5118151210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511d50f640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f511806f440 0x7f5118151210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511d50f640 1 -- 172.21.15.1:0/1034917327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51180b4c80 con 0x7f511815ca30 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.830+0000 7f511cd0e640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f511806f440 0x7f5118151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:13:32.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.831+0000 7f511d50f640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815ca30 0x7f51180b4740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5110007d80 tx=0x7f511000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.831+0000 7f51157fa640 1 -- 172.21.15.1:0/1034917327 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110014420 con 0x7f511815ca30 2024-05-09T05:13:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.831+0000 7f51157fa640 1 -- 172.21.15.1:0/1034917327 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5110019070 con 0x7f511815ca30 2024-05-09T05:13:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.831+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f511815dc30 con 0x7f511815ca30 2024-05-09T05:13:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.831+0000 7f51157fa640 1 -- 172.21.15.1:0/1034917327 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5110014800 con 0x7f511815ca30 2024-05-09T05:13:32.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.831+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f511815e1f0 con 0x7f511815ca30 2024-05-09T05:13:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.832+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5118050870 con 0x7f511815ca30 2024-05-09T05:13:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.832+0000 7f51157fa640 1 -- 172.21.15.1:0/1034917327 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f51100149a0 con 0x7f511815ca30 2024-05-09T05:13:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.833+0000 7f51157fa640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50fc0762d0 0x7f50fc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.833+0000 7f51157fa640 1 -- 172.21.15.1:0/1034917327 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5110097f60 con 0x7f511815ca30 2024-05-09T05:13:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.833+0000 7f511cd0e640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50fc0762d0 0x7f50fc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:32.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.834+0000 7f511cd0e640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50fc0762d0 0x7f50fc0787e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f510c00ba30 tx=0x7f510c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:32.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:32.838+0000 7f51157fa640 1 -- 172.21.15.1:0/1034917327 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f51100617a0 con 0x7f511815ca30 2024-05-09T05:13:33.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.117+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f51180c50d0 con 0x7f511815ca30 2024-05-09T05:13:33.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.118+0000 7f51157fa640 1 -- 172.21.15.1:0/1034917327 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f51100610a0 con 0x7f511815ca30 2024-05-09T05:13:33.119 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:13:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50fc0762d0 msgr2=0x7f50fc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50fc0762d0 0x7f50fc0787e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f510c00ba30 tx=0x7f510c008040 comp rx=0 tx=0).stop 2024-05-09T05:13:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815ca30 msgr2=0x7f51180b4740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815ca30 0x7f51180b4740 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5110007d80 tx=0x7f511000e4b0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 shutdown_connections 2024-05-09T05:13:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50fc0762d0 0x7f50fc0787e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f511815ca30 0x7f51180b4740 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f51180ce4b0 0x7f51181519a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 --2- 172.21.15.1:0/1034917327 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f511806f440 0x7f5118151210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 >> 172.21.15.1:0/1034917327 conn(0x7f51180b3670 msgr2=0x7f51180ca790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 shutdown_connections 2024-05-09T05:13:33.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.120+0000 7f511ef72640 1 -- 172.21.15.1:0/1034917327 wait complete. 2024-05-09T05:13:33.132 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:13:33.133 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:13:33.133 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:13:33.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.265+0000 7f3d1e814640 1 Processor -- start 2024-05-09T05:13:33.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.265+0000 7f3d1e814640 1 -- start start 2024-05-09T05:13:33.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.266+0000 7f3d1e814640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3d1815d130 0x7f3d181569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.266+0000 7f3d1e814640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d180c3ce0 con 0x7f3d180c48c0 2024-05-09T05:13:33.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.266+0000 7f3d1e814640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d180c3fc0 con 0x7f3d1806c6f0 2024-05-09T05:13:33.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.266+0000 7f3d1e814640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f3d1815d540 con 0x7f3d1815d130 2024-05-09T05:13:33.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.267+0000 7f3d1cdb1640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3d180c48c0 0x7f3d180c4cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52462/0 (socket says 172.21.15.1:52462) 2024-05-09T05:13:33.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.267+0000 7f3d1cdb1640 1 -- 172.21.15.1:0/3450104310 learned_addr learned my addr 172.21.15.1:0/3450104310 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:33.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.267+0000 7f3d17fff640 1 --2- 172.21.15.1:0/3450104310 >> v2:172.21.15.196:3300/0 conn(0x7f3d1815d130 0x7f3d181569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.267+0000 7f3d16ffd640 1 -- 172.21.15.1:0/3450104310 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1825033609 0 0) 0x7f3d180c3ce0 con 0x7f3d180c48c0 2024-05-09T05:13:33.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.267+0000 7f3d16ffd640 1 -- 172.21.15.1:0/3450104310 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3d00003520 con 0x7f3d180c48c0 2024-05-09T05:13:33.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.267+0000 7f3d17fff640 1 -- 172.21.15.1:0/3450104310 >> v1:172.21.15.196:6789/0 conn(0x7f3d1806c6f0 legacy=0x7f3d180716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.267+0000 7f3d17fff640 1 -- 172.21.15.1:0/3450104310 >> v1:172.21.15.1:6789/0 conn(0x7f3d180c48c0 legacy=0x7f3d180c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.267+0000 7f3d17fff640 1 -- 172.21.15.1:0/3450104310 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1806e4a0 con 0x7f3d1815d130 2024-05-09T05:13:33.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.268+0000 7f3d17fff640 1 --2- 172.21.15.1:0/3450104310 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d1815d130 0x7f3d181569c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3d080097a0 tx=0x7f3d0802a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=184371b886b0a35f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.268+0000 7f3d16ffd640 1 -- 172.21.15.1:0/3450104310 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08037070 con 0x7f3d1815d130 2024-05-09T05:13:33.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.268+0000 7f3d16ffd640 1 -- 172.21.15.1:0/3450104310 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d08032440 con 0x7f3d1815d130 2024-05-09T05:13:33.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.268+0000 7f3d16ffd640 1 -- 172.21.15.1:0/3450104310 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d080327f0 con 0x7f3d1815d130 2024-05-09T05:13:33.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.269+0000 7f3d1e814640 1 -- 172.21.15.1:0/3450104310 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d1815d130 msgr2=0x7f3d181569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.269+0000 7f3d1e814640 1 --2- 172.21.15.1:0/3450104310 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d1815d130 0x7f3d181569c0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f3d080097a0 tx=0x7f3d0802a230 comp rx=0 tx=0).stop 2024-05-09T05:13:33.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.269+0000 7f3d1e814640 1 -- 172.21.15.1:0/3450104310 shutdown_connections 2024-05-09T05:13:33.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.269+0000 7f3d1e814640 1 --2- 172.21.15.1:0/3450104310 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d1815d130 0x7f3d181569c0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.269+0000 7f3d1e814640 1 -- 172.21.15.1:0/3450104310 >> 172.21.15.1:0/3450104310 conn(0x7f3d180747b0 msgr2=0x7f3d180ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:33.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.269+0000 7f3d1e814640 1 -- 172.21.15.1:0/3450104310 shutdown_connections 2024-05-09T05:13:33.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.269+0000 7f3d1e814640 1 -- 172.21.15.1:0/3450104310 wait complete. 2024-05-09T05:13:33.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.271+0000 7f3d1e814640 1 Processor -- start 2024-05-09T05:13:33.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.271+0000 7f3d1e814640 1 -- start start 2024-05-09T05:13:33.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.272+0000 7f3d1e814640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d1806c6f0 0x7f3d181f3900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.272+0000 7f3d1e814640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d180c48c0 0x7f3d1814d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1e814640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1815d130 0x7f3d181535a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1e814640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1806cb00 con 0x7f3d1815d130 2024-05-09T05:13:33.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1e814640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1806e1a0 con 0x7f3d1806c6f0 2024-05-09T05:13:33.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1e814640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1806cc80 con 0x7f3d180c48c0 2024-05-09T05:13:33.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1815d130 0x7f3d181535a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1815d130 0x7f3d181535a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33010/0 (socket says 172.21.15.1:33010) 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 -- 172.21.15.1:0/923295497 learned_addr learned my addr 172.21.15.1:0/923295497 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d17fff640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d1806c6f0 0x7f3d181f3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d177fe640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d180c48c0 0x7f3d1814d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 -- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d180c48c0 msgr2=0x7f3d1814d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d180c48c0 0x7f3d1814d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 -- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d1806c6f0 msgr2=0x7f3d181f3900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d1806c6f0 0x7f3d181f3900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 -- 172.21.15.1:0/923295497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d1814ddc0 con 0x7f3d1815d130 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d177fe640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d180c48c0 0x7f3d1814d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d1cdb1640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1815d130 0x7f3d181535a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000cad0 tx=0x7f3d1000cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.273+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d1000ac30 con 0x7f3d1815d130 2024-05-09T05:13:33.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.274+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d1000add0 con 0x7f3d1815d130 2024-05-09T05:13:33.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.274+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d1815c0d0 con 0x7f3d1815d130 2024-05-09T05:13:33.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.274+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d10012600 con 0x7f3d1815d130 2024-05-09T05:13:33.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.274+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d1815b960 con 0x7f3d1815d130 2024-05-09T05:13:33.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.275+0000 7f3d1e814640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1814df90 0x7f3d18155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.275+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3d1815b500 con 0x7f3d1814df90 2024-05-09T05:13:33.278 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.275+0000 7f3d17fff640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1814df90 0x7f3d18155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.275+0000 7f3d17fff640 1 -- 172.21.15.1:0/923295497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d1815a940 con 0x7f3d1814df90 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.275+0000 7f3d17fff640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1814df90 0x7f3d18155960 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d180c43d0 tx=0x7f3d080046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.275+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08043020 con 0x7f3d1814df90 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3d100127a0 con 0x7f3d1815d130 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d14ff9640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d040761d0 0x7f3d040786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3d10097b40 con 0x7f3d1815d130 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d08038040 con 0x7f3d1814df90 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1814df90 msgr2=0x7f3d18155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d14ff9640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1814df90 0x7f3d18155960 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d180c43d0 tx=0x7f3d080046c0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1814df90 msgr2=0x7f3d18155960 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d14ff9640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1814df90 0x7f3d18155960 secure :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d180c43d0 tx=0x7f3d080046c0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.276+0000 7f3d177fe640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d040761d0 0x7f3d040786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.277+0000 7f3d177fe640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d040761d0 0x7f3d040786e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c007fd0 tx=0x7f3d0c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.284+0000 7f3d1e814640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d180b4790 0x7f3d18155ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.284+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3d181563e0 con 0x7f3d180b4790 2024-05-09T05:13:33.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.284+0000 7f3d17fff640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d180b4790 0x7f3d18155ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.284+0000 7f3d17fff640 1 -- 172.21.15.1:0/923295497 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d18050870 con 0x7f3d180b4790 2024-05-09T05:13:33.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.284+0000 7f3d17fff640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d180b4790 0x7f3d18155ea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d0802a270 tx=0x7f3d08008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d08037040 con 0x7f3d180b4790 2024-05-09T05:13:33.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f3d08003810 con 0x7f3d180b4790 2024-05-09T05:13:33.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d180b4790 msgr2=0x7f3d18155ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d14ff9640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d180b4790 0x7f3d18155ea0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f3d0802a270 tx=0x7f3d08008030 comp rx=0 tx=0).stop 2024-05-09T05:13:33.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d14ff9640 1 -- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d180b4790 msgr2=0x7f3d18155ea0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:33.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d14ff9640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d180b4790 0x7f3d18155ea0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.291 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:13:33.291 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:13:33.292 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:13:33.292 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:13:33.292 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 62, 2024-05-09T05:13:33.292 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 15480, 2024-05-09T05:13:33.292 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:13:33.293 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:13:33.293 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:13:33.293 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:13:33.293 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.293 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:13:33.294 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:33.294 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:13:33.294 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:13:33.294 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.294 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.294 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.294 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.295 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:13:33.296 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.297 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.298 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.299 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:13:33.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:13:33.302 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.303 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.304 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.305 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.305 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:13:33.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.305 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.305 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.305 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:13:33.305 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:13:33.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d040761d0 msgr2=0x7f3d040786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1e814640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3d040761d0 0x7f3d040786e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f3d0c007fd0 tx=0x7f3d0c009040 comp rx=0 tx=0).stop 2024-05-09T05:13:33.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1815d130 msgr2=0x7f3d181535a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1e814640 1 --2- 172.21.15.1:0/923295497 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d1815d130 0x7f3d181535a0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f3d1000cad0 tx=0x7f3d1000cf90 comp rx=0 tx=0).stop 2024-05-09T05:13:33.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1cdb1640 1 -- 172.21.15.1:0/923295497 reap_dead start 2024-05-09T05:13:33.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1cdb1640 1 -- 172.21.15.1:0/923295497 reap_dead start 2024-05-09T05:13:33.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 shutdown_connections 2024-05-09T05:13:33.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 >> 172.21.15.1:0/923295497 conn(0x7f3d180747b0 msgr2=0x7f3d18075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:33.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.285+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 shutdown_connections 2024-05-09T05:13:33.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.286+0000 7f3d1e814640 1 -- 172.21.15.1:0/923295497 wait complete. 2024-05-09T05:13:33.307 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:13:33.307 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:13:33.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:13:33.308 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:13:33.311 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:13:33.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.432+0000 7fde3de7e640 1 Processor -- start 2024-05-09T05:13:33.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.432+0000 7fde3de7e640 1 -- start start 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde3de7e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde380b4ec0 0x7fde380cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde3de7e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde3815c7b0 con 0x7fde380ce010 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde3de7e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fde380cca50 con 0x7fde38160460 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde3de7e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fde3815d0c0 con 0x7fde380b4ec0 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde2ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fde380ce010 0x7fde3815fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52476/0 (socket says 172.21.15.1:52476) 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde2ffff640 1 -- 172.21.15.1:0/2625997783 learned_addr learned my addr 172.21.15.1:0/2625997783 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde377fe640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde380b4ec0 0x7fde380cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 637121951 0 0) 0x7fde3815c7b0 con 0x7fde380ce010 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fde24003520 con 0x7fde380ce010 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.434+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2645415824 0 0) 0x7fde24003520 con 0x7fde380ce010 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde380b4ec0 msgr2=0x7fde380cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:33.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde380b4ec0 0x7fde380cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 >> v1:172.21.15.196:6789/0 conn(0x7fde38160460 legacy=0x7fde38167130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde380ccd30 con 0x7fde380ce010 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fde200016f0 con 0x7fde380ce010 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 >> v1:172.21.15.1:6789/0 conn(0x7fde380ce010 legacy=0x7fde3815fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde240039e0 0x7fde24003e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde24004800 0x7fde24004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde2400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fde3815d0c0 con 0x7fde24005150 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fde2400c010 con 0x7fde24004800 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fde2400c210 con 0x7fde240039e0 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde377fe640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde380b4ec0 0x7fde380cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde37fff640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde2400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde2ffff640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde24004800 0x7fde24004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde377fe640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde240039e0 0x7fde24003e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde37fff640 1 -- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde240039e0 msgr2=0x7fde24003e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde37fff640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde240039e0 0x7fde24003e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde37fff640 1 -- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde24004800 msgr2=0x7fde24004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde37fff640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde24004800 0x7fde24004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.435+0000 7fde37fff640 1 -- 172.21.15.1:0/2625997783 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde24003520 con 0x7fde24005150 2024-05-09T05:13:33.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.436+0000 7fde37fff640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde2400b9c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde3808fd60 tx=0x7fde24005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72be90252eda57a3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.436+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30001bb0 con 0x7fde24005150 2024-05-09T05:13:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.436+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde30031600 con 0x7fde24005150 2024-05-09T05:13:33.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.436+0000 7fde36ffd640 1 -- 172.21.15.1:0/2625997783 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde30031a10 con 0x7fde24005150 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.437+0000 7fde3de7e640 1 -- 172.21.15.1:0/2625997783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 msgr2=0x7fde2400b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.437+0000 7fde3de7e640 1 --2- 172.21.15.1:0/2625997783 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde2400b9c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fde3808fd60 tx=0x7fde24005830 comp rx=0 tx=0).stop 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.437+0000 7fde37fff640 1 -- 172.21.15.1:0/2625997783 reap_dead start 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.437+0000 7fde3de7e640 1 -- 172.21.15.1:0/2625997783 shutdown_connections 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.437+0000 7fde3de7e640 1 -- 172.21.15.1:0/2625997783 >> 172.21.15.1:0/2625997783 conn(0x7fde380d2360 msgr2=0x7fde380ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.437+0000 7fde3de7e640 1 -- 172.21.15.1:0/2625997783 shutdown_connections 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.437+0000 7fde3de7e640 1 -- 172.21.15.1:0/2625997783 wait complete. 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.439+0000 7fde3de7e640 1 Processor -- start 2024-05-09T05:13:33.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.440+0000 7fde3de7e640 1 -- start start 2024-05-09T05:13:33.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde3de7e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde3815fda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde3de7e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150410 0x7fde38154850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde3de7e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde38154d90 0x7fde3815a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde3de7e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fde380cc750 con 0x7fde24005150 2024-05-09T05:13:33.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde3de7e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fde3815cd50 con 0x7fde38150410 2024-05-09T05:13:33.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde3de7e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fde38156ae0 con 0x7fde38154d90 2024-05-09T05:13:33.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde377fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde3815fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde377fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde3815fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33040/0 (socket says 172.21.15.1:33040) 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde377fe640 1 -- 172.21.15.1:0/4047441162 learned_addr learned my addr 172.21.15.1:0/4047441162 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde37fff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde38154d90 0x7fde3815a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde2ffff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150410 0x7fde38154850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde377fe640 1 -- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde38154d90 msgr2=0x7fde3815a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde377fe640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fde38154d90 0x7fde3815a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde377fe640 1 -- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150410 msgr2=0x7fde38154850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde377fe640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150410 0x7fde38154850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde377fe640 1 -- 172.21.15.1:0/4047441162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde38151d10 con 0x7fde24005150 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.441+0000 7fde2ffff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150410 0x7fde38154850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.442+0000 7fde377fe640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde3815fda0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fde280115d0 tx=0x7fde28011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.442+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2800f8f0 con 0x7fde24005150 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.442+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fde2800fa90 con 0x7fde24005150 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.442+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde2800fe80 con 0x7fde24005150 2024-05-09T05:13:33.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.442+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde381522b0 con 0x7fde24005150 2024-05-09T05:13:33.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.442+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde381f33e0 con 0x7fde24005150 2024-05-09T05:13:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.443+0000 7fde3de7e640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde381f35e0 0x7fde3815ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.443+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fde38151590 con 0x7fde381f35e0 2024-05-09T05:13:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.443+0000 7fde2ffff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde381f35e0 0x7fde3815ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.444+0000 7fde2ffff640 1 -- 172.21.15.1:0/4047441162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fde38050870 con 0x7fde381f35e0 2024-05-09T05:13:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.444+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fde28015460 con 0x7fde24005150 2024-05-09T05:13:33.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.444+0000 7fde2ffff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde381f35e0 0x7fde3815ab60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde38154490 tx=0x7fde20007720 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.444+0000 7fde34ff9640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fde18076300 0x7fde18078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.444+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fde28097f40 con 0x7fde24005150 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.445+0000 7fde37fff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fde18076300 0x7fde18078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.445+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde20011930 con 0x7fde381f35e0 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.445+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fde2001a070 con 0x7fde381f35e0 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.445+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde381f35e0 msgr2=0x7fde3815ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.445+0000 7fde34ff9640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde381f35e0 0x7fde3815ab60 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde38154490 tx=0x7fde20007720 comp rx=0 tx=0).stop 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.445+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde381f35e0 msgr2=0x7fde3815ab60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.445+0000 7fde34ff9640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde381f35e0 0x7fde3815ab60 secure :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fde38154490 tx=0x7fde20007720 comp rx=0 tx=0).stop 2024-05-09T05:13:33.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.445+0000 7fde37fff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fde18076300 0x7fde18078810 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fde30034000 tx=0x7fde3002f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.453+0000 7fde3de7e640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150820 0x7fde381f6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.453+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fde38152060 con 0x7fde38150820 2024-05-09T05:13:33.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.453+0000 7fde2ffff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150820 0x7fde381f6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.453+0000 7fde2ffff640 1 -- 172.21.15.1:0/4047441162 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fde380b65d0 con 0x7fde38150820 2024-05-09T05:13:33.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.454+0000 7fde2ffff640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150820 0x7fde381f6320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde38154490 tx=0x7fde380ccc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.454+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde20015470 con 0x7fde38150820 2024-05-09T05:13:33.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.454+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4550 (secure 0 0 0) 0x7fde20015470 con 0x7fde38150820 2024-05-09T05:13:33.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.454+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150820 msgr2=0x7fde381f6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.454+0000 7fde34ff9640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150820 0x7fde381f6320 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fde38154490 tx=0x7fde380ccc20 comp rx=0 tx=0).stop 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.454+0000 7fde34ff9640 1 -- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150820 msgr2=0x7fde381f6320 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.454+0000 7fde34ff9640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fde38150820 0x7fde381f6320 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 62, 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 75734, 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:13:33.456 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.457 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.458 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:13:33.459 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.460 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:13:33.461 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:13:33.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.463 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.465 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.466 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:13:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.468 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:13:33.468 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:13:33.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fde18076300 msgr2=0x7fde18078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fde18076300 0x7fde18078810 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fde30034000 tx=0x7fde3002f000 comp rx=0 tx=0).stop 2024-05-09T05:13:33.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 msgr2=0x7fde3815fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde3815fda0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fde280115d0 tx=0x7fde28011a90 comp rx=0 tx=0).stop 2024-05-09T05:13:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde37fff640 1 -- 172.21.15.1:0/4047441162 reap_dead start 2024-05-09T05:13:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 shutdown_connections 2024-05-09T05:13:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 --2- 172.21.15.1:0/4047441162 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fde24005150 0x7fde3815fda0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 >> 172.21.15.1:0/4047441162 conn(0x7fde380d2360 msgr2=0x7fde380cd5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 shutdown_connections 2024-05-09T05:13:33.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.455+0000 7fde3de7e640 1 -- 172.21.15.1:0/4047441162 wait complete. 2024-05-09T05:13:33.470 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:13:33.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:13:33.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:13:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.594+0000 7f71f76ee640 1 Processor -- start 2024-05-09T05:13:33.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.594+0000 7f71f76ee640 1 -- start start 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f76ee640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f71f00c89a0 0x7f71f00ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f76ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 0x7f71f0153490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f76ee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f71f00c3be0 con 0x7f71f00c46e0 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f76ee640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f71f00c9290 con 0x7f71f00c89a0 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f76ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71f006e890 con 0x7f71f00c4af0 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f4c89640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f71f00c46e0 0x7f71f0159650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52478/0 (socket says 172.21.15.1:52478) 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f4c89640 1 -- 172.21.15.1:0/2212944436 learned_addr learned my addr 172.21.15.1:0/2212944436 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f548a640 1 --2- 172.21.15.1:0/2212944436 >> v2:172.21.15.196:3300/0 conn(0x7f71f00c89a0 0x7f71f00ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f5c8b640 1 --2- 172.21.15.1:0/2212944436 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 0x7f71f0153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71e7fff640 1 -- 172.21.15.1:0/2212944436 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3277857236 0 0) 0x7f71e0001110 con 0x7f71f00c46e0 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71e7fff640 1 -- 172.21.15.1:0/2212944436 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f71dc003520 con 0x7f71f00c46e0 2024-05-09T05:13:33.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f5c8b640 1 -- 172.21.15.1:0/2212944436 >> v2:172.21.15.196:3300/0 conn(0x7f71f00c89a0 msgr2=0x7f71f00ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f5c8b640 1 --2- 172.21.15.1:0/2212944436 >> v2:172.21.15.196:3300/0 conn(0x7f71f00c89a0 0x7f71f00ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f5c8b640 1 -- 172.21.15.1:0/2212944436 >> v1:172.21.15.1:6789/0 conn(0x7f71f00c46e0 legacy=0x7f71f0159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f5c8b640 1 -- 172.21.15.1:0/2212944436 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f00c3e60 con 0x7f71f00c4af0 2024-05-09T05:13:33.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.596+0000 7f71f548a640 1 --2- 172.21.15.1:0/2212944436 >> v2:172.21.15.196:3300/0 conn(0x7f71f00c89a0 0x7f71f00ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:13:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.597+0000 7f71f5c8b640 1 --2- 172.21.15.1:0/2212944436 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 0x7f71f0153490 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009d90 tx=0x7f71e802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6762482c2d591302 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71e7fff640 1 -- 172.21.15.1:0/2212944436 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e8007740 con 0x7f71f00c4af0 2024-05-09T05:13:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71e7fff640 1 -- 172.21.15.1:0/2212944436 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71e80078e0 con 0x7f71f00c4af0 2024-05-09T05:13:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71e7fff640 1 -- 172.21.15.1:0/2212944436 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e8007c90 con 0x7f71f00c4af0 2024-05-09T05:13:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71f76ee640 1 -- 172.21.15.1:0/2212944436 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 msgr2=0x7f71f0153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71f76ee640 1 --2- 172.21.15.1:0/2212944436 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 0x7f71f0153490 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f71e8009d90 tx=0x7f71e802a660 comp rx=0 tx=0).stop 2024-05-09T05:13:33.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71f76ee640 1 -- 172.21.15.1:0/2212944436 shutdown_connections 2024-05-09T05:13:33.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71f76ee640 1 --2- 172.21.15.1:0/2212944436 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 0x7f71f0153490 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71f76ee640 1 --2- 172.21.15.1:0/2212944436 >> v2:172.21.15.196:3300/0 conn(0x7f71f00c89a0 0x7f71f00ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71f76ee640 1 -- 172.21.15.1:0/2212944436 >> 172.21.15.1:0/2212944436 conn(0x7f71f00747b0 msgr2=0x7f71f00ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:33.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.598+0000 7f71f76ee640 1 -- 172.21.15.1:0/2212944436 shutdown_connections 2024-05-09T05:13:33.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.599+0000 7f71f76ee640 1 -- 172.21.15.1:0/2212944436 wait complete. 2024-05-09T05:13:33.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.600+0000 7f71f76ee640 1 Processor -- start 2024-05-09T05:13:33.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.600+0000 7f71f76ee640 1 -- start start 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f76ee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71f00c46e0 0x7f71f01574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f76ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 0x7f71f01f38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f76ee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71f00c89a0 0x7f71f0159b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f76ee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f71f00c97c0 con 0x7f71f00c46e0 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f76ee640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f71f00c94e0 con 0x7f71f00c89a0 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f76ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71f00cae70 con 0x7f71f00c4af0 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71f00c46e0 0x7f71f01574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71f00c46e0 0x7f71f01574c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33044/0 (socket says 172.21.15.1:33044) 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 -- 172.21.15.1:0/492646189 learned_addr learned my addr 172.21.15.1:0/492646189 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:33.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f4c89640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 0x7f71f01f38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f5c8b640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71f00c89a0 0x7f71f0159b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 -- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 msgr2=0x7f71f01f38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f00c4af0 0x7f71f01f38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 -- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71f00c89a0 msgr2=0x7f71f0159b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f71f00c89a0 0x7f71f0159b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 -- 172.21.15.1:0/492646189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71f01f4170 con 0x7f71f00c46e0 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.602+0000 7f71f548a640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71f00c46e0 0x7f71f01574c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00e990 tx=0x7f71ec00ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.603+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ec017070 con 0x7f71f00c46e0 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.603+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f71ec00c9e0 con 0x7f71f00c46e0 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.603+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71f01f4710 con 0x7f71f00c46e0 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.603+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71ec00cdb0 con 0x7f71f00c46e0 2024-05-09T05:13:33.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.603+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f71f01f09d0 con 0x7f71f00c46e0 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.604+0000 7f71f76ee640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0155440 0x7f71f0149050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.604+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71f01f44c0 con 0x7f71f0155440 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.604+0000 7f71f4c89640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0155440 0x7f71f0149050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.605+0000 7f71f4c89640 1 -- 172.21.15.1:0/492646189 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f71f0166f10 con 0x7f71f0155440 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.605+0000 7f71f4c89640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0155440 0x7f71f0149050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f71f006e300 tx=0x7f71e000fc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.605+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e00058b0 con 0x7f71f0155440 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.606+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f71e0018070 con 0x7f71f0155440 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.606+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0155440 msgr2=0x7f71f0149050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.606+0000 7f71e5ffb640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0155440 0x7f71f0149050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f71f006e300 tx=0x7f71e000fc00 comp rx=0 tx=0).stop 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.606+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0155440 msgr2=0x7f71f0149050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.606+0000 7f71e5ffb640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0155440 0x7f71f0149050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.606+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f71ec01a070 con 0x7f71f00c46e0 2024-05-09T05:13:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.606+0000 7f71e5ffb640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4075d40 0x7f71d4078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.606+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f71ec0974b0 con 0x7f71f00c46e0 2024-05-09T05:13:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.607+0000 7f71f4c89640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4075d40 0x7f71d4078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.607+0000 7f71f4c89640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4075d40 0x7f71d4078250 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f71e0008fd0 tx=0x7f71e0007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.615+0000 7f71f76ee640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0149590 0x7f71f01542a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.615+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f71f0050870 con 0x7f71f0149590 2024-05-09T05:13:33.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.615+0000 7f71f5c8b640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0149590 0x7f71f01542a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.615+0000 7f71f5c8b640 1 -- 172.21.15.1:0/492646189 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f71f00ca360 con 0x7f71f0149590 2024-05-09T05:13:33.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.616+0000 7f71f5c8b640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0149590 0x7f71f01542a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f71f00739e0 tx=0x7f71e8004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.616+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f71e8004820 con 0x7f71f0149590 2024-05-09T05:13:33.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.616+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5061 (secure 0 0 0) 0x7f71e8009a30 con 0x7f71f0149590 2024-05-09T05:13:33.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.616+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0149590 msgr2=0x7f71f01542a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.616+0000 7f71e5ffb640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0149590 0x7f71f01542a0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f71f00739e0 tx=0x7f71e8004a40 comp rx=0 tx=0).stop 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.616+0000 7f71e5ffb640 1 -- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0149590 msgr2=0x7f71f01542a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.616+0000 7f71e5ffb640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f71f0149590 0x7f71f01542a0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 62, 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 371615, 2024-05-09T05:13:33.618 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.619 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:13:33.621 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:33.622 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:13:33.623 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.624 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.628 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.629 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:33.630 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:33.631 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:13:33.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:33.631 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:33.631 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:33.631 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:13:33.631 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:13:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4075d40 msgr2=0x7f71d4078250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f71d4075d40 0x7f71d4078250 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f71e0008fd0 tx=0x7f71e0007000 comp rx=0 tx=0).stop 2024-05-09T05:13:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71f00c46e0 msgr2=0x7f71f01574c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71f00c46e0 0x7f71f01574c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71ec00e990 tx=0x7f71ec00ee50 comp rx=0 tx=0).stop 2024-05-09T05:13:33.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f5c8b640 1 -- 172.21.15.1:0/492646189 reap_dead start 2024-05-09T05:13:33.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 shutdown_connections 2024-05-09T05:13:33.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 --2- 172.21.15.1:0/492646189 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f71f00c46e0 0x7f71f01574c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 >> 172.21.15.1:0/492646189 conn(0x7f71f00747b0 msgr2=0x7f71f006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:33.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 shutdown_connections 2024-05-09T05:13:33.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.617+0000 7f71f76ee640 1 -- 172.21.15.1:0/492646189 wait complete. 2024-05-09T05:13:33.633 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:13:33.633 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:13:33.633 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:13:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.755+0000 7fdf53f34640 1 Processor -- start 2024-05-09T05:13:33.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.755+0000 7fdf53f34640 1 -- start start 2024-05-09T05:13:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf53f34640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c0d16b0 0x7fdf4c0d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf53f34640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf4c06f1b0 0x7fdf4c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf53f34640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf4c06f6d0 con 0x7fdf4c15ca90 2024-05-09T05:13:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf53f34640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdf4c0cd0e0 con 0x7fdf4c06f1b0 2024-05-09T05:13:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf53f34640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdf4c0cc4d0 con 0x7fdf4c0d16b0 2024-05-09T05:13:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf514cf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf4c06f1b0 0x7fdf4c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf524d1640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdf4c15ca90 0x7fdf4c15cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:50314/0 (socket says 172.21.15.1:50314) 2024-05-09T05:13:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf524d1640 1 -- 172.21.15.1:0/3778937971 learned_addr learned my addr 172.21.15.1:0/3778937971 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf514cf640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdf4c06f1b0 0x7fdf4c0ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33046/0 (socket says 172.21.15.1:33046) 2024-05-09T05:13:33.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.757+0000 7fdf51cd0640 1 --2- 172.21.15.1:0/3778937971 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c0d16b0 0x7fdf4c0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.758+0000 7fdf50cce640 1 -- 172.21.15.1:0/3778937971 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3992030846 0 0) 0x7fdf4c06f6d0 con 0x7fdf4c15ca90 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.758+0000 7fdf514cf640 1 -- 172.21.15.1:0/3778937971 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c0d16b0 msgr2=0x7fdf4c0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.758+0000 7fdf514cf640 1 --2- 172.21.15.1:0/3778937971 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c0d16b0 0x7fdf4c0d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.758+0000 7fdf514cf640 1 -- 172.21.15.1:0/3778937971 >> v1:172.21.15.196:6789/0 conn(0x7fdf4c15ca90 legacy=0x7fdf4c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.758+0000 7fdf514cf640 1 -- 172.21.15.1:0/3778937971 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4c0cc7b0 con 0x7fdf4c06f1b0 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.758+0000 7fdf514cf640 1 --2- 172.21.15.1:0/3778937971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c06f1b0 0x7fdf4c0ce740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0097a0 tx=0x7fdf3c02a210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26c60ee78ee1707d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.759+0000 7fdf50cce640 1 -- 172.21.15.1:0/3778937971 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c037070 con 0x7fdf4c06f1b0 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.759+0000 7fdf50cce640 1 -- 172.21.15.1:0/3778937971 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf3c032440 con 0x7fdf4c06f1b0 2024-05-09T05:13:33.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.759+0000 7fdf50cce640 1 -- 172.21.15.1:0/3778937971 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf3c037070 con 0x7fdf4c06f1b0 2024-05-09T05:13:33.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.759+0000 7fdf53f34640 1 -- 172.21.15.1:0/3778937971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c06f1b0 msgr2=0x7fdf4c0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.759+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3778937971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c06f1b0 0x7fdf4c0ce740 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fdf3c0097a0 tx=0x7fdf3c02a210 comp rx=0 tx=0).stop 2024-05-09T05:13:33.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.759+0000 7fdf53f34640 1 -- 172.21.15.1:0/3778937971 shutdown_connections 2024-05-09T05:13:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.760+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3778937971 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c06f1b0 0x7fdf4c0ce740 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.760+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3778937971 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c0d16b0 0x7fdf4c0d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.760+0000 7fdf53f34640 1 -- 172.21.15.1:0/3778937971 >> 172.21.15.1:0/3778937971 conn(0x7fdf4c0b3660 msgr2=0x7fdf4c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.760+0000 7fdf53f34640 1 -- 172.21.15.1:0/3778937971 shutdown_connections 2024-05-09T05:13:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.760+0000 7fdf53f34640 1 -- 172.21.15.1:0/3778937971 wait complete. 2024-05-09T05:13:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.761+0000 7fdf53f34640 1 Processor -- start 2024-05-09T05:13:33.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.762+0000 7fdf53f34640 1 -- start start 2024-05-09T05:13:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf53f34640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c06f1b0 0x7fdf4c1f2fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf53f34640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf4c0d16b0 0x7fdf4c0b6a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf53f34640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c15ca90 0x7fdf4c0cb4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf53f34640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c0d0910 con 0x7fdf4c15ca90 2024-05-09T05:13:33.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf53f34640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdf4c0ccd70 con 0x7fdf4c0d16b0 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf53f34640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdf4c0cec80 con 0x7fdf4c06f1b0 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf524d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c15ca90 0x7fdf4c0cb4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf524d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c15ca90 0x7fdf4c0cb4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33048/0 (socket says 172.21.15.1:33048) 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf524d1640 1 -- 172.21.15.1:0/3797634141 learned_addr learned my addr 172.21.15.1:0/3797634141 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf514cf640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf4c0d16b0 0x7fdf4c0b6a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf51cd0640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c06f1b0 0x7fdf4c1f2fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf524d1640 1 -- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c06f1b0 msgr2=0x7fdf4c1f2fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.763+0000 7fdf524d1640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c06f1b0 0x7fdf4c1f2fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf524d1640 1 -- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf4c0d16b0 msgr2=0x7fdf4c0b6a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf524d1640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf4c0d16b0 0x7fdf4c0b6a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf524d1640 1 -- 172.21.15.1:0/3797634141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf4c156a70 con 0x7fdf4c15ca90 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf524d1640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c15ca90 0x7fdf4c0cb4b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400c6b0 tx=0x7fdf4400cb70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf427fc640 1 -- 172.21.15.1:0/3797634141 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf44018070 con 0x7fdf4c15ca90 2024-05-09T05:13:33.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf427fc640 1 -- 172.21.15.1:0/3797634141 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf44004da0 con 0x7fdf4c15ca90 2024-05-09T05:13:33.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf4c0b5ab0 con 0x7fdf4c15ca90 2024-05-09T05:13:33.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf427fc640 1 -- 172.21.15.1:0/3797634141 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4400a880 con 0x7fdf4c15ca90 2024-05-09T05:13:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.764+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf4c0b76b0 con 0x7fdf4c15ca90 2024-05-09T05:13:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.765+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdf4c0b7120 con 0x7fdf4c15ca90 2024-05-09T05:13:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.766+0000 7fdf427fc640 1 -- 172.21.15.1:0/3797634141 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdf4401b070 con 0x7fdf4c15ca90 2024-05-09T05:13:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.767+0000 7fdf427fc640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf300762d0 0x7fdf300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.767+0000 7fdf427fc640 1 -- 172.21.15.1:0/3797634141 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdf44096bc0 con 0x7fdf4c15ca90 2024-05-09T05:13:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.767+0000 7fdf51cd0640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf300762d0 0x7fdf300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:33.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.768+0000 7fdf51cd0640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf300762d0 0x7fdf300787e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009aa0 tx=0x7fdf48009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:33.773 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:33.772+0000 7fdf427fc640 1 -- 172.21.15.1:0/3797634141 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdf4405fe50 con 0x7fdf4c15ca90 2024-05-09T05:13:34.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.072+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fdf4c050870 con 0x7fdf4c15ca90 2024-05-09T05:13:34.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.073+0000 7fdf427fc640 1 -- 172.21.15.1:0/3797634141 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fdf44064150 con 0x7fdf4c15ca90 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf300762d0 msgr2=0x7fdf300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf300762d0 0x7fdf300787e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fdf48009aa0 tx=0x7fdf48009190 comp rx=0 tx=0).stop 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c15ca90 msgr2=0x7fdf4c0cb4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c15ca90 0x7fdf4c0cb4b0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fdf4400c6b0 tx=0x7fdf4400cb70 comp rx=0 tx=0).stop 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 shutdown_connections 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf300762d0 0x7fdf300787e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf4c0d16b0 0x7fdf4c0b6a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf4c15ca90 0x7fdf4c0cb4b0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 --2- 172.21.15.1:0/3797634141 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf4c06f1b0 0x7fdf4c1f2fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 >> 172.21.15.1:0/3797634141 conn(0x7fdf4c0b3660 msgr2=0x7fdf4c1533a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 shutdown_connections 2024-05-09T05:13:34.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:34.075+0000 7fdf53f34640 1 -- 172.21.15.1:0/3797634141 wait complete. 2024-05-09T05:13:34.089 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:13:44.091 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:13:44.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:13:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.208+0000 7fb91ba53640 1 Processor -- start 2024-05-09T05:13:44.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.209+0000 7fb91ba53640 1 -- start start 2024-05-09T05:13:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb91ba53640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9140d16b0 0x7fb9140d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb91ba53640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb91406f1b0 0x7fb9140ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb91ba53640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb91406f6d0 con 0x7fb91415ca90 2024-05-09T05:13:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb91ba53640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb9140cd0e0 con 0x7fb91406f1b0 2024-05-09T05:13:44.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb91ba53640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb9140cc4d0 con 0x7fb9140d16b0 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb919ff0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb91415ca90 0x7fb91415cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39766/0 (socket says 172.21.15.1:39766) 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb919ff0640 1 -- 172.21.15.1:0/2900419648 learned_addr learned my addr 172.21.15.1:0/2900419648 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb9197ef640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9140d16b0 0x7fb9140d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.210+0000 7fb918fee640 1 --2- 172.21.15.1:0/2900419648 >> v2:172.21.15.196:3300/0 conn(0x7fb91406f1b0 0x7fb9140ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1662474117 0 0) 0x7fb91406f6d0 con 0x7fb91415ca90 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb900003520 con 0x7fb91415ca90 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1351906416 0 0) 0x7fb900003520 con 0x7fb91415ca90 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9140d16b0 msgr2=0x7fb9140d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9140d16b0 0x7fb9140d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 >> v2:172.21.15.196:3300/0 conn(0x7fb91406f1b0 msgr2=0x7fb9140ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 --2- 172.21.15.1:0/2900419648 >> v2:172.21.15.196:3300/0 conn(0x7fb91406f1b0 0x7fb9140ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9140cc7b0 con 0x7fb91415ca90 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fb90c001a70 con 0x7fb91415ca90 2024-05-09T05:13:44.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 >> v1:172.21.15.1:6789/0 conn(0x7fb91415ca90 legacy=0x7fb91415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9140d16b0 0x7fb900003a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb900003fd0 0x7fb900004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb900004d40 0x7fb900005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb9140cd0e0 con 0x7fb900003fd0 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb9140cc4d0 con 0x7fb900004d40 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb90000bb50 con 0x7fb9140d16b0 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb918fee640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb900003fd0 0x7fb900004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb9197ef640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9140d16b0 0x7fb900003a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb919ff0640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb900004d40 0x7fb900005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb918fee640 1 -- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9140d16b0 msgr2=0x7fb900003a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb918fee640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb9140d16b0 0x7fb900003a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb918fee640 1 -- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb900004d40 msgr2=0x7fb900005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb918fee640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb900004d40 0x7fb900005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.211+0000 7fb918fee640 1 -- 172.21.15.1:0/2900419648 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb900003520 con 0x7fb900003fd0 2024-05-09T05:13:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.212+0000 7fb918fee640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb900003fd0 0x7fb900004800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb91415a370 tx=0x7fb900005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa51685183edaef5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.212+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908036070 con 0x7fb900003fd0 2024-05-09T05:13:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.212+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb908031400 con 0x7fb900003fd0 2024-05-09T05:13:44.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.212+0000 7fb907fff640 1 -- 172.21.15.1:0/2900419648 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908009690 con 0x7fb900003fd0 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.213+0000 7fb91ba53640 1 -- 172.21.15.1:0/2900419648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb900003fd0 msgr2=0x7fb900004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.213+0000 7fb91ba53640 1 --2- 172.21.15.1:0/2900419648 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb900003fd0 0x7fb900004800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb91415a370 tx=0x7fb900005830 comp rx=0 tx=0).stop 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.213+0000 7fb919ff0640 1 -- 172.21.15.1:0/2900419648 reap_dead start 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.213+0000 7fb91ba53640 1 -- 172.21.15.1:0/2900419648 shutdown_connections 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.213+0000 7fb91ba53640 1 -- 172.21.15.1:0/2900419648 >> 172.21.15.1:0/2900419648 conn(0x7fb9140b3660 msgr2=0x7fb9140b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.213+0000 7fb91ba53640 1 -- 172.21.15.1:0/2900419648 shutdown_connections 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.214+0000 7fb91ba53640 1 -- 172.21.15.1:0/2900419648 wait complete. 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.215+0000 7fb91ba53640 1 Processor -- start 2024-05-09T05:13:44.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.215+0000 7fb91ba53640 1 -- start start 2024-05-09T05:13:44.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb91ba53640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb900003fd0 0x7fb9141f6110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb91ba53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9141f2e40 0x7fb914151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb91ba53640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9140cb320 0x7fb9140cb730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb91ba53640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb9140cec80 con 0x7fb9141f2e40 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb91ba53640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb9140ccd70 con 0x7fb9140cb320 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb91ba53640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb9140cee00 con 0x7fb900003fd0 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9141f2e40 0x7fb914151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9141f2e40 0x7fb914151e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53256/0 (socket says 172.21.15.1:53256) 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 -- 172.21.15.1:0/2826488604 learned_addr learned my addr 172.21.15.1:0/2826488604 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb919ff0640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9140cb320 0x7fb9140cb730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb9197ef640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb900003fd0 0x7fb9141f6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 -- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb900003fd0 msgr2=0x7fb9141f6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb900003fd0 0x7fb9141f6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 -- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9140cb320 msgr2=0x7fb9140cb730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9140cb320 0x7fb9140cb730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 -- 172.21.15.1:0/2826488604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb9141f6650 con 0x7fb9141f2e40 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb919ff0640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9140cb320 0x7fb9140cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb9197ef640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb900003fd0 0x7fb9141f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb918fee640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9141f2e40 0x7fb914151e90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb908009b50 tx=0x7fb908004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb905ffb640 1 -- 172.21.15.1:0/2826488604 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908040020 con 0x7fb9141f2e40 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb905ffb640 1 -- 172.21.15.1:0/2826488604 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb908003a40 con 0x7fb9141f2e40 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.217+0000 7fb905ffb640 1 -- 172.21.15.1:0/2826488604 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb908036040 con 0x7fb9141f2e40 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.218+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9141599c0 con 0x7fb9141f2e40 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.218+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9141f2640 con 0x7fb9141f2e40 2024-05-09T05:13:44.220 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.219+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb914050870 con 0x7fb9141f2e40 2024-05-09T05:13:44.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.220+0000 7fb905ffb640 1 -- 172.21.15.1:0/2826488604 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb908043020 con 0x7fb9141f2e40 2024-05-09T05:13:44.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.220+0000 7fb905ffb640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb8f8076300 0x7fb8f8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.220+0000 7fb905ffb640 1 -- 172.21.15.1:0/2826488604 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb9080b7890 con 0x7fb9141f2e40 2024-05-09T05:13:44.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.221+0000 7fb9197ef640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb8f8076300 0x7fb8f8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.221+0000 7fb9197ef640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb8f8076300 0x7fb8f8078810 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb910009a80 tx=0x7fb910008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.228+0000 7fb905ffb640 1 -- 172.21.15.1:0/2826488604 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb90804a070 con 0x7fb9141f2e40 2024-05-09T05:13:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.522+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb914159750 con 0x7fb9141f2e40 2024-05-09T05:13:44.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.522+0000 7fb905ffb640 1 -- 172.21.15.1:0/2826488604 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb908081170 con 0x7fb9141f2e40 2024-05-09T05:13:44.523 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":11,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.524+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb8f8076300 msgr2=0x7fb8f8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.524+0000 7fb91ba53640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb8f8076300 0x7fb8f8078810 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fb910009a80 tx=0x7fb910008040 comp rx=0 tx=0).stop 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9141f2e40 msgr2=0x7fb914151e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9141f2e40 0x7fb914151e90 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb908009b50 tx=0x7fb908004730 comp rx=0 tx=0).stop 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 shutdown_connections 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb8f8076300 0x7fb8f8078810 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb9140cb320 0x7fb9140cb730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb9141f2e40 0x7fb914151e90 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 --2- 172.21.15.1:0/2826488604 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb900003fd0 0x7fb9141f6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 >> 172.21.15.1:0/2826488604 conn(0x7fb9140b3660 msgr2=0x7fb91415ae20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:44.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 shutdown_connections 2024-05-09T05:13:44.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.525+0000 7fb91ba53640 1 -- 172.21.15.1:0/2826488604 wait complete. 2024-05-09T05:13:44.536 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:13:44.536 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:13:44.536 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:13:44.536 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:13:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.666+0000 7f4aa45b2640 1 Processor -- start 2024-05-09T05:13:44.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.666+0000 7f4aa45b2640 1 -- start start 2024-05-09T05:13:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.667+0000 7f4aa45b2640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f4a9c0714f0 0x7f4a9c0ca6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.667+0000 7f4aa45b2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a9c15d760 con 0x7f4a9c0710e0 2024-05-09T05:13:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.667+0000 7f4aa45b2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4a9c15c900 con 0x7f4a9c0c6900 2024-05-09T05:13:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.667+0000 7f4aa45b2640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f4a9c15cbe0 con 0x7f4a9c0714f0 2024-05-09T05:13:44.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.667+0000 7f4aa1b4d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f4a9c0710e0 0x7f4a9c156650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39780/0 (socket says 172.21.15.1:39780) 2024-05-09T05:13:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.668+0000 7f4aa1b4d640 1 -- 172.21.15.1:0/2844002251 learned_addr learned my addr 172.21.15.1:0/2844002251 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.668+0000 7f4aa2b4f640 1 --2- 172.21.15.1:0/2844002251 >> v2:172.21.15.196:3300/0 conn(0x7f4a9c0714f0 0x7f4a9c0ca6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.668+0000 7f4aa134c640 1 -- 172.21.15.1:0/2844002251 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 55362233 0 0) 0x7f4a9c15d760 con 0x7f4a9c0710e0 2024-05-09T05:13:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.668+0000 7f4aa134c640 1 -- 172.21.15.1:0/2844002251 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4a88003520 con 0x7f4a9c0710e0 2024-05-09T05:13:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.668+0000 7f4aa134c640 1 -- 172.21.15.1:0/2844002251 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2161725918 0 0) 0x7f4a88003520 con 0x7f4a9c0710e0 2024-05-09T05:13:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.668+0000 7f4aa2b4f640 1 -- 172.21.15.1:0/2844002251 >> v1:172.21.15.196:6789/0 conn(0x7f4a9c0c6900 legacy=0x7f4a9c06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.668+0000 7f4aa2b4f640 1 -- 172.21.15.1:0/2844002251 >> v1:172.21.15.1:6789/0 conn(0x7f4a9c0710e0 legacy=0x7f4a9c156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.668+0000 7f4aa2b4f640 1 -- 172.21.15.1:0/2844002251 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c15cec0 con 0x7f4a9c0714f0 2024-05-09T05:13:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.669+0000 7f4aa2b4f640 1 --2- 172.21.15.1:0/2844002251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a9c0714f0 0x7f4a9c0ca6a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009d90 tx=0x7f4a9402a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69b2e0ef96d14865 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.669+0000 7f4aa134c640 1 -- 172.21.15.1:0/2844002251 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c0016f0 con 0x7f4a9c0714f0 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.669+0000 7f4aa134c640 1 -- 172.21.15.1:0/2844002251 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a94007540 con 0x7f4a9c0714f0 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.669+0000 7f4aa134c640 1 -- 172.21.15.1:0/2844002251 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a94007910 con 0x7f4a9c0714f0 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.670+0000 7f4aa45b2640 1 -- 172.21.15.1:0/2844002251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a9c0714f0 msgr2=0x7f4a9c0ca6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.670+0000 7f4aa45b2640 1 --2- 172.21.15.1:0/2844002251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a9c0714f0 0x7f4a9c0ca6a0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4a94009d90 tx=0x7f4a9402a710 comp rx=0 tx=0).stop 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.670+0000 7f4aa45b2640 1 -- 172.21.15.1:0/2844002251 shutdown_connections 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.670+0000 7f4aa45b2640 1 --2- 172.21.15.1:0/2844002251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a9c0714f0 0x7f4a9c0ca6a0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.670+0000 7f4aa45b2640 1 -- 172.21.15.1:0/2844002251 >> 172.21.15.1:0/2844002251 conn(0x7f4a9c0d0470 msgr2=0x7f4a9c0cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.670+0000 7f4aa45b2640 1 -- 172.21.15.1:0/2844002251 shutdown_connections 2024-05-09T05:13:44.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.671+0000 7f4aa45b2640 1 -- 172.21.15.1:0/2844002251 wait complete. 2024-05-09T05:13:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.673+0000 7f4aa45b2640 1 Processor -- start 2024-05-09T05:13:44.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.673+0000 7f4aa45b2640 1 -- start start 2024-05-09T05:13:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa45b2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a9c0710e0 0x7f4a9c0c6d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa45b2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0714f0 0x7f4a9c0c7b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa45b2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4a9c0c6900 0x7f4a9c0c8080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa45b2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c072390 con 0x7f4a9c0714f0 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa45b2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c15d480 con 0x7f4a9c0710e0 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa45b2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4a9c156d60 con 0x7f4a9c0c6900 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa1b4d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0714f0 0x7f4a9c0c7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa1b4d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0714f0 0x7f4a9c0c7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53266/0 (socket says 172.21.15.1:53266) 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa1b4d640 1 -- 172.21.15.1:0/1934920366 learned_addr learned my addr 172.21.15.1:0/1934920366 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa2b4f640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4a9c0c6900 0x7f4a9c0c8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.674+0000 7f4aa234e640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a9c0710e0 0x7f4a9c0c6d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa1b4d640 1 -- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4a9c0c6900 msgr2=0x7f4a9c0c8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa1b4d640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4a9c0c6900 0x7f4a9c0c8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa1b4d640 1 -- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a9c0710e0 msgr2=0x7f4a9c0c6d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa1b4d640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4a9c0710e0 0x7f4a9c0c6d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa1b4d640 1 -- 172.21.15.1:0/1934920366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a9c14f070 con 0x7f4a9c0714f0 2024-05-09T05:13:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa2b4f640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4a9c0c6900 0x7f4a9c0c8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:13:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa1b4d640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0714f0 0x7f4a9c0c7b40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00c4d0 tx=0x7f4a8c00c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c017070 con 0x7f4a9c0714f0 2024-05-09T05:13:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4a8c005ce0 con 0x7f4a9c0714f0 2024-05-09T05:13:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a8c007290 con 0x7f4a9c0714f0 2024-05-09T05:13:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a9c152dc0 con 0x7f4a9c0714f0 2024-05-09T05:13:44.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.675+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a9c14e4a0 con 0x7f4a9c0714f0 2024-05-09T05:13:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.676+0000 7f4aa45b2640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0c99f0 0x7f4a9c1f5910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.676+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c14d220 con 0x7f4a9c0c99f0 2024-05-09T05:13:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.676+0000 7f4aa234e640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0c99f0 0x7f4a9c1f5910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.677+0000 7f4aa234e640 1 -- 172.21.15.1:0/1934920366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a9c14eca0 con 0x7f4a9c0c99f0 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.677+0000 7f4aa234e640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0c99f0 0x7f4a9c1f5910 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800f520 tx=0x7f4a9800f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.677+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a980052e0 con 0x7f4a9c0c99f0 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.677+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4a8c01a070 con 0x7f4a9c0714f0 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.678+0000 7f4a92ffd640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4a80076200 0x7f4a80078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.678+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4a8c096870 con 0x7f4a9c0714f0 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.678+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4a98018070 con 0x7f4a9c0c99f0 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.678+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0c99f0 msgr2=0x7f4a9c1f5910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.678+0000 7f4a92ffd640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0c99f0 0x7f4a9c1f5910 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800f520 tx=0x7f4a9800f9e0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.678+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0c99f0 msgr2=0x7f4a9c1f5910 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.678+0000 7f4a92ffd640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0c99f0 0x7f4a9c1f5910 secure :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800f520 tx=0x7f4a9800f9e0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.678+0000 7f4aa2b4f640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4a80076200 0x7f4a80078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.679+0000 7f4aa2b4f640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4a80076200 0x7f4a80078710 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4a94034000 tx=0x7f4a9402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.685+0000 7f4aa45b2640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0b5880 0x7f4a9c1f5e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.685+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4a9c0c7560 con 0x7f4a9c0b5880 2024-05-09T05:13:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.685+0000 7f4aa234e640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0b5880 0x7f4a9c1f5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.685+0000 7f4aa234e640 1 -- 172.21.15.1:0/1934920366 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4a9c050870 con 0x7f4a9c0b5880 2024-05-09T05:13:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.685+0000 7f4aa234e640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0b5880 0x7f4a9c1f5e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800f520 tx=0x7f4a98001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a98003ad0 con 0x7f4a9c0b5880 2024-05-09T05:13:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f4a98003ad0 con 0x7f4a9c0b5880 2024-05-09T05:13:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0b5880 msgr2=0x7f4a9c1f5e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4a92ffd640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0b5880 0x7f4a9c1f5e50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4a9800f520 tx=0x7f4a98001080 comp rx=0 tx=0).stop 2024-05-09T05:13:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4a92ffd640 1 -- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0b5880 msgr2=0x7f4a9c1f5e50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:44.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4a92ffd640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0b5880 0x7f4a9c1f5e50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 62, 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 26881, 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:13:44.687 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 11, 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:44.688 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:44.689 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:13:44.690 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:44.691 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:44.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:13:44.693 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.695 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.696 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:13:44.697 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:13:44.698 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4a80076200 msgr2=0x7f4a80078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa45b2640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4a80076200 0x7f4a80078710 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f4a94034000 tx=0x7f4a9402f000 comp rx=0 tx=0).stop 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0714f0 msgr2=0x7f4a9c0c7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa45b2640 1 --2- 172.21.15.1:0/1934920366 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4a9c0714f0 0x7f4a9c0c7b40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c00c4d0 tx=0x7f4a8c00c990 comp rx=0 tx=0).stop 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa2b4f640 1 -- 172.21.15.1:0/1934920366 reap_dead start 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa2b4f640 1 -- 172.21.15.1:0/1934920366 reap_dead start 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 shutdown_connections 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 >> 172.21.15.1:0/1934920366 conn(0x7f4a9c0d0470 msgr2=0x7f4a9c1558c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 shutdown_connections 2024-05-09T05:13:44.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.686+0000 7f4aa45b2640 1 -- 172.21.15.1:0/1934920366 wait complete. 2024-05-09T05:13:44.701 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:13:44.701 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:13:44.701 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:13:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.825+0000 7f14503eb640 1 Processor -- start 2024-05-09T05:13:44.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.825+0000 7f14503eb640 1 -- start start 2024-05-09T05:13:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f14503eb640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 0x7f144815d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f14503eb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14480c5600 0x7f144806d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f14503eb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f14480cc1b0 con 0x7f14480736b0 2024-05-09T05:13:44.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f14503eb640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f144806d710 con 0x7f14480c5600 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f14503eb640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f14480cbf60 con 0x7f144806dde0 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144e988640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14480c5600 0x7f144806d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144d986640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f14480736b0 0x7f14480c70b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39782/0 (socket says 172.21.15.1:39782) 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144e988640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f14480c5600 0x7f144806d1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53288/0 (socket says 172.21.15.1:53288) 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144d986640 1 -- 172.21.15.1:0/1373552494 learned_addr learned my addr 172.21.15.1:0/1373552494 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144e187640 1 --2- 172.21.15.1:0/1373552494 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 0x7f144815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144e988640 1 -- 172.21.15.1:0/1373552494 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 msgr2=0x7f144815d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144e988640 1 --2- 172.21.15.1:0/1373552494 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 0x7f144815d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144e988640 1 -- 172.21.15.1:0/1373552494 >> v1:172.21.15.1:6789/0 conn(0x7f14480736b0 legacy=0x7f14480c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.827+0000 7f144e988640 1 -- 172.21.15.1:0/1373552494 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14480cc490 con 0x7f14480c5600 2024-05-09T05:13:44.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.828+0000 7f144e988640 1 --2- 172.21.15.1:0/1373552494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480c5600 0x7f144806d1d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1440009d90 tx=0x7f144002a640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e3c1b531f5c11b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.828+0000 7f144d185640 1 -- 172.21.15.1:0/1373552494 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f144002ae60 con 0x7f14480c5600 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.828+0000 7f144d185640 1 -- 172.21.15.1:0/1373552494 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1440007740 con 0x7f14480c5600 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.828+0000 7f144d185640 1 -- 172.21.15.1:0/1373552494 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1440007b10 con 0x7f14480c5600 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.829+0000 7f14503eb640 1 -- 172.21.15.1:0/1373552494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480c5600 msgr2=0x7f144806d1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.829+0000 7f14503eb640 1 --2- 172.21.15.1:0/1373552494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480c5600 0x7f144806d1d0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f1440009d90 tx=0x7f144002a640 comp rx=0 tx=0).stop 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.829+0000 7f14503eb640 1 -- 172.21.15.1:0/1373552494 shutdown_connections 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.829+0000 7f14503eb640 1 --2- 172.21.15.1:0/1373552494 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480c5600 0x7f144806d1d0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.829+0000 7f14503eb640 1 --2- 172.21.15.1:0/1373552494 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 0x7f144815d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.829+0000 7f14503eb640 1 -- 172.21.15.1:0/1373552494 >> 172.21.15.1:0/1373552494 conn(0x7f14480ce180 msgr2=0x7f14480ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.829+0000 7f14503eb640 1 -- 172.21.15.1:0/1373552494 shutdown_connections 2024-05-09T05:13:44.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.829+0000 7f14503eb640 1 -- 172.21.15.1:0/1373552494 wait complete. 2024-05-09T05:13:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.831+0000 7f14503eb640 1 Processor -- start 2024-05-09T05:13:44.831 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.831+0000 7f14503eb640 1 -- start start 2024-05-09T05:13:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f14503eb640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 0x7f14481609b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f14503eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480736b0 0x7f144814f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f14503eb640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480c5600 0x7f144814c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f14503eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f144806ccb0 con 0x7f14480736b0 2024-05-09T05:13:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f14503eb640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f144806c9d0 con 0x7f14480c5600 2024-05-09T05:13:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f14503eb640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f14480c77c0 con 0x7f144806dde0 2024-05-09T05:13:44.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480736b0 0x7f144814f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480736b0 0x7f144814f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53300/0 (socket says 172.21.15.1:53300) 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 -- 172.21.15.1:0/53807621 learned_addr learned my addr 172.21.15.1:0/53807621 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144e988640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480c5600 0x7f144814c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144e187640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 0x7f14481609b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 -- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 msgr2=0x7f14481609b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 0x7f14481609b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 -- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480c5600 msgr2=0x7f144814c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480c5600 0x7f144814c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 -- 172.21.15.1:0/53807621 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f144814a8a0 con 0x7f14480736b0 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144e988640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480c5600 0x7f144814c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144e187640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f144806dde0 0x7f14481609b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.833+0000 7f144d986640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480736b0 0x7f144814f050 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f143800c400 tx=0x7f143800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.834+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438017070 con 0x7f14480736b0 2024-05-09T05:13:44.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.834+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1438005d00 con 0x7f14480736b0 2024-05-09T05:13:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.834+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f144814ab80 con 0x7f14480736b0 2024-05-09T05:13:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.834+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1438007310 con 0x7f14480736b0 2024-05-09T05:13:44.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.834+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f144814b140 con 0x7f14480736b0 2024-05-09T05:13:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.835+0000 7f14503eb640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f144814b340 0x7f144814cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.835+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f144814c9d0 con 0x7f144814b340 2024-05-09T05:13:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.835+0000 7f144e187640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f144814b340 0x7f144814cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.835+0000 7f144e187640 1 -- 172.21.15.1:0/53807621 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f14481f38a0 con 0x7f144814b340 2024-05-09T05:13:44.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.835+0000 7f144e187640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f144814b340 0x7f144814cfb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f144400f550 tx=0x7f144400fa10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.836+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f143801a070 con 0x7f14480736b0 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.836+0000 7f143effd640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f142c0762d0 0x7f142c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.836+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444019020 con 0x7f144814b340 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.836+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1438096980 con 0x7f14480736b0 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.836+0000 7f144e988640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f142c0762d0 0x7f142c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.836+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1444014870 con 0x7f144814b340 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.836+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f144814b340 msgr2=0x7f144814cfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.837+0000 7f143effd640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f144814b340 0x7f144814cfb0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f144400f550 tx=0x7f144400fa10 comp rx=0 tx=0).stop 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.837+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f144814b340 msgr2=0x7f144814cfb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.837+0000 7f143effd640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f144814b340 0x7f144814cfb0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.837+0000 7f144e988640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f142c0762d0 0x7f142c0787e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1440034000 tx=0x7f1440033040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.845+0000 7f14503eb640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480746b0 0x7f144814d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.845+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1448050870 con 0x7f14480746b0 2024-05-09T05:13:44.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.845+0000 7f144e187640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480746b0 0x7f144814d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.845+0000 7f144e187640 1 -- 172.21.15.1:0/53807621 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f14480c8a10 con 0x7f14480746b0 2024-05-09T05:13:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f144e187640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480746b0 0x7f144814d4f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f144401a000 tx=0x7f14440046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1444019070 con 0x7f14480746b0 2024-05-09T05:13:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f1444004220 con 0x7f14480746b0 2024-05-09T05:13:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480746b0 msgr2=0x7f144814d4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f143effd640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480746b0 0x7f144814d4f0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f144401a000 tx=0x7f14440046c0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f143effd640 1 -- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480746b0 msgr2=0x7f144814d4f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:44.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f143effd640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f14480746b0 0x7f144814d4f0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 62, 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 87125, 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:13:44.847 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:44.848 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:44.849 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:13:44.850 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:44.851 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.852 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:44.853 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.854 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:13:44.855 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.856 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:13:44.857 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:13:44.858 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f142c0762d0 msgr2=0x7f142c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f14503eb640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f142c0762d0 0x7f142c0787e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f1440034000 tx=0x7f1440033040 comp rx=0 tx=0).stop 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480736b0 msgr2=0x7f144814f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f14503eb640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480736b0 0x7f144814f050 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f143800c400 tx=0x7f143800c8c0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f144e988640 1 -- 172.21.15.1:0/53807621 reap_dead start 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 shutdown_connections 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f14503eb640 1 --2- 172.21.15.1:0/53807621 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f14480736b0 0x7f144814f050 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.846+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 >> 172.21.15.1:0/53807621 conn(0x7f14480ce180 msgr2=0x7f1448155f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.847+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 shutdown_connections 2024-05-09T05:13:44.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.847+0000 7f14503eb640 1 -- 172.21.15.1:0/53807621 wait complete. 2024-05-09T05:13:44.861 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:13:44.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:13:44.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:13:44.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.989+0000 7f6ca4a5b640 1 Processor -- start 2024-05-09T05:13:44.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.989+0000 7f6ca4a5b640 1 -- start start 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.991+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca00c7990 0x7f6ca00c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.991+0000 7f6ca4a5b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca015c7b0 con 0x7f6ca00b4ec0 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.991+0000 7f6ca4a5b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ca00cca50 con 0x7f6ca00ce010 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.991+0000 7f6ca4a5b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca015d0c0 con 0x7f6ca00c7990 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.991+0000 7f6c9e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6ca00b4ec0 0x7f6ca00cf9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39792/0 (socket says 172.21.15.1:39792) 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9e59c640 1 -- 172.21.15.1:0/4279115544 learned_addr learned my addr 172.21.15.1:0/4279115544 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9ed9d640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca00c7990 0x7f6ca00c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3208897524 0 0) 0x7f6ca015c7b0 con 0x7f6ca00b4ec0 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6c8c003520 con 0x7f6ca00b4ec0 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1903126262 0 0) 0x7f6ca00cca50 con 0x7f6ca00ce010 2024-05-09T05:13:44.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6ca015c7b0 con 0x7f6ca00ce010 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2787808541 0 0) 0x7f6c8c003520 con 0x7f6ca00b4ec0 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca00c7990 msgr2=0x7f6ca00c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca00c7990 0x7f6ca00c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 >> v1:172.21.15.196:6789/0 conn(0x7f6ca00ce010 legacy=0x7f6ca00c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca00ccd30 con 0x7f6ca00b4ec0 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f6c900016f0 con 0x7f6ca00b4ec0 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 >> v1:172.21.15.1:6789/0 conn(0x7f6ca00b4ec0 legacy=0x7f6ca00cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c8c003c10 0x7f6c8c004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c8c004800 0x7f6c8c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca00c7990 0x7f6c8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca015d0c0 con 0x7f6c8c003c10 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6c8c0054b0 con 0x7f6c8c004800 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.992+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6c8c00bf00 con 0x7f6ca00c7990 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.993+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c8c003c10 0x7f6c8c004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.993+0000 7f6c9ed9d640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca00c7990 0x7f6c8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.993+0000 7f6c97fff640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c8c004800 0x7f6c8c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.993+0000 7f6c9e59c640 1 -- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca00c7990 msgr2=0x7f6c8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.993+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca00c7990 0x7f6c8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.993+0000 7f6c9e59c640 1 -- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c8c004800 msgr2=0x7f6c8c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.993+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6c8c004800 0x7f6c8c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.993+0000 7f6c9e59c640 1 -- 172.21.15.1:0/4279115544 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c8c003520 con 0x7f6c8c003c10 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.994+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c8c003c10 0x7f6c8c004060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000c160 tx=0x7f6c8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39113e7bd1f641ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.994+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90007ca0 con 0x7f6c8c003c10 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.994+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c90005af0 con 0x7f6c8c003c10 2024-05-09T05:13:44.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.994+0000 7f6c9dd9b640 1 -- 172.21.15.1:0/4279115544 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90007070 con 0x7f6c8c003c10 2024-05-09T05:13:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.994+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/4279115544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c8c003c10 msgr2=0x7f6c8c004060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.994+0000 7f6ca4a5b640 1 --2- 172.21.15.1:0/4279115544 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6c8c003c10 0x7f6c8c004060 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6c9000c160 tx=0x7f6c8c005830 comp rx=0 tx=0).stop 2024-05-09T05:13:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.994+0000 7f6c9ed9d640 1 -- 172.21.15.1:0/4279115544 reap_dead start 2024-05-09T05:13:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.995+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/4279115544 shutdown_connections 2024-05-09T05:13:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.995+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/4279115544 >> 172.21.15.1:0/4279115544 conn(0x7f6ca00d2360 msgr2=0x7f6ca00ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.995+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/4279115544 shutdown_connections 2024-05-09T05:13:44.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.995+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/4279115544 wait complete. 2024-05-09T05:13:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.997+0000 7f6ca4a5b640 1 Processor -- start 2024-05-09T05:13:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.997+0000 7f6ca4a5b640 1 -- start start 2024-05-09T05:13:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.998+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6c8c003c10 0x7f6ca01f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.998+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ca01507a0 0x7f6ca0151410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6ca4a5b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01e83d0 0x7f6ca014a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6ca4a5b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca00cc750 con 0x7f6ca01e83d0 2024-05-09T05:13:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6ca4a5b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6ca015cd50 con 0x7f6ca01507a0 2024-05-09T05:13:45.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6ca4a5b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca0156ae0 con 0x7f6c8c003c10 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01e83d0 0x7f6ca014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01e83d0 0x7f6ca014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53314/0 (socket says 172.21.15.1:53314) 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 -- 172.21.15.1:0/668884462 learned_addr learned my addr 172.21.15.1:0/668884462 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6c8c003c10 0x7f6ca01f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c97fff640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ca01507a0 0x7f6ca0151410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 -- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6c8c003c10 msgr2=0x7f6ca01f68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6c8c003c10 0x7f6ca01f68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 -- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ca01507a0 msgr2=0x7f6ca0151410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6ca01507a0 0x7f6ca0151410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 -- 172.21.15.1:0/668884462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ca01f6e30 con 0x7f6ca01e83d0 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c9ed9d640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01e83d0 0x7f6ca014a360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ca015a3a0 tx=0x7f6c9800f580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003990 con 0x7f6ca01e83d0 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6c9801c070 con 0x7f6ca01e83d0 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:44.999+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ca0151950 con 0x7f6ca01e83d0 2024-05-09T05:13:45.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.000+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c98003d50 con 0x7f6ca01e83d0 2024-05-09T05:13:45.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.000+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ca01f3000 con 0x7f6ca01e83d0 2024-05-09T05:13:45.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.001+0000 7f6ca4a5b640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca0151b20 0x7f6ca014a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.001+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca01f2d90 con 0x7f6ca0151b20 2024-05-09T05:13:45.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.001+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca0151b20 0x7f6ca014a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.001+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6c98004030 con 0x7f6ca01e83d0 2024-05-09T05:13:45.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.001+0000 7f6c9e59c640 1 -- 172.21.15.1:0/668884462 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6ca014ade0 con 0x7f6ca0151b20 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca0151b20 0x7f6ca014a8a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6c90007fd0 tx=0x7f6ca00c86d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c96ffd640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6c80076300 0x7f6c80078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6c98097fe0 con 0x7f6ca01e83d0 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c97fff640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6c80076300 0x7f6c80078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90041020 con 0x7f6ca0151b20 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6c90038040 con 0x7f6ca0151b20 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca0151b20 msgr2=0x7f6ca014a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c96ffd640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca0151b20 0x7f6ca014a8a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f6c90007fd0 tx=0x7f6ca00c86d0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca0151b20 msgr2=0x7f6ca014a8a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.002+0000 7f6c96ffd640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca0151b20 0x7f6ca014a8a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.003+0000 7f6c97fff640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6c80076300 0x7f6c80078810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0150400 tx=0x7f6c88009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:45.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.010+0000 7f6ca4a5b640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca01e87e0 0x7f6ca01f32e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.010+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6ca00cd2b0 con 0x7f6ca01e87e0 2024-05-09T05:13:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.010+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca01e87e0 0x7f6ca01f32e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.010+0000 7f6c9e59c640 1 -- 172.21.15.1:0/668884462 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f6ca00cc000 con 0x7f6ca01e87e0 2024-05-09T05:13:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.011+0000 7f6c9e59c640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca01e87e0 0x7f6ca01f32e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6c90001cf0 tx=0x7f6c9002e000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.011+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c90048e60 con 0x7f6ca01e87e0 2024-05-09T05:13:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.011+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f6c900503f0 con 0x7f6ca01e87e0 2024-05-09T05:13:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.011+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca01e87e0 msgr2=0x7f6ca01f32e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.011+0000 7f6c96ffd640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca01e87e0 0x7f6ca01f32e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6c90001cf0 tx=0x7f6c9002e000 comp rx=0 tx=0).stop 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.011+0000 7f6c96ffd640 1 -- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca01e87e0 msgr2=0x7f6ca01f32e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.011+0000 7f6c96ffd640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ca01e87e0 0x7f6ca01f32e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 62, 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 383009, 2024-05-09T05:13:45.012 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:45.013 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:45.014 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:13:45.015 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:13:45.016 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:45.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:45.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:45.020 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:13:45.021 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:45.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:45.023 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:45.024 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:13:45.025 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6c80076300 msgr2=0x7f6c80078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6c80076300 0x7f6c80078810 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f6ca0150400 tx=0x7f6c88009040 comp rx=0 tx=0).stop 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6c9ed9d640 1 -- 172.21.15.1:0/668884462 reap_dead start 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01e83d0 msgr2=0x7f6ca014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01e83d0 0x7f6ca014a360 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ca015a3a0 tx=0x7f6c9800f580 comp rx=0 tx=0).stop 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 shutdown_connections 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 --2- 172.21.15.1:0/668884462 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ca01e83d0 0x7f6ca014a360 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 >> 172.21.15.1:0/668884462 conn(0x7f6ca00d2360 msgr2=0x7f6ca015b7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 shutdown_connections 2024-05-09T05:13:45.027 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.012+0000 7f6ca4a5b640 1 -- 172.21.15.1:0/668884462 wait complete. 2024-05-09T05:13:45.028 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:13:45.028 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-05-09T05:13:45.028 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:13:45.028 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-05-09T05:13:45.028 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:13:45.029 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:13:45.029 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:13:45.048 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:13:45.048 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:13:45.048 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:13:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.176+0000 7fcd7ec61640 1 Processor -- start 2024-05-09T05:13:45.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.176+0000 7fcd7ec61640 1 -- start start 2024-05-09T05:13:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.178+0000 7fcd7ec61640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcd7806dd70 0x7fcd780cbff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.178+0000 7fcd7ec61640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd780cd2b0 0x7fcd78167180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.178+0000 7fcd7ec61640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcd780cd0a0 con 0x7fcd780cd7f0 2024-05-09T05:13:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.178+0000 7fcd7ec61640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcd780cc530 con 0x7fcd780cd2b0 2024-05-09T05:13:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.178+0000 7fcd7ec61640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fcd780cce00 con 0x7fcd7806dd70 2024-05-09T05:13:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.178+0000 7fcd7d1fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd780cd2b0 0x7fcd78167180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.178+0000 7fcd7d1fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcd780cd2b0 0x7fcd78167180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53392/0 (socket says 172.21.15.1:53392) 2024-05-09T05:13:45.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.178+0000 7fcd7d1fe640 1 -- 172.21.15.1:0/251834276 learned_addr learned my addr 172.21.15.1:0/251834276 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.179+0000 7fcd7c9fd640 1 --2- 172.21.15.1:0/251834276 >> v2:172.21.15.196:3300/0 conn(0x7fcd7806dd70 0x7fcd780cbff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.179+0000 7fcd7d1fe640 1 -- 172.21.15.1:0/251834276 >> v2:172.21.15.196:3300/0 conn(0x7fcd7806dd70 msgr2=0x7fcd780cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.179+0000 7fcd7d1fe640 1 --2- 172.21.15.1:0/251834276 >> v2:172.21.15.196:3300/0 conn(0x7fcd7806dd70 0x7fcd780cbff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.179+0000 7fcd7d1fe640 1 -- 172.21.15.1:0/251834276 >> v1:172.21.15.196:6789/0 conn(0x7fcd780cd7f0 legacy=0x7fcd780c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.179+0000 7fcd7d1fe640 1 -- 172.21.15.1:0/251834276 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd780c56a0 con 0x7fcd780cd2b0 2024-05-09T05:13:45.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.179+0000 7fcd7c9fd640 1 --2- 172.21.15.1:0/251834276 >> v2:172.21.15.196:3300/0 conn(0x7fcd7806dd70 0x7fcd780cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:13:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.180+0000 7fcd7d1fe640 1 --2- 172.21.15.1:0/251834276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd780cd2b0 0x7fcd78167180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd78074740 tx=0x7fcd7002a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f542a4a1ff864a8a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.180+0000 7fcd6f7fe640 1 -- 172.21.15.1:0/251834276 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70036070 con 0x7fcd780cd2b0 2024-05-09T05:13:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.180+0000 7fcd6f7fe640 1 -- 172.21.15.1:0/251834276 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70031600 con 0x7fcd780cd2b0 2024-05-09T05:13:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.180+0000 7fcd6f7fe640 1 -- 172.21.15.1:0/251834276 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd700319d0 con 0x7fcd780cd2b0 2024-05-09T05:13:45.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.181+0000 7fcd7ec61640 1 -- 172.21.15.1:0/251834276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd780cd2b0 msgr2=0x7fcd78167180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.181+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/251834276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd780cd2b0 0x7fcd78167180 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fcd78074740 tx=0x7fcd7002a730 comp rx=0 tx=0).stop 2024-05-09T05:13:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.181+0000 7fcd7ec61640 1 -- 172.21.15.1:0/251834276 shutdown_connections 2024-05-09T05:13:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.181+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/251834276 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd780cd2b0 0x7fcd78167180 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.181+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/251834276 >> v2:172.21.15.196:3300/0 conn(0x7fcd7806dd70 0x7fcd780cbff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.181+0000 7fcd7ec61640 1 -- 172.21.15.1:0/251834276 >> 172.21.15.1:0/251834276 conn(0x7fcd780ce110 msgr2=0x7fcd780ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.181+0000 7fcd7ec61640 1 -- 172.21.15.1:0/251834276 shutdown_connections 2024-05-09T05:13:45.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.181+0000 7fcd7ec61640 1 -- 172.21.15.1:0/251834276 wait complete. 2024-05-09T05:13:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.183+0000 7fcd7ec61640 1 Processor -- start 2024-05-09T05:13:45.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.183+0000 7fcd7ec61640 1 -- start start 2024-05-09T05:13:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.184+0000 7fcd7ec61640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd780cd2b0 0x7fcd7815b010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.184+0000 7fcd7ec61640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd780cd7f0 0x7fcd780c9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.184+0000 7fcd7ec61640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd78157170 0x7fcd78148930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.184+0000 7fcd7ec61640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780c5b00 con 0x7fcd78157170 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.184+0000 7fcd7ec61640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcd780c5980 con 0x7fcd780cd7f0 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.184+0000 7fcd7ec61640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcd78074a20 con 0x7fcd780cd2b0 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.184+0000 7fcd7c9fd640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd780cd2b0 msgr2=0x7fcd7815b010 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.184+0000 7fcd7c9fd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd780cd2b0 0x7fcd7815b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd78157170 0x7fcd78148930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd78157170 0x7fcd78148930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53402/0 (socket says 172.21.15.1:53402) 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 -- 172.21.15.1:0/3079198927 learned_addr learned my addr 172.21.15.1:0/3079198927 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd6ffff640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd780cd7f0 0x7fcd780c9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 -- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd780cd2b0 msgr2=0x7fcd7815b010 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd780cd2b0 0x7fcd7815b010 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 -- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd780cd7f0 msgr2=0x7fcd780c9ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd780cd7f0 0x7fcd780c9ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 -- 172.21.15.1:0/3079198927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd78158ac0 con 0x7fcd78157170 2024-05-09T05:13:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7d1fe640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd78157170 0x7fcd78148930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd7002ad20 tx=0x7fcd70004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd6d7fa640 1 -- 172.21.15.1:0/3079198927 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd7003f020 con 0x7fcd78157170 2024-05-09T05:13:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd6d7fa640 1 -- 172.21.15.1:0/3079198927 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcd70037040 con 0x7fcd78157170 2024-05-09T05:13:45.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd7815b550 con 0x7fcd78157170 2024-05-09T05:13:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd6d7fa640 1 -- 172.21.15.1:0/3079198927 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd70036070 con 0x7fcd78157170 2024-05-09T05:13:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.185+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd78150f90 con 0x7fcd78157170 2024-05-09T05:13:45.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.187+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcd78158c90 con 0x7fcd78157170 2024-05-09T05:13:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.187+0000 7fcd6d7fa640 1 -- 172.21.15.1:0/3079198927 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcd70042020 con 0x7fcd78157170 2024-05-09T05:13:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.188+0000 7fcd6d7fa640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd640762b0 0x7fcd640787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.188+0000 7fcd6d7fa640 1 -- 172.21.15.1:0/3079198927 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcd700b7d70 con 0x7fcd78157170 2024-05-09T05:13:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.188+0000 7fcd7c9fd640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd640762b0 0x7fcd640787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:45.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.189+0000 7fcd7c9fd640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd640762b0 0x7fcd640787c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcd74006f00 tx=0x7fcd74006df0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:45.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.194+0000 7fcd6d7fa640 1 -- 172.21.15.1:0/3079198927 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcd70049020 con 0x7fcd78157170 2024-05-09T05:13:45.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.472+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fcd780cc6b0 con 0x7fcd78157170 2024-05-09T05:13:45.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.472+0000 7fcd6d7fa640 1 -- 172.21.15.1:0/3079198927 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fcd700815d0 con 0x7fcd78157170 2024-05-09T05:13:45.472 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd640762b0 msgr2=0x7fcd640787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd640762b0 0x7fcd640787c0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fcd74006f00 tx=0x7fcd74006df0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd78157170 msgr2=0x7fcd78148930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd78157170 0x7fcd78148930 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fcd7002ad20 tx=0x7fcd70004a40 comp rx=0 tx=0).stop 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 shutdown_connections 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcd640762b0 0x7fcd640787c0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcd780cd7f0 0x7fcd780c9ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcd78157170 0x7fcd78148930 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 --2- 172.21.15.1:0/3079198927 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcd780cd2b0 0x7fcd7815b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:45.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 >> 172.21.15.1:0/3079198927 conn(0x7fcd780ce110 msgr2=0x7fcd7814d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:45.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 shutdown_connections 2024-05-09T05:13:45.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:45.474+0000 7fcd7ec61640 1 -- 172.21.15.1:0/3079198927 wait complete. 2024-05-09T05:13:48.487 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:13:48.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.614+0000 7f07a1fb7640 1 Processor -- start 2024-05-09T05:13:48.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.614+0000 7f07a1fb7640 1 -- start start 2024-05-09T05:13:48.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f07a1fb7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c0c5ca0 0x7f079c0ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f07a1fb7640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f079c156a80 0x7f079c06df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f07a1fb7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f079c0c86a0 con 0x7f079c156e90 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f07a1fb7640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f079c0cb910 con 0x7f079c156a80 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f07a1fb7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f079c0c8980 con 0x7f079c0c5ca0 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f079b7fe640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c0c5ca0 msgr2=0x7f079c0ca4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f079b7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c0c5ca0 0x7f079c0ca4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f079bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f079c156e90 0x7f079c0c7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39800/0 (socket says 172.21.15.1:39800) 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f079bfff640 1 -- 172.21.15.1:0/2504900177 learned_addr learned my addr 172.21.15.1:0/2504900177 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f0793fff640 1 --2- 172.21.15.1:0/2504900177 >> v2:172.21.15.196:3300/0 conn(0x7f079c156a80 0x7f079c06df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f079affd640 1 -- 172.21.15.1:0/2504900177 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1456074466 0 0) 0x7f079c0c86a0 con 0x7f079c156e90 2024-05-09T05:13:48.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.616+0000 7f079affd640 1 -- 172.21.15.1:0/2504900177 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0788003520 con 0x7f079c156e90 2024-05-09T05:13:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.617+0000 7f079affd640 1 -- 172.21.15.1:0/2504900177 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1079557163 0 0) 0x7f0788003520 con 0x7f079c156e90 2024-05-09T05:13:48.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.617+0000 7f0793fff640 1 -- 172.21.15.1:0/2504900177 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c0c5ca0 msgr2=0x7f079c0ca4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:13:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.617+0000 7f0793fff640 1 --2- 172.21.15.1:0/2504900177 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c0c5ca0 0x7f079c0ca4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.617+0000 7f0793fff640 1 -- 172.21.15.1:0/2504900177 >> v1:172.21.15.1:6789/0 conn(0x7f079c156e90 legacy=0x7f079c0c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.617+0000 7f0793fff640 1 -- 172.21.15.1:0/2504900177 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c0c8c60 con 0x7f079c156a80 2024-05-09T05:13:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.618+0000 7f0793fff640 1 --2- 172.21.15.1:0/2504900177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156a80 0x7f079c06df20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f07840097a0 tx=0x7f078402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d17b081682c9f9c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:48.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.618+0000 7f079affd640 1 -- 172.21.15.1:0/2504900177 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0784037070 con 0x7f079c156a80 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.618+0000 7f079affd640 1 -- 172.21.15.1:0/2504900177 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078402ac50 con 0x7f079c156a80 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.618+0000 7f079affd640 1 -- 172.21.15.1:0/2504900177 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0794001a70 con 0x7f079c156a80 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.618+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2504900177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156a80 msgr2=0x7f079c06df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.618+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2504900177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156a80 0x7f079c06df20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f07840097a0 tx=0x7f078402a1c0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.619+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2504900177 shutdown_connections 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.619+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2504900177 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156a80 0x7f079c06df20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.619+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2504900177 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c0c5ca0 0x7f079c0ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.619+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2504900177 >> 172.21.15.1:0/2504900177 conn(0x7f079c0745a0 msgr2=0x7f079c071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.619+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2504900177 shutdown_connections 2024-05-09T05:13:48.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.619+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2504900177 wait complete. 2024-05-09T05:13:48.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.620+0000 7f07a1fb7640 1 Processor -- start 2024-05-09T05:13:48.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.621+0000 7f07a1fb7640 1 -- start start 2024-05-09T05:13:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f07a1fb7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f079c0c5ca0 0x7f079c1f6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f07a1fb7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c156a80 0x7f079c1f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f07a1fb7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156e90 0x7f079c14dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f07a1fb7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f079c0cad60 con 0x7f079c0c5ca0 2024-05-09T05:13:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f07a1fb7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f079c0cb5a0 con 0x7f079c156e90 2024-05-09T05:13:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f07a1fb7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f079c0ca9e0 con 0x7f079c156a80 2024-05-09T05:13:48.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f079c0c5ca0 0x7f079c1f6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f079c0c5ca0 0x7f079c1f6510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53412/0 (socket says 172.21.15.1:53412) 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f0793fff640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c156a80 msgr2=0x7f079c1f2bc0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079b7fe640 1 -- 172.21.15.1:0/2140659416 learned_addr learned my addr 172.21.15.1:0/2140659416 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f0793fff640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c156a80 0x7f079c1f2bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079bfff640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156e90 0x7f079c14dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079b7fe640 1 -- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c156a80 msgr2=0x7f079c1f2bc0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079b7fe640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c156a80 0x7f079c1f2bc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079b7fe640 1 -- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156e90 msgr2=0x7f079c14dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079b7fe640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156e90 0x7f079c14dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079b7fe640 1 -- 172.21.15.1:0/2140659416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f079c1f3100 con 0x7f079c0c5ca0 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.622+0000 7f079bfff640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156e90 0x7f079c14dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.623+0000 7f079b7fe640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f079c0c5ca0 0x7f079c1f6510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f078c00b5b0 tx=0x7f078c00ba70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:48.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.623+0000 7f0798ff9640 1 -- 172.21.15.1:0/2140659416 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c013020 con 0x7f079c0c5ca0 2024-05-09T05:13:48.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.623+0000 7f0798ff9640 1 -- 172.21.15.1:0/2140659416 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f078c003de0 con 0x7f079c0c5ca0 2024-05-09T05:13:48.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.623+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f079c0ce9c0 con 0x7f079c0c5ca0 2024-05-09T05:13:48.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.623+0000 7f0798ff9640 1 -- 172.21.15.1:0/2140659416 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f078c0052b0 con 0x7f079c0c5ca0 2024-05-09T05:13:48.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.623+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f079c0cef80 con 0x7f079c0c5ca0 2024-05-09T05:13:48.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.624+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f079c050870 con 0x7f079c0c5ca0 2024-05-09T05:13:48.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.625+0000 7f0798ff9640 1 -- 172.21.15.1:0/2140659416 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f078c016020 con 0x7f079c0c5ca0 2024-05-09T05:13:48.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.625+0000 7f0798ff9640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f077c0762b0 0x7f077c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:48.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.625+0000 7f0798ff9640 1 -- 172.21.15.1:0/2140659416 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f078c097530 con 0x7f079c0c5ca0 2024-05-09T05:13:48.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.626+0000 7f0793fff640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f077c0762b0 0x7f077c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:48.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.626+0000 7f0793fff640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f077c0762b0 0x7f077c0787c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0784034000 tx=0x7f078402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:48.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.631+0000 7f0798ff9640 1 -- 172.21.15.1:0/2140659416 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f078c01d020 con 0x7f079c0c5ca0 2024-05-09T05:13:48.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.936+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f079c0cb000 con 0x7f079c0c5ca0 2024-05-09T05:13:48.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.937+0000 7f0798ff9640 1 -- 172.21.15.1:0/2140659416 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f078c060d80 con 0x7f079c0c5ca0 2024-05-09T05:13:48.937 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:13:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f077c0762b0 msgr2=0x7f077c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f077c0762b0 0x7f077c0787c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f0784034000 tx=0x7f078402e000 comp rx=0 tx=0).stop 2024-05-09T05:13:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f079c0c5ca0 msgr2=0x7f079c1f6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f079c0c5ca0 0x7f079c1f6510 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f078c00b5b0 tx=0x7f078c00ba70 comp rx=0 tx=0).stop 2024-05-09T05:13:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 shutdown_connections 2024-05-09T05:13:48.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f077c0762b0 0x7f077c0787c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f079c156e90 0x7f079c14dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f079c156a80 0x7f079c1f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 --2- 172.21.15.1:0/2140659416 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f079c0c5ca0 0x7f079c1f6510 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 >> 172.21.15.1:0/2140659416 conn(0x7f079c0745a0 msgr2=0x7f079c15aea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 shutdown_connections 2024-05-09T05:13:48.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:48.939+0000 7f07a1fb7640 1 -- 172.21.15.1:0/2140659416 wait complete. 2024-05-09T05:13:51.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:13:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.080+0000 7f220ff5f640 1 Processor -- start 2024-05-09T05:13:52.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.080+0000 7f220ff5f640 1 -- start start 2024-05-09T05:13:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.081+0000 7f220ff5f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f22080d16b0 0x7f22080d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.081+0000 7f220ff5f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f220815ca90 0x7f220815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.081+0000 7f220ff5f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f220806f6d0 con 0x7f220806f1b0 2024-05-09T05:13:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.081+0000 7f220ff5f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f22080cee50 con 0x7f22080d16b0 2024-05-09T05:13:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.081+0000 7f220ff5f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f22080cd0e0 con 0x7f220815ca90 2024-05-09T05:13:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.082+0000 7f220dcfb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f22080d16b0 0x7f22080d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:52.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.082+0000 7f220dcfb640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f22080d16b0 0x7f22080d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37432/0 (socket says 172.21.15.1:37432) 2024-05-09T05:13:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.082+0000 7f220dcfb640 1 -- 172.21.15.1:0/3560267261 learned_addr learned my addr 172.21.15.1:0/3560267261 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.082+0000 7f220e4fc640 1 --2- 172.21.15.1:0/3560267261 >> v2:172.21.15.196:3300/0 conn(0x7f220815ca90 0x7f220815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.082+0000 7f220dcfb640 1 -- 172.21.15.1:0/3560267261 >> v2:172.21.15.196:3300/0 conn(0x7f220815ca90 msgr2=0x7f220815cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.082+0000 7f220dcfb640 1 --2- 172.21.15.1:0/3560267261 >> v2:172.21.15.196:3300/0 conn(0x7f220815ca90 0x7f220815cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.082+0000 7f220dcfb640 1 -- 172.21.15.1:0/3560267261 >> v1:172.21.15.196:6789/0 conn(0x7f220806f1b0 legacy=0x7f22080ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:52.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.082+0000 7f220dcfb640 1 -- 172.21.15.1:0/3560267261 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22080cc4d0 con 0x7f22080d16b0 2024-05-09T05:13:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.083+0000 7f220dcfb640 1 --2- 172.21.15.1:0/3560267261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22080d16b0 0x7f22080d03d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f22080b7cc0 tx=0x7f220402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b60905cd3c969dd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.083+0000 7f220ccf9640 1 -- 172.21.15.1:0/3560267261 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204037070 con 0x7f22080d16b0 2024-05-09T05:13:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.083+0000 7f220ccf9640 1 -- 172.21.15.1:0/3560267261 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f220402ae50 con 0x7f22080d16b0 2024-05-09T05:13:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.083+0000 7f220ccf9640 1 -- 172.21.15.1:0/3560267261 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204032860 con 0x7f22080d16b0 2024-05-09T05:13:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.084+0000 7f220ff5f640 1 -- 172.21.15.1:0/3560267261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22080d16b0 msgr2=0x7f22080d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:52.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.084+0000 7f220ff5f640 1 --2- 172.21.15.1:0/3560267261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22080d16b0 0x7f22080d03d0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f22080b7cc0 tx=0x7f220402a300 comp rx=0 tx=0).stop 2024-05-09T05:13:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.084+0000 7f220ff5f640 1 -- 172.21.15.1:0/3560267261 shutdown_connections 2024-05-09T05:13:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.084+0000 7f220ff5f640 1 --2- 172.21.15.1:0/3560267261 >> v2:172.21.15.196:3300/0 conn(0x7f220815ca90 0x7f220815cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.084+0000 7f220ff5f640 1 --2- 172.21.15.1:0/3560267261 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22080d16b0 0x7f22080d03d0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.084+0000 7f220ff5f640 1 -- 172.21.15.1:0/3560267261 >> 172.21.15.1:0/3560267261 conn(0x7f22080b3660 msgr2=0x7f22080b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.084+0000 7f220ff5f640 1 -- 172.21.15.1:0/3560267261 shutdown_connections 2024-05-09T05:13:52.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.084+0000 7f220ff5f640 1 -- 172.21.15.1:0/3560267261 wait complete. 2024-05-09T05:13:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.086+0000 7f220ff5f640 1 Processor -- start 2024-05-09T05:13:52.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.086+0000 7f220ff5f640 1 -- start start 2024-05-09T05:13:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220ff5f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f220806f1b0 0x7f22080b5740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220ff5f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22080d16b0 0x7f2208151cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220ff5f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f220815ca90 0x7f2208152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220dcfb640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f220806f1b0 msgr2=0x7f22080b5740 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220ff5f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22080ccd70 con 0x7f220815ca90 2024-05-09T05:13:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220ff5f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f22080cca90 con 0x7f22080d16b0 2024-05-09T05:13:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220dcfb640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f220806f1b0 0x7f22080b5740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:52.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220ff5f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f22080d0910 con 0x7f220806f1b0 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f220815ca90 0x7f2208152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f220815ca90 0x7f2208152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37436/0 (socket says 172.21.15.1:37436) 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 -- 172.21.15.1:0/497187201 learned_addr learned my addr 172.21.15.1:0/497187201 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220d4fa640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22080d16b0 0x7f2208151cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 -- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f220806f1b0 msgr2=0x7f22080b5740 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f220806f1b0 0x7f22080b5740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 -- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22080d16b0 msgr2=0x7f2208151cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22080d16b0 0x7f2208151cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 -- 172.21.15.1:0/497187201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2208152770 con 0x7f220815ca90 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220d4fa640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22080d16b0 0x7f2208151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.088+0000 7f220e4fc640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f220815ca90 0x7f2208152230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2200009cb0 tx=0x7f220000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.089+0000 7f21fe7fc640 1 -- 172.21.15.1:0/497187201 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200018020 con 0x7f220815ca90 2024-05-09T05:13:52.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.089+0000 7f21fe7fc640 1 -- 172.21.15.1:0/497187201 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2200014480 con 0x7f220815ca90 2024-05-09T05:13:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.089+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2208159260 con 0x7f220815ca90 2024-05-09T05:13:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.089+0000 7f21fe7fc640 1 -- 172.21.15.1:0/497187201 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2200014870 con 0x7f220815ca90 2024-05-09T05:13:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.089+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2208159820 con 0x7f220815ca90 2024-05-09T05:13:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.090+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2208050870 con 0x7f220815ca90 2024-05-09T05:13:52.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.091+0000 7f21fe7fc640 1 -- 172.21.15.1:0/497187201 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2200005d10 con 0x7f220815ca90 2024-05-09T05:13:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.092+0000 7f21fe7fc640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f21ec076280 0x7f21ec078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.092+0000 7f21fe7fc640 1 -- 172.21.15.1:0/497187201 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2200098360 con 0x7f220815ca90 2024-05-09T05:13:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.092+0000 7f220dcfb640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f21ec076280 0x7f21ec078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:52.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.092+0000 7f220dcfb640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f21ec076280 0x7f21ec078790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2204034000 tx=0x7f220402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:52.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.097+0000 7f21fe7fc640 1 -- 172.21.15.1:0/497187201 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2200022030 con 0x7f220815ca90 2024-05-09T05:13:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.457+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f2208158fb0 con 0x7f220815ca90 2024-05-09T05:13:52.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.458+0000 7f21fe7fc640 1 -- 172.21.15.1:0/497187201 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f2200010040 con 0x7f220815ca90 2024-05-09T05:13:52.458 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":62,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":19,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:13:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f21ec076280 msgr2=0x7f21ec078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f21ec076280 0x7f21ec078790 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f2204034000 tx=0x7f220402e000 comp rx=0 tx=0).stop 2024-05-09T05:13:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f220815ca90 msgr2=0x7f2208152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f220815ca90 0x7f2208152230 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f2200009cb0 tx=0x7f220000c490 comp rx=0 tx=0).stop 2024-05-09T05:13:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 shutdown_connections 2024-05-09T05:13:52.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f21ec076280 0x7f21ec078790 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22080d16b0 0x7f2208151cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f220815ca90 0x7f2208152230 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 --2- 172.21.15.1:0/497187201 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f220806f1b0 0x7f22080b5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 >> 172.21.15.1:0/497187201 conn(0x7f22080b3660 msgr2=0x7f220806dda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 shutdown_connections 2024-05-09T05:13:52.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:52.460+0000 7f220ff5f640 1 -- 172.21.15.1:0/497187201 wait complete. 2024-05-09T05:13:55.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:13:55.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.592+0000 7f16f1619640 1 Processor -- start 2024-05-09T05:13:55.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.592+0000 7f16f1619640 1 -- start start 2024-05-09T05:13:55.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16f1619640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 0x7f16ec15e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:55.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16f1619640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16f1619640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16ec0cb060 con 0x7f16ec0c7eb0 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16f1619640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f16ec15ec50 con 0x7f16ec154850 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16f1619640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f16ec15a820 con 0x7f16ec0ca9b0 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16e3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16eaffd640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 msgr2=0x7f16ec15e710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16e3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37438/0 (socket says 172.21.15.1:37438) 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 learned_addr learned my addr 172.21.15.1:0/1309861705 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 0x7f16ec15e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16eb7fe640 1 --1- 172.21.15.1:0/1309861705 >> v1:172.21.15.1:6789/0 conn(0x7f16ec0c7eb0 0x7f16ec0cb2b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:39822/0 (socket says 172.21.15.1:39822) 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:13:55.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.594+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.795+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.795+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 msgr2=0x7f16ec15e710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.795+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 0x7f16ec15e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:13:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.795+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:13:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.795+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:13:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.795+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:13:55.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:55.795+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:13:56.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.196+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 msgr2=0x7f16ec15e710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:56.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.196+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:56.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.196+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 0x7f16ec15e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:13:56.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.196+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:13:56.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.196+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:13:56.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.196+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:13:56.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.196+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:13:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.997+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 msgr2=0x7f16ec15e710 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.997+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 0x7f16ec15e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:13:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.997+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.997+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:13:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.997+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:13:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.997+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:13:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:56.997+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:13:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16e97fa640 1 -- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 msgr2=0x7f16ec15e710 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:13:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16e97fa640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16ec0ca9b0 0x7f16ec15e710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16e97fa640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 msgr2=0x7f16ec15afe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16e97fa640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.1:3300/0 conn(0x7f16ec154850 0x7f16ec15afe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16e97fa640 1 -- 172.21.15.1:0/1309861705 >> v1:172.21.15.1:6789/0 conn(0x7f16ec0c7eb0 legacy=0x7f16ec0cb2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16e97fa640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16e97fa640 1 -- 172.21.15.1:0/1309861705 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f16ec0cadc0 con 0x7f16cc0061e0 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16e97fa640 1 -- 172.21.15.1:0/1309861705 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f16ec15ec50 con 0x7f16cc0036c0 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.594+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.595+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 msgr2=0x7f16cc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.595+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 msgr2=0x7f16cc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.595+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:13:58.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.595+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:58.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.796+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:58.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.796+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 msgr2=0x7f16cc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:13:58.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.796+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 msgr2=0x7f16cc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:13:58.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.796+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:13:58.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:58.796+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:13:59.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.197+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:59.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.197+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 msgr2=0x7f16cc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:13:59.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.197+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 msgr2=0x7f16cc005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:13:59.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.197+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:13:59.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.197+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:13:59.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.835+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 922370843 0 0) 0x7f16ec0cadc0 con 0x7f16cc0061e0 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.835+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f16d8003520 con 0x7f16cc0061e0 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3827286326 0 0) 0x7f16d8003520 con 0x7f16cc0061e0 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 msgr2=0x7f16cc005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 --2- 172.21.15.1:0/1309861705 >> v2:172.21.15.196:3300/0 conn(0x7f16cc0036c0 0x7f16cc005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16cc00afa0 con 0x7f16cc0061e0 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f16d400f040 con 0x7f16cc0061e0 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 >> v1:172.21.15.196:6789/0 conn(0x7f16cc0061e0 legacy=0x7f16cc0086f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16eb7fe640 1 -- 172.21.15.1:0/1309861705 reap_dead start 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16d8003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 0x7f16d8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16d800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec15ec50 con 0x7f16d8005350 2024-05-09T05:13:59.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f16d800c280 con 0x7f16d80039b0 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f16d800c480 con 0x7f16d8004800 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16e3fff640 1 -- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 msgr2=0x7f16d8004c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16e3fff640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 0x7f16d8004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.836+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16d8003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.837+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 msgr2=0x7f16d8004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.837+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 0x7f16d8004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.837+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 msgr2=0x7f16d800b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.837+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16d800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:59.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.837+0000 7f16eaffd640 1 -- 172.21.15.1:0/1309861705 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16d8003520 con 0x7f16d80039b0 2024-05-09T05:13:59.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.837+0000 7f16eaffd640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16d8003e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009ef0 tx=0x7f16d8005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb9df78a9192b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:59.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.838+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc031070 con 0x7f16d80039b0 2024-05-09T05:13:59.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.838+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16dc02bbe0 con 0x7f16d80039b0 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.838+0000 7f16ea7fc640 1 -- 172.21.15.1:0/1309861705 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc009af0 con 0x7f16d80039b0 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 -- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 msgr2=0x7f16d8003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16d8003e00 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f16dc009ef0 tx=0x7f16d8005830 comp rx=0 tx=0).stop 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 -- 172.21.15.1:0/1309861705 shutdown_connections 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16d800b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 0x7f16d8004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 --2- 172.21.15.1:0/1309861705 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16d8003e00 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 -- 172.21.15.1:0/1309861705 >> 172.21.15.1:0/1309861705 conn(0x7f16ec0745a0 msgr2=0x7f16ec071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 -- 172.21.15.1:0/1309861705 shutdown_connections 2024-05-09T05:13:59.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.839+0000 7f16f1619640 1 -- 172.21.15.1:0/1309861705 wait complete. 2024-05-09T05:13:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.841+0000 7f16f1619640 1 Processor -- start 2024-05-09T05:13:59.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.842+0000 7f16f1619640 1 -- start start 2024-05-09T05:13:59.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16f1619640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16ec14f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:59.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16f1619640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 0x7f16ec1e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16f1619640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16ec1e7150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16f1619640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec15aac0 con 0x7f16d8005350 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16f1619640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f16ec0c9810 con 0x7f16d80039b0 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16f1619640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f16ec0c9af0 con 0x7f16d8004800 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16e3fff640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 msgr2=0x7f16ec1e6c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16e3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 0x7f16ec1e6c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16ec1e7150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16eb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16ec1e7150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39574/0 (socket says 172.21.15.1:39574) 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16eaffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16ec14f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.843+0000 7f16eb7fe640 1 -- 172.21.15.1:0/3993104013 learned_addr learned my addr 172.21.15.1:0/3993104013 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16eaffd640 1 -- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 msgr2=0x7f16ec1e6c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16eaffd640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 0x7f16ec1e6c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16eaffd640 1 -- 172.21.15.1:0/3993104013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 msgr2=0x7f16ec1e7150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:13:59.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16eaffd640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16ec1e7150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:13:59.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16eaffd640 1 -- 172.21.15.1:0/3993104013 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16ec14aa30 con 0x7f16d80039b0 2024-05-09T05:13:59.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16eb7fe640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16ec1e7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:13:59.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16eaffd640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16ec14f0b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f16d800c450 tx=0x7f16dc005d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:59.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16e37fe640 1 -- 172.21.15.1:0/3993104013 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc009af0 con 0x7f16d80039b0 2024-05-09T05:13:59.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16e37fe640 1 -- 172.21.15.1:0/3993104013 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f16dc044070 con 0x7f16d80039b0 2024-05-09T05:13:59.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16ec14db80 con 0x7f16d80039b0 2024-05-09T05:13:59.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16e37fe640 1 -- 172.21.15.1:0/3993104013 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16dc03f6a0 con 0x7f16d80039b0 2024-05-09T05:13:59.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.844+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16ec14b260 con 0x7f16d80039b0 2024-05-09T05:13:59.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.846+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16ec050870 con 0x7f16d80039b0 2024-05-09T05:13:59.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.846+0000 7f16e37fe640 1 -- 172.21.15.1:0/3993104013 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f16dc03f840 con 0x7f16d80039b0 2024-05-09T05:13:59.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.846+0000 7f16e37fe640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16cc076b90 0x7f16cc0790a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:13:59.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.847+0000 7f16e37fe640 1 -- 172.21.15.1:0/3993104013 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f16dc0b8810 con 0x7f16d80039b0 2024-05-09T05:13:59.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.847+0000 7f16e3fff640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16cc076b90 0x7f16cc0790a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:13:59.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.847+0000 7f16e3fff640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16cc076b90 0x7f16cc0790a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f16d4000c00 tx=0x7f16d4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:13:59.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:13:59.852+0000 7f16e37fe640 1 -- 172.21.15.1:0/3993104013 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f16dc0820d0 con 0x7f16d80039b0 2024-05-09T05:14:00.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.182+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f16ec0c9f80 con 0x7f16d80039b0 2024-05-09T05:14:00.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.182+0000 7f16e37fe640 1 -- 172.21.15.1:0/3993104013 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f16dc08a0c0 con 0x7f16d80039b0 2024-05-09T05:14:00.182 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:00.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16cc076b90 msgr2=0x7f16cc0790a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16cc076b90 0x7f16cc0790a0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f16d4000c00 tx=0x7f16d4008040 comp rx=0 tx=0).stop 2024-05-09T05:14:00.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 msgr2=0x7f16ec14f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16ec14f0b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f16d800c450 tx=0x7f16dc005d40 comp rx=0 tx=0).stop 2024-05-09T05:14:00.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 shutdown_connections 2024-05-09T05:14:00.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f16cc076b90 0x7f16cc0790a0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f16d8005350 0x7f16ec1e7150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f16d8004800 0x7f16ec1e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 --2- 172.21.15.1:0/3993104013 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f16d80039b0 0x7f16ec14f0b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 >> 172.21.15.1:0/3993104013 conn(0x7f16ec0745a0 msgr2=0x7f16ec158830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:00.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 shutdown_connections 2024-05-09T05:14:00.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.185+0000 7f16f1619640 1 -- 172.21.15.1:0/3993104013 wait complete. 2024-05-09T05:14:00.199 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:14:00.199 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:14:00.199 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:14:00.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.323+0000 7eff897ce640 1 Processor -- start 2024-05-09T05:14:00.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.323+0000 7eff897ce640 1 -- start start 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.324+0000 7eff897ce640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7eff840c5e10 0x7eff8406e090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.324+0000 7eff897ce640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff8406e6b0 con 0x7eff84156910 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.324+0000 7eff897ce640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7eff840c7960 con 0x7eff8406db90 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.324+0000 7eff897ce640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7eff840c7c40 con 0x7eff840c5e10 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff837fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7eff84156910 0x7eff8415c8a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60334/0 (socket says 172.21.15.1:60334) 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff837fe640 1 -- 172.21.15.1:0/379379561 learned_addr learned my addr 172.21.15.1:0/379379561 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff827fc640 1 --2- 172.21.15.1:0/379379561 >> v2:172.21.15.196:3300/0 conn(0x7eff840c5e10 0x7eff8406e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3228872479 0 0) 0x7eff8406e6b0 con 0x7eff84156910 2024-05-09T05:14:00.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7eff6c003520 con 0x7eff84156910 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1134814443 0 0) 0x7eff6c003520 con 0x7eff84156910 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 >> v2:172.21.15.196:3300/0 conn(0x7eff840c5e10 msgr2=0x7eff8406e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 --2- 172.21.15.1:0/379379561 >> v2:172.21.15.196:3300/0 conn(0x7eff840c5e10 0x7eff8406e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 >> v1:172.21.15.196:6789/0 conn(0x7eff8406db90 legacy=0x7eff8406fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff840c7f20 con 0x7eff84156910 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7eff7c001a70 con 0x7eff84156910 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 >> v1:172.21.15.1:6789/0 conn(0x7eff84156910 legacy=0x7eff8415c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c003a50 0x7eff6c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff6c004800 0x7eff6c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c005250 0x7eff6c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff840c7c40 con 0x7eff6c004800 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7eff6c00c1d0 con 0x7eff6c005250 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7eff6c00c3d0 con 0x7eff6c003a50 2024-05-09T05:14:00.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.325+0000 7eff82ffd640 1 -- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c003a50 msgr2=0x7eff6c003ec0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff82ffd640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c003a50 0x7eff6c003ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff827fc640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff6c004800 0x7eff6c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff837fe640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c005250 0x7eff6c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff827fc640 1 -- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c003a50 msgr2=0x7eff6c003ec0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff827fc640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c003a50 0x7eff6c003ec0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff827fc640 1 -- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c005250 msgr2=0x7eff6c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff827fc640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff6c005250 0x7eff6c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff827fc640 1 -- 172.21.15.1:0/379379561 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff6c003520 con 0x7eff6c004800 2024-05-09T05:14:00.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.326+0000 7eff837fe640 1 -- 172.21.15.1:0/379379561 reap_dead start 2024-05-09T05:14:00.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.327+0000 7eff827fc640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff6c004800 0x7eff6c004c10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7eff78009ac0 tx=0x7eff6c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c0564dd9c2d830a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.327+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff78036070 con 0x7eff6c004800 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.327+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff78031400 con 0x7eff6c004800 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.327+0000 7eff81ffb640 1 -- 172.21.15.1:0/379379561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff78009690 con 0x7eff6c004800 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.328+0000 7eff897ce640 1 -- 172.21.15.1:0/379379561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff6c004800 msgr2=0x7eff6c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.328+0000 7eff897ce640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff6c004800 0x7eff6c004c10 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7eff78009ac0 tx=0x7eff6c005830 comp rx=0 tx=0).stop 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.328+0000 7eff897ce640 1 -- 172.21.15.1:0/379379561 shutdown_connections 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.328+0000 7eff897ce640 1 --2- 172.21.15.1:0/379379561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff6c004800 0x7eff6c004c10 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.328+0000 7eff897ce640 1 -- 172.21.15.1:0/379379561 >> 172.21.15.1:0/379379561 conn(0x7eff840ce910 msgr2=0x7eff840ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.328+0000 7eff897ce640 1 -- 172.21.15.1:0/379379561 shutdown_connections 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.328+0000 7eff897ce640 1 -- 172.21.15.1:0/379379561 wait complete. 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.330+0000 7eff897ce640 1 Processor -- start 2024-05-09T05:14:00.330 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.330+0000 7eff897ce640 1 -- start start 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff897ce640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c004800 0x7eff841f6f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff897ce640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff8414f100 0x7eff84150340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff897ce640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff841e7e00 0x7eff8414d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff897ce640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff8406e090 con 0x7eff841e7e00 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff897ce640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7eff84156d20 con 0x7eff8414f100 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff897ce640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7eff8406e210 con 0x7eff6c004800 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff82ffd640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c004800 msgr2=0x7eff841f6f70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff82ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c004800 0x7eff841f6f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff841e7e00 0x7eff8414d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff841e7e00 0x7eff8414d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39590/0 (socket says 172.21.15.1:39590) 2024-05-09T05:14:00.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 -- 172.21.15.1:0/2484207576 learned_addr learned my addr 172.21.15.1:0/2484207576 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff827fc640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff8414f100 0x7eff84150340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 -- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c004800 msgr2=0x7eff841f6f70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7eff6c004800 0x7eff841f6f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 -- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff8414f100 msgr2=0x7eff84150340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7eff8414f100 0x7eff84150340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 -- 172.21.15.1:0/2484207576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff84150880 con 0x7eff841e7e00 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.332+0000 7eff837fe640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff841e7e00 0x7eff8414d840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0013d0 tx=0x7eff7c007a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.333+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c00aae0 con 0x7eff841e7e00 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.333+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7eff7c019070 con 0x7eff841e7e00 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.333+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff7c0133f0 con 0x7eff841e7e00 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.333+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8415bc40 con 0x7eff841e7e00 2024-05-09T05:14:00.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.333+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff8415c1e0 con 0x7eff841e7e00 2024-05-09T05:14:00.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.334+0000 7eff897ce640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff84150a50 0x7eff8414dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.334+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff8406fc30 con 0x7eff84150a50 2024-05-09T05:14:00.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.334+0000 7eff82ffd640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff84150a50 0x7eff8414dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.334+0000 7eff82ffd640 1 -- 172.21.15.1:0/2484207576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7eff8406a3c0 con 0x7eff84150a50 2024-05-09T05:14:00.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.334+0000 7eff82ffd640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff84150a50 0x7eff8414dd80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7eff840cc7e0 tx=0x7eff7400c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.334+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff740075a0 con 0x7eff84150a50 2024-05-09T05:14:00.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.335+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7eff74004020 con 0x7eff84150a50 2024-05-09T05:14:00.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.335+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff84150a50 msgr2=0x7eff8414dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.335+0000 7eff6b7fe640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff84150a50 0x7eff8414dd80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7eff840cc7e0 tx=0x7eff7400c810 comp rx=0 tx=0).stop 2024-05-09T05:14:00.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.335+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff84150a50 msgr2=0x7eff8414dd80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:00.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.335+0000 7eff6b7fe640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff84150a50 0x7eff8414dd80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.335+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7eff7c013590 con 0x7eff841e7e00 2024-05-09T05:14:00.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.336+0000 7eff6b7fe640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff70075d40 0x7eff70078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.336+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7eff7c097a60 con 0x7eff841e7e00 2024-05-09T05:14:00.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.336+0000 7eff82ffd640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff70075d40 0x7eff70078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.337+0000 7eff82ffd640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff70075d40 0x7eff70078250 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7eff84069670 tx=0x7eff7400a000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.342+0000 7eff897ce640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff8415c3b0 0x7eff840cc7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.342+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7eff8406e6b0 con 0x7eff8415c3b0 2024-05-09T05:14:00.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.342+0000 7eff827fc640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff8415c3b0 0x7eff840cc7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.343+0000 7eff827fc640 1 -- 172.21.15.1:0/2484207576 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7eff8406cbc0 con 0x7eff8415c3b0 2024-05-09T05:14:00.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.343+0000 7eff827fc640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff8415c3b0 0x7eff840cc7e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7eff78009a90 tx=0x7eff78004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.343+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff78036040 con 0x7eff8415c3b0 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.343+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5051 (secure 0 0 0) 0x7eff78037040 con 0x7eff8415c3b0 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.343+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff8415c3b0 msgr2=0x7eff840cc7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.343+0000 7eff6b7fe640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff8415c3b0 0x7eff840cc7e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7eff78009a90 tx=0x7eff78004730 comp rx=0 tx=0).stop 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff6b7fe640 1 -- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff8415c3b0 msgr2=0x7eff840cc7e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff6b7fe640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff8415c3b0 0x7eff840cc7e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:14:00.344 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 42538, 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:00.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:00.346 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:00.347 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:00.348 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:00.349 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.350 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:00.351 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:00.352 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:00.353 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:00.354 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.355 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.356 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:00.357 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:00.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff70075d40 msgr2=0x7eff70078250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7eff70075d40 0x7eff70078250 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7eff84069670 tx=0x7eff7400a000 comp rx=0 tx=0).stop 2024-05-09T05:14:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff841e7e00 msgr2=0x7eff8414d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff841e7e00 0x7eff8414d840 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7eff7c0013d0 tx=0x7eff7c007a90 comp rx=0 tx=0).stop 2024-05-09T05:14:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff837fe640 1 -- 172.21.15.1:0/2484207576 reap_dead start 2024-05-09T05:14:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 shutdown_connections 2024-05-09T05:14:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 --2- 172.21.15.1:0/2484207576 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7eff841e7e00 0x7eff8414d840 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 >> 172.21.15.1:0/2484207576 conn(0x7eff840ce910 msgr2=0x7eff841e8cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 shutdown_connections 2024-05-09T05:14:00.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.344+0000 7eff897ce640 1 -- 172.21.15.1:0/2484207576 wait complete. 2024-05-09T05:14:00.359 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:14:00.359 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:14:00.360 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:14:00.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.490+0000 7fe60f684640 1 Processor -- start 2024-05-09T05:14:00.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.490+0000 7fe60f684640 1 -- start start 2024-05-09T05:14:00.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.492+0000 7fe60f684640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe6080c68a0 0x7fe6081569a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.492+0000 7fe60f684640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60806e000 con 0x7fe6081609f0 2024-05-09T05:14:00.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.492+0000 7fe60f684640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe60806e2c0 con 0x7fe6080cb220 2024-05-09T05:14:00.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.492+0000 7fe60f684640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe608160560 con 0x7fe6080c68a0 2024-05-09T05:14:00.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.492+0000 7fe60dc21640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fe6081609f0 0x7fe6080cc780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60342/0 (socket says 172.21.15.1:60342) 2024-05-09T05:14:00.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.492+0000 7fe60dc21640 1 -- 172.21.15.1:0/1873625897 learned_addr learned my addr 172.21.15.1:0/1873625897 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:14:00.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.492+0000 7fe60d420640 1 --2- 172.21.15.1:0/1873625897 >> v2:172.21.15.196:3300/0 conn(0x7fe6080c68a0 0x7fe6081569a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.493+0000 7fe5fffff640 1 -- 172.21.15.1:0/1873625897 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1101299040 0 0) 0x7fe60806e000 con 0x7fe6081609f0 2024-05-09T05:14:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.493+0000 7fe5fffff640 1 -- 172.21.15.1:0/1873625897 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe5f4003520 con 0x7fe6081609f0 2024-05-09T05:14:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.493+0000 7fe5fffff640 1 -- 172.21.15.1:0/1873625897 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3608842711 0 0) 0x7fe60806e2c0 con 0x7fe6080cb220 2024-05-09T05:14:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.493+0000 7fe5fffff640 1 -- 172.21.15.1:0/1873625897 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fe60806e000 con 0x7fe6080cb220 2024-05-09T05:14:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.493+0000 7fe5fffff640 1 -- 172.21.15.1:0/1873625897 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1412427579 0 0) 0x7fe5f4003520 con 0x7fe6081609f0 2024-05-09T05:14:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.493+0000 7fe60d420640 1 -- 172.21.15.1:0/1873625897 >> v1:172.21.15.196:6789/0 conn(0x7fe6080cb220 legacy=0x7fe60815c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.493+0000 7fe60d420640 1 -- 172.21.15.1:0/1873625897 >> v1:172.21.15.1:6789/0 conn(0x7fe6081609f0 legacy=0x7fe6080cc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.493+0000 7fe60d420640 1 -- 172.21.15.1:0/1873625897 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe60815ff10 con 0x7fe6080c68a0 2024-05-09T05:14:00.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.494+0000 7fe60d420640 1 --2- 172.21.15.1:0/1873625897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6080c68a0 0x7fe6081569a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe6040097a0 tx=0x7fe60402a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=333dfe45090986ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.494+0000 7fe5fffff640 1 -- 172.21.15.1:0/1873625897 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604037070 con 0x7fe6080c68a0 2024-05-09T05:14:00.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.494+0000 7fe5fffff640 1 -- 172.21.15.1:0/1873625897 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe60402ac50 con 0x7fe6080c68a0 2024-05-09T05:14:00.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.494+0000 7fe5fffff640 1 -- 172.21.15.1:0/1873625897 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600001a70 con 0x7fe6080c68a0 2024-05-09T05:14:00.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.495+0000 7fe60f684640 1 -- 172.21.15.1:0/1873625897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6080c68a0 msgr2=0x7fe6081569a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.495+0000 7fe60f684640 1 --2- 172.21.15.1:0/1873625897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6080c68a0 0x7fe6081569a0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe6040097a0 tx=0x7fe60402a1c0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.495+0000 7fe60f684640 1 -- 172.21.15.1:0/1873625897 shutdown_connections 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.495+0000 7fe60f684640 1 --2- 172.21.15.1:0/1873625897 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6080c68a0 0x7fe6081569a0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.495+0000 7fe60f684640 1 -- 172.21.15.1:0/1873625897 >> 172.21.15.1:0/1873625897 conn(0x7fe608072e50 msgr2=0x7fe608073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.495+0000 7fe60f684640 1 -- 172.21.15.1:0/1873625897 shutdown_connections 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.495+0000 7fe60f684640 1 -- 172.21.15.1:0/1873625897 wait complete. 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.497+0000 7fe60f684640 1 Processor -- start 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.497+0000 7fe60f684640 1 -- start start 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60f684640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6080c68a0 0x7fe608152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60f684640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6080cb220 0x7fe60814dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60f684640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6081609f0 0x7fe608151cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60f684640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe6080cb870 con 0x7fe6081609f0 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60f684640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe608156ee0 con 0x7fe6080cb220 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60f684640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe60815dfc0 con 0x7fe6080c68a0 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60dc21640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6081609f0 0x7fe608151cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60d420640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6080c68a0 msgr2=0x7fe608152860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60d420640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6080c68a0 0x7fe608152860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60dc21640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6081609f0 0x7fe608151cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39602/0 (socket says 172.21.15.1:39602) 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60dc21640 1 -- 172.21.15.1:0/138367743 learned_addr learned my addr 172.21.15.1:0/138367743 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.498+0000 7fe60cc1f640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6080cb220 0x7fe60814dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe60dc21640 1 -- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6080c68a0 msgr2=0x7fe608152860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe60dc21640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe6080c68a0 0x7fe608152860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe60dc21640 1 -- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6080cb220 msgr2=0x7fe60814dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe60dc21640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe6080cb220 0x7fe60814dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe60dc21640 1 -- 172.21.15.1:0/138367743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe608153380 con 0x7fe6081609f0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe60dc21640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6081609f0 0x7fe608151cc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe60000cb10 tx=0x7fe60000a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600017070 con 0x7fe6081609f0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe60000ac70 con 0x7fe6081609f0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe600012620 con 0x7fe6081609f0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe60815ab00 con 0x7fe6081609f0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.499+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe608152da0 con 0x7fe6081609f0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.500+0000 7fe60f684640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe60815b5e0 0x7fe608152200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.500+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe6081530f0 con 0x7fe60815b5e0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.501+0000 7fe60d420640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe60815b5e0 0x7fe608152200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.501+0000 7fe60d420640 1 -- 172.21.15.1:0/138367743 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe608154520 con 0x7fe60815b5e0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.501+0000 7fe60d420640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe60815b5e0 0x7fe608152200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe6080c6cb0 tx=0x7fe6040046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.501+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe60001a070 con 0x7fe6081609f0 2024-05-09T05:14:00.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe5fdffb640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe5ec076280 0x7fe5ec078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe6000973b0 con 0x7fe6081609f0 2024-05-09T05:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604041020 con 0x7fe60815b5e0 2024-05-09T05:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe604003810 con 0x7fe60815b5e0 2024-05-09T05:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe60cc1f640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe5ec076280 0x7fe5ec078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe60815b5e0 msgr2=0x7fe608152200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe5fdffb640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe60815b5e0 0x7fe608152200 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe6080c6cb0 tx=0x7fe6040046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe60815b5e0 msgr2=0x7fe608152200 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.502+0000 7fe5fdffb640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe60815b5e0 0x7fe608152200 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.503+0000 7fe60cc1f640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe5ec076280 0x7fe5ec078790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe608153710 tx=0x7fe5f8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.509+0000 7fe60f684640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe608153960 0x7fe60815d960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.509+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe60815caf0 con 0x7fe608153960 2024-05-09T05:14:00.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.510+0000 7fe60d420640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe608153960 0x7fe60815d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:00.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.510+0000 7fe60d420640 1 -- 172.21.15.1:0/138367743 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe608050870 con 0x7fe608153960 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.510+0000 7fe60d420640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe608153960 0x7fe60815d960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe6040097a0 tx=0x7fe604009190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.510+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe604041070 con 0x7fe608153960 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7fe60403b430 con 0x7fe608153960 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe608153960 msgr2=0x7fe60815d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe5fdffb640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe608153960 0x7fe60815d960 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fe6040097a0 tx=0x7fe604009190 comp rx=0 tx=0).stop 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe5fdffb640 1 -- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe608153960 msgr2=0x7fe60815d960 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe5fdffb640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe608153960 0x7fe60815d960 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:00.511 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 66, 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 102789, 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:00.512 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:00.513 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:00.514 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:00.515 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:00.516 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:00.518 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:00.519 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:00.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:00.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:00.523 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:00.524 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe5ec076280 msgr2=0x7fe5ec078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60f684640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe5ec076280 0x7fe5ec078790 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fe608153710 tx=0x7fe5f8009040 comp rx=0 tx=0).stop 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6081609f0 msgr2=0x7fe608151cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60f684640 1 --2- 172.21.15.1:0/138367743 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe6081609f0 0x7fe608151cc0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fe60000cb10 tx=0x7fe60000a060 comp rx=0 tx=0).stop 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60dc21640 1 -- 172.21.15.1:0/138367743 reap_dead start 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60dc21640 1 -- 172.21.15.1:0/138367743 reap_dead start 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 shutdown_connections 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 >> 172.21.15.1:0/138367743 conn(0x7fe608072e50 msgr2=0x7fe608155a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 shutdown_connections 2024-05-09T05:14:00.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:00.511+0000 7fe60f684640 1 -- 172.21.15.1:0/138367743 wait complete. 2024-05-09T05:14:00.527 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:14:00.527 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:14:03.379 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:14:03.382 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:14:20.528 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:14:20.528 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-05-09T05:14:20.528 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T05:14:20.528 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T05:14:20.531 INFO:tasks.ceph.mon.c:Started 2024-05-09T05:14:20.531 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:14:20.532 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:14:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.658+0000 7fc0cc1c7640 1 Processor -- start 2024-05-09T05:14:20.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.658+0000 7fc0cc1c7640 1 -- start start 2024-05-09T05:14:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0cc1c7640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc0c40b9f50 0x7fc0c406c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0cc1c7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 0x7fc0c415a630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:20.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0cc1c7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc0c4068cd0 con 0x7fc0c4154760 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0cc1c7640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc0c40c89d0 con 0x7fc0c40b9f50 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0cc1c7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c40c8c70 con 0x7fc0c40688c0 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c9f63640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc0c4154760 0x7fc0c40c76a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60812/0 (socket says 172.21.15.1:60812) 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c9f63640 1 -- 172.21.15.1:0/1869617060 learned_addr learned my addr 172.21.15.1:0/1869617060 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c9762640 1 --2- 172.21.15.1:0/1869617060 >> v2:172.21.15.196:3300/0 conn(0x7fc0c40b9f50 0x7fc0c406c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0ca764640 1 --2- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 0x7fc0c415a630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c8f61640 1 -- 172.21.15.1:0/1869617060 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1055705351 0 0) 0x7fc0c0001110 con 0x7fc0c4154760 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c8f61640 1 -- 172.21.15.1:0/1869617060 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc0b0003520 con 0x7fc0c4154760 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0ca764640 1 -- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 msgr2=0x7fc0c415a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0ca764640 1 -- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 msgr2=0x7fc0c415a630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0ca764640 1 --2- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 0x7fc0c415a630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0ca764640 1 --2- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 0x7fc0c415a630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c9762640 1 -- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 msgr2=0x7fc0c415a630 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:20.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c9762640 1 --2- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 0x7fc0c415a630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c9762640 1 -- 172.21.15.1:0/1869617060 >> v1:172.21.15.1:6789/0 conn(0x7fc0c4154760 legacy=0x7fc0c40c76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c8f61640 1 -- 172.21.15.1:0/1869617060 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 713572984 0 0) 0x7fc0b0003520 con 0x7fc0c4154760 2024-05-09T05:14:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.660+0000 7fc0c9762640 1 -- 172.21.15.1:0/1869617060 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c40c93f0 con 0x7fc0c40b9f50 2024-05-09T05:14:20.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.661+0000 7fc0c9762640 1 --2- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 0x7fc0c406c6f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40097a0 tx=0x7fc0b402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b807d23932d8fb15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.661+0000 7fc0c8f61640 1 -- 172.21.15.1:0/1869617060 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4037070 con 0x7fc0c40b9f50 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.661+0000 7fc0c8f61640 1 -- 172.21.15.1:0/1869617060 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b402aba0 con 0x7fc0c40b9f50 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.661+0000 7fc0c8f61640 1 -- 172.21.15.1:0/1869617060 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c00018c0 con 0x7fc0c40b9f50 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.662+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 msgr2=0x7fc0c406c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.662+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 0x7fc0c406c6f0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40097a0 tx=0x7fc0b402a110 comp rx=0 tx=0).stop 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.662+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1869617060 shutdown_connections 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.662+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c40688c0 0x7fc0c415a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.662+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1869617060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 0x7fc0c406c6f0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.662+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1869617060 >> 172.21.15.1:0/1869617060 conn(0x7fc0c40caec0 msgr2=0x7fc0c40cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.663+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1869617060 shutdown_connections 2024-05-09T05:14:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.663+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1869617060 wait complete. 2024-05-09T05:14:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.665+0000 7fc0cc1c7640 1 Processor -- start 2024-05-09T05:14:20.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.665+0000 7fc0cc1c7640 1 -- start start 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.666+0000 7fc0cc1c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c40688c0 0x7fc0c414a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.666+0000 7fc0cc1c7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 0x7fc0c4149340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.666+0000 7fc0cc1c7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 0x7fc0c4159c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.666+0000 7fc0cc1c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c40c9c90 con 0x7fc0c40688c0 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.666+0000 7fc0cc1c7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc0c40c99b0 con 0x7fc0c40b9f50 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.666+0000 7fc0cc1c7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc0c40c8f50 con 0x7fc0c4154760 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.666+0000 7fc0c9f63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c40688c0 0x7fc0c414a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9f63640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c40688c0 0x7fc0c414a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35862/0 (socket says 172.21.15.1:35862) 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9f63640 1 -- 172.21.15.1:0/1435702208 learned_addr learned my addr 172.21.15.1:0/1435702208 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9762640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 0x7fc0c4149340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0ca764640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 0x7fc0c4159c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0ca764640 1 -- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 msgr2=0x7fc0c4159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:14:20.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0ca764640 1 -- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 msgr2=0x7fc0c4159c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0ca764640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 0x7fc0c4159c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0ca764640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 0x7fc0c4159c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9762640 1 -- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 msgr2=0x7fc0c4159c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9762640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 0x7fc0c4159c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9762640 1 -- 172.21.15.1:0/1435702208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c40688c0 msgr2=0x7fc0c414a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9762640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c40688c0 0x7fc0c414a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9762640 1 -- 172.21.15.1:0/1435702208 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0c4149880 con 0x7fc0c40b9f50 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9f63640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c40688c0 0x7fc0c414a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0c9762640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 0x7fc0c4149340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40c8360 tx=0x7fc0b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0ba7fc640 1 -- 172.21.15.1:0/1435702208 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4041070 con 0x7fc0c40b9f50 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0ba7fc640 1 -- 172.21.15.1:0/1435702208 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc0b403c890 con 0x7fc0c40b9f50 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0c41595f0 con 0x7fc0c40b9f50 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0ba7fc640 1 -- 172.21.15.1:0/1435702208 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b4037040 con 0x7fc0c40b9f50 2024-05-09T05:14:20.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.667+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0c41f32c0 con 0x7fc0c40b9f50 2024-05-09T05:14:20.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.668+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc0c4152a20 con 0x7fc0c40b9f50 2024-05-09T05:14:20.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.669+0000 7fc0ba7fc640 1 -- 172.21.15.1:0/1435702208 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc0b4044070 con 0x7fc0c40b9f50 2024-05-09T05:14:20.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.670+0000 7fc0ba7fc640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0a80762d0 0x7fc0a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.670+0000 7fc0ba7fc640 1 -- 172.21.15.1:0/1435702208 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc0b40b7d80 con 0x7fc0c40b9f50 2024-05-09T05:14:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.670+0000 7fc0c9f63640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0a80762d0 0x7fc0a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:20.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.670+0000 7fc0c9f63640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0a80762d0 0x7fc0a80787e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0001a30 tx=0x7fc0c0001010 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:20.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.675+0000 7fc0ba7fc640 1 -- 172.21.15.1:0/1435702208 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc0c4152a20 con 0x7fc0c40b9f50 2024-05-09T05:14:20.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.978+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc0c4050870 con 0x7fc0c40b9f50 2024-05-09T05:14:20.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.979+0000 7fc0ba7fc640 1 -- 172.21.15.1:0/1435702208 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fc0b4081010 con 0x7fc0c40b9f50 2024-05-09T05:14:20.978 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0a80762d0 msgr2=0x7fc0a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0a80762d0 0x7fc0a80787e0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7fc0c0001a30 tx=0x7fc0c0001010 comp rx=0 tx=0).stop 2024-05-09T05:14:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 msgr2=0x7fc0c4149340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:20.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 0x7fc0c4149340 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc0c40c8360 tx=0x7fc0b40046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 shutdown_connections 2024-05-09T05:14:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0a80762d0 0x7fc0a80787e0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc0c40b9f50 0x7fc0c4149340 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc0c4154760 0x7fc0c4159c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 --2- 172.21.15.1:0/1435702208 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc0c40688c0 0x7fc0c414a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 >> 172.21.15.1:0/1435702208 conn(0x7fc0c40caec0 msgr2=0x7fc0c414f880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.981+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 shutdown_connections 2024-05-09T05:14:20.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:20.982+0000 7fc0cc1c7640 1 -- 172.21.15.1:0/1435702208 wait complete. 2024-05-09T05:14:23.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:14:24.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.119+0000 7f927ab86640 1 Processor -- start 2024-05-09T05:14:24.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.119+0000 7f927ab86640 1 -- start start 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f927ab86640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f927406efa0 0x7f9274156900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f927ab86640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f9274160500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f927ab86640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f927406de20 con 0x7f92740cc560 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f927ab86640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f927415c9a0 con 0x7f927406efa0 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f927ab86640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f92740b51a0 con 0x7f92740cc970 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f9278922640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f927406efa0 0x7f9274156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f9278922640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f927406efa0 0x7f9274156900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35878/0 (socket says 172.21.15.1:35878) 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f9278922640 1 -- 172.21.15.1:0/4128897584 learned_addr learned my addr 172.21.15.1:0/4128897584 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f9279123640 1 --2- 172.21.15.1:0/4128897584 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f9274160500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f9278922640 1 -- 172.21.15.1:0/4128897584 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 msgr2=0x7f9274160500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f9278922640 1 --2- 172.21.15.1:0/4128897584 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f9274160500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f9278922640 1 -- 172.21.15.1:0/4128897584 >> v1:172.21.15.1:6789/0 conn(0x7f92740cc560 legacy=0x7f927415fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:24.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.121+0000 7f9278922640 1 -- 172.21.15.1:0/4128897584 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f927415d720 con 0x7f927406efa0 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.122+0000 7f9278922640 1 --2- 172.21.15.1:0/4128897584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f9274156900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9268009870 tx=0x7f926802a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bd93ffbb88ad110 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.122+0000 7f92737fe640 1 -- 172.21.15.1:0/4128897584 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9268037070 con 0x7f927406efa0 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.122+0000 7f92737fe640 1 -- 172.21.15.1:0/4128897584 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f926802ae50 con 0x7f927406efa0 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.122+0000 7f92737fe640 1 -- 172.21.15.1:0/4128897584 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9268032860 con 0x7f927406efa0 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.123+0000 7f927ab86640 1 -- 172.21.15.1:0/4128897584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 msgr2=0x7f9274156900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.123+0000 7f927ab86640 1 --2- 172.21.15.1:0/4128897584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f9274156900 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9268009870 tx=0x7f926802a300 comp rx=0 tx=0).stop 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.123+0000 7f927ab86640 1 -- 172.21.15.1:0/4128897584 shutdown_connections 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.123+0000 7f927ab86640 1 --2- 172.21.15.1:0/4128897584 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f9274160500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.123+0000 7f927ab86640 1 --2- 172.21.15.1:0/4128897584 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f9274156900 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.123+0000 7f927ab86640 1 -- 172.21.15.1:0/4128897584 >> 172.21.15.1:0/4128897584 conn(0x7f9274068d90 msgr2=0x7f92740d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:24.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.123+0000 7f927ab86640 1 -- 172.21.15.1:0/4128897584 shutdown_connections 2024-05-09T05:14:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.123+0000 7f927ab86640 1 -- 172.21.15.1:0/4128897584 wait complete. 2024-05-09T05:14:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.125+0000 7f927ab86640 1 Processor -- start 2024-05-09T05:14:24.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.125+0000 7f927ab86640 1 -- start start 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.126+0000 7f927ab86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f927414f8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.126+0000 7f927ab86640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92740cc560 0x7f9274150b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.126+0000 7f927ab86640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f92740d1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.126+0000 7f927ab86640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f927415cf20 con 0x7f927406efa0 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.126+0000 7f927ab86640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f927415cc40 con 0x7f92740cc560 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.126+0000 7f927ab86640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f927415d200 con 0x7f92740cc970 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.126+0000 7f9278922640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f927414f8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9278922640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f927414f8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35892/0 (socket says 172.21.15.1:35892) 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9278922640 1 -- 172.21.15.1:0/4202530419 learned_addr learned my addr 172.21.15.1:0/4202530419 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9273fff640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92740cc560 0x7f9274150b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:24.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9279123640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f92740d1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9279123640 1 -- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 msgr2=0x7f92740d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9279123640 1 -- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 msgr2=0x7f92740d1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9279123640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f92740d1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9279123640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f92740d1130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9278922640 1 -- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 msgr2=0x7f92740d1130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9278922640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f92740d1130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9278922640 1 -- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92740cc560 msgr2=0x7f9274150b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9278922640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92740cc560 0x7f9274150b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9278922640 1 -- 172.21.15.1:0/4202530419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92740d1cc0 con 0x7f927406efa0 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f9278922640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f927414f8a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9268009870 tx=0x7f92680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.127+0000 7f92717fa640 1 -- 172.21.15.1:0/4202530419 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9268041070 con 0x7f927406efa0 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.128+0000 7f92717fa640 1 -- 172.21.15.1:0/4202530419 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f926803c670 con 0x7f927406efa0 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.128+0000 7f92717fa640 1 -- 172.21.15.1:0/4202530419 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9268037040 con 0x7f927406efa0 2024-05-09T05:14:24.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.128+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92740d1f30 con 0x7f927406efa0 2024-05-09T05:14:24.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.128+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92741508a0 con 0x7f927406efa0 2024-05-09T05:14:24.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.129+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9274150050 con 0x7f927406efa0 2024-05-09T05:14:24.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.130+0000 7f92717fa640 1 -- 172.21.15.1:0/4202530419 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9268044070 con 0x7f927406efa0 2024-05-09T05:14:24.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.131+0000 7f92717fa640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f92580762d0 0x7f92580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:24.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.131+0000 7f92717fa640 1 -- 172.21.15.1:0/4202530419 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f92680b83e0 con 0x7f927406efa0 2024-05-09T05:14:24.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.131+0000 7f9273fff640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f92580762d0 0x7f92580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:24.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.131+0000 7f9273fff640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f92580762d0 0x7f92580787e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f92741f0c20 tx=0x7f9260009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:24.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.135+0000 7f92717fa640 1 -- 172.21.15.1:0/4202530419 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9268081750 con 0x7f927406efa0 2024-05-09T05:14:24.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.502+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9274050870 con 0x7f927406efa0 2024-05-09T05:14:24.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.502+0000 7f92717fa640 1 -- 172.21.15.1:0/4202530419 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f9268085a50 con 0x7f927406efa0 2024-05-09T05:14:24.502 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":66,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:24.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f92580762d0 msgr2=0x7f92580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:24.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f92580762d0 0x7f92580787e0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f92741f0c20 tx=0x7f9260009040 comp rx=0 tx=0).stop 2024-05-09T05:14:24.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 msgr2=0x7f927414f8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:24.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f927414f8a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f9268009870 tx=0x7f92680046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 shutdown_connections 2024-05-09T05:14:24.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f92580762d0 0x7f92580787e0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f92740cc560 0x7f9274150b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f92740cc970 0x7f92740d1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 --2- 172.21.15.1:0/4202530419 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f927406efa0 0x7f927414f8a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:24.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 >> 172.21.15.1:0/4202530419 conn(0x7f9274068d90 msgr2=0x7f927414b8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:24.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 shutdown_connections 2024-05-09T05:14:24.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:24.505+0000 7f927ab86640 1 -- 172.21.15.1:0/4202530419 wait complete. 2024-05-09T05:14:27.519 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:14:27.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.641+0000 7f5ea927e640 1 Processor -- start 2024-05-09T05:14:27.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.641+0000 7f5ea927e640 1 -- start start 2024-05-09T05:14:27.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.643+0000 7f5ea927e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5ea40cf990 0x7f5ea4156b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:27.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.643+0000 7f5ea927e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ea406f5c0 con 0x7f5ea40d0510 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.643+0000 7f5ea927e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5ea40cc6b0 con 0x7f5ea40cec10 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.643+0000 7f5ea927e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f5ea40ccfe0 con 0x7f5ea40cf990 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea27fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5ea40d0510 0x7f5ea40ce500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60834/0 (socket says 172.21.15.1:60834) 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea27fc640 1 -- 172.21.15.1:0/4164406676 learned_addr learned my addr 172.21.15.1:0/4164406676 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea2ffd640 1 --2- 172.21.15.1:0/4164406676 >> v2:172.21.15.196:3300/0 conn(0x7f5ea40cf990 0x7f5ea4156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1862734029 0 0) 0x7f5ea406f5c0 con 0x7f5ea40d0510 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5e8c003520 con 0x7f5ea40d0510 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4240568790 0 0) 0x7f5e8c003520 con 0x7f5ea40d0510 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 >> v2:172.21.15.196:3300/0 conn(0x7f5ea40cf990 msgr2=0x7f5ea4156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 --2- 172.21.15.1:0/4164406676 >> v2:172.21.15.196:3300/0 conn(0x7f5ea40cf990 0x7f5ea4156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 >> v1:172.21.15.196:6789/0 conn(0x7f5ea40cec10 legacy=0x7f5ea415fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea40cc990 con 0x7f5ea40d0510 2024-05-09T05:14:27.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f5e980016f0 con 0x7f5ea40d0510 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 >> v1:172.21.15.1:6789/0 conn(0x7f5ea40d0510 legacy=0x7f5ea40ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c003a50 0x7f5e8c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e8c004800 0x7f5e8c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e8c005250 0x7f5e8c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea40ccfe0 con 0x7f5e8c004800 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5e8c00c1e0 con 0x7f5e8c005250 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5e8c00c3e0 con 0x7f5e8c003a50 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.644+0000 7f5ea27fc640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e8c004800 0x7f5e8c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.645+0000 7f5ea2ffd640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c003a50 0x7f5e8c003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.645+0000 7f5ea37fe640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e8c005250 0x7f5e8c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.645+0000 7f5ea27fc640 1 -- 172.21.15.1:0/4164406676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c003a50 msgr2=0x7f5e8c003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.645+0000 7f5ea27fc640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c003a50 0x7f5e8c003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.645+0000 7f5ea27fc640 1 -- 172.21.15.1:0/4164406676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e8c005250 msgr2=0x7f5e8c00b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:27.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.645+0000 7f5ea27fc640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5e8c005250 0x7f5e8c00b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.645+0000 7f5ea27fc640 1 -- 172.21.15.1:0/4164406676 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e8c003520 con 0x7f5e8c004800 2024-05-09T05:14:27.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.645+0000 7f5ea37fe640 1 -- 172.21.15.1:0/4164406676 reap_dead start 2024-05-09T05:14:27.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.646+0000 7f5ea27fc640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e8c004800 0x7f5e8c004c10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e98001050 tx=0x7f5e8c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9df077a2b4bd6bab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:27.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.646+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e98007d10 con 0x7f5e8c004800 2024-05-09T05:14:27.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.646+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e98005ad0 con 0x7f5e8c004800 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.646+0000 7f5ea1ffb640 1 -- 172.21.15.1:0/4164406676 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e98007070 con 0x7f5e8c004800 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.647+0000 7f5ea927e640 1 -- 172.21.15.1:0/4164406676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e8c004800 msgr2=0x7f5e8c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.647+0000 7f5ea927e640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e8c004800 0x7f5e8c004c10 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f5e98001050 tx=0x7f5e8c005830 comp rx=0 tx=0).stop 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.647+0000 7f5ea927e640 1 -- 172.21.15.1:0/4164406676 shutdown_connections 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.647+0000 7f5ea927e640 1 --2- 172.21.15.1:0/4164406676 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5e8c004800 0x7f5e8c004c10 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.647+0000 7f5ea927e640 1 -- 172.21.15.1:0/4164406676 >> 172.21.15.1:0/4164406676 conn(0x7f5ea40b36c0 msgr2=0x7f5ea40b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.647+0000 7f5ea927e640 1 -- 172.21.15.1:0/4164406676 shutdown_connections 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.647+0000 7f5ea927e640 1 -- 172.21.15.1:0/4164406676 wait complete. 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.649+0000 7f5ea927e640 1 Processor -- start 2024-05-09T05:14:27.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.649+0000 7f5ea927e640 1 -- start start 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea927e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c004800 0x7f5ea4157070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea927e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ea40b4930 0x7f5ea415ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea927e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ea4154cd0 0x7f5ea41550e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea927e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea406f1f0 con 0x7f5ea40b4930 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea927e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5ea40d0920 con 0x7f5ea4154cd0 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea927e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5ea406f370 con 0x7f5e8c004800 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ea40b4930 0x7f5ea415ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ea40b4930 0x7f5ea415ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35910/0 (socket says 172.21.15.1:35910) 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.650+0000 7f5ea27fc640 1 -- 172.21.15.1:0/678547060 learned_addr learned my addr 172.21.15.1:0/678547060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea37fe640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ea4154cd0 0x7f5ea41550e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea2ffd640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c004800 0x7f5ea4157070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea27fc640 1 -- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c004800 msgr2=0x7f5ea4157070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:27.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea27fc640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c004800 0x7f5ea4157070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea27fc640 1 -- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ea4154cd0 msgr2=0x7f5ea41550e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea27fc640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ea4154cd0 0x7f5ea41550e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea27fc640 1 -- 172.21.15.1:0/678547060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5ea41575b0 con 0x7f5ea40b4930 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea37fe640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ea4154cd0 0x7f5ea41550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea27fc640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ea40b4930 0x7f5ea415ca80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5e98000c00 tx=0x7f5ea40ccf50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5e8b7fe640 1 -- 172.21.15.1:0/678547060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e98040030 con 0x7f5ea40b4930 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5e8b7fe640 1 -- 172.21.15.1:0/678547060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5e980014b0 con 0x7f5ea40b4930 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5ea41f3160 con 0x7f5ea40b4930 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5e8b7fe640 1 -- 172.21.15.1:0/678547060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e98037040 con 0x7f5ea40b4930 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.651+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5ea41f3420 con 0x7f5ea40b4930 2024-05-09T05:14:27.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.652+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5ea4050870 con 0x7f5ea40b4930 2024-05-09T05:14:27.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.653+0000 7f5e8b7fe640 1 -- 172.21.15.1:0/678547060 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5e98043070 con 0x7f5ea40b4930 2024-05-09T05:14:27.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.654+0000 7f5e8b7fe640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5e900762d0 0x7f5e900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:27.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.654+0000 7f5e8b7fe640 1 -- 172.21.15.1:0/678547060 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5e980b6f00 con 0x7f5ea40b4930 2024-05-09T05:14:27.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.654+0000 7f5ea2ffd640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5e900762d0 0x7f5e900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:27.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.655+0000 7f5ea2ffd640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5e900762d0 0x7f5e900787e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009b80 tx=0x7f5e94008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:27.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.659+0000 7f5e8b7fe640 1 -- 172.21.15.1:0/678547060 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5e9804a070 con 0x7f5ea40b4930 2024-05-09T05:14:27.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.961+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5ea406df60 con 0x7f5ea40b4930 2024-05-09T05:14:27.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.962+0000 7f5e8b7fe640 1 -- 172.21.15.1:0/678547060 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f5e98080870 con 0x7f5ea40b4930 2024-05-09T05:14:27.961 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":1,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:27.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5e900762d0 msgr2=0x7f5e900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:27.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5e900762d0 0x7f5e900787e0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f5e94009b80 tx=0x7f5e94008040 comp rx=0 tx=0).stop 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ea40b4930 msgr2=0x7f5ea415ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ea40b4930 0x7f5ea415ca80 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5e98000c00 tx=0x7f5ea40ccf50 comp rx=0 tx=0).stop 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 shutdown_connections 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5e900762d0 0x7f5e900787e0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5ea4154cd0 0x7f5ea41550e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5ea40b4930 0x7f5ea415ca80 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 --2- 172.21.15.1:0/678547060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5e8c004800 0x7f5ea4157070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.964+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 >> 172.21.15.1:0/678547060 conn(0x7f5ea40b36c0 msgr2=0x7f5ea41561f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.965+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 shutdown_connections 2024-05-09T05:14:27.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:27.965+0000 7f5ea927e640 1 -- 172.21.15.1:0/678547060 wait complete. 2024-05-09T05:14:27.977 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:14:27.977 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:14:27.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:14:28.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.101+0000 7f2fd2605640 1 Processor -- start 2024-05-09T05:14:28.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.101+0000 7f2fd2605640 1 -- start start 2024-05-09T05:14:28.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.102+0000 7f2fd2605640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fcc0736b0 0x7f2fcc15d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.102+0000 7f2fd2605640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 0x7f2fcc06d1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.102+0000 7f2fd2605640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2fcc0cc1b0 con 0x7f2fcc06dde0 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.102+0000 7f2fd2605640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f2fcc06d710 con 0x7f2fcc0736b0 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.102+0000 7f2fd2605640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2fcc0cbf60 con 0x7f2fcc0c5600 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.102+0000 7f2fc3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fcc0736b0 0x7f2fcc15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.102+0000 7f2fc3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f2fcc0736b0 0x7f2fcc15d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35914/0 (socket says 172.21.15.1:35914) 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.102+0000 7f2fc3fff640 1 -- 172.21.15.1:0/2619073885 learned_addr learned my addr 172.21.15.1:0/2619073885 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fd0ba2640 1 --2- 172.21.15.1:0/2619073885 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 0x7f2fcc06d1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fd0ba2640 1 -- 172.21.15.1:0/2619073885 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 msgr2=0x7f2fcc06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fd0ba2640 1 -- 172.21.15.1:0/2619073885 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 msgr2=0x7f2fcc06d1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fd0ba2640 1 --2- 172.21.15.1:0/2619073885 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 0x7f2fcc06d1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fd0ba2640 1 --2- 172.21.15.1:0/2619073885 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 0x7f2fcc06d1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fc3fff640 1 -- 172.21.15.1:0/2619073885 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 msgr2=0x7f2fcc06d1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fc3fff640 1 --2- 172.21.15.1:0/2619073885 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 0x7f2fcc06d1d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fc3fff640 1 -- 172.21.15.1:0/2619073885 >> v1:172.21.15.196:6789/0 conn(0x7f2fcc06dde0 legacy=0x7f2fcc0c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/2619073885 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2029827452 0 0) 0x7f2fcc0cc1b0 con 0x7f2fcc06dde0 2024-05-09T05:14:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.103+0000 7f2fc3fff640 1 -- 172.21.15.1:0/2619073885 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fcc0cc490 con 0x7f2fcc0736b0 2024-05-09T05:14:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.104+0000 7f2fc3fff640 1 --2- 172.21.15.1:0/2619073885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc15d330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009870 tx=0x7f2fb402a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3d0074f4de23f0e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.104+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/2619073885 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4037070 con 0x7f2fcc0736b0 2024-05-09T05:14:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.104+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/2619073885 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb4032440 con 0x7f2fcc0736b0 2024-05-09T05:14:28.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.104+0000 7f2fcb7fe640 1 -- 172.21.15.1:0/2619073885 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4032810 con 0x7f2fcc0736b0 2024-05-09T05:14:28.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.105+0000 7f2fd2605640 1 -- 172.21.15.1:0/2619073885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 msgr2=0x7f2fcc15d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.105+0000 7f2fd2605640 1 --2- 172.21.15.1:0/2619073885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc15d330 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009870 tx=0x7f2fb402a240 comp rx=0 tx=0).stop 2024-05-09T05:14:28.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.105+0000 7f2fd2605640 1 -- 172.21.15.1:0/2619073885 shutdown_connections 2024-05-09T05:14:28.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.105+0000 7f2fd2605640 1 --2- 172.21.15.1:0/2619073885 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc0c5600 0x7f2fcc06d1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.105+0000 7f2fd2605640 1 --2- 172.21.15.1:0/2619073885 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc15d330 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.105+0000 7f2fd2605640 1 -- 172.21.15.1:0/2619073885 >> 172.21.15.1:0/2619073885 conn(0x7f2fcc0ce180 msgr2=0x7f2fcc0ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:28.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.105+0000 7f2fd2605640 1 -- 172.21.15.1:0/2619073885 shutdown_connections 2024-05-09T05:14:28.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.105+0000 7f2fd2605640 1 -- 172.21.15.1:0/2619073885 wait complete. 2024-05-09T05:14:28.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.107+0000 7f2fd2605640 1 Processor -- start 2024-05-09T05:14:28.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.107+0000 7f2fd2605640 1 -- start start 2024-05-09T05:14:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fd2605640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc06dde0 0x7f2fcc1f2520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fd2605640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc1601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fd2605640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fcc0c5600 0x7f2fcc14f9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fd2605640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc06ccb0 con 0x7f2fcc0736b0 2024-05-09T05:14:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fd2605640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc06c9d0 con 0x7f2fcc0c5600 2024-05-09T05:14:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fd2605640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2fcc0c77c0 con 0x7f2fcc06dde0 2024-05-09T05:14:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fc3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc1601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fc3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc1601f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35930/0 (socket says 172.21.15.1:35930) 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fc3fff640 1 -- 172.21.15.1:0/3089767585 learned_addr learned my addr 172.21.15.1:0/3089767585 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fd0ba2640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fcc0c5600 0x7f2fcc14f9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.108+0000 7f2fcbfff640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc06dde0 0x7f2fcc1f2520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc3fff640 1 -- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc06dde0 msgr2=0x7f2fcc1f2520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc3fff640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2fcc06dde0 0x7f2fcc1f2520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc3fff640 1 -- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fcc0c5600 msgr2=0x7f2fcc14f9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc3fff640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fcc0c5600 0x7f2fcc14f9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc3fff640 1 -- 172.21.15.1:0/3089767585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fcc160730 con 0x7f2fcc0736b0 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fd0ba2640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2fcc0c5600 0x7f2fcc14f9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc3fff640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc1601f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009870 tx=0x7f2fb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4041020 con 0x7f2fcc0736b0 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2fb40306c0 con 0x7f2fcc0736b0 2024-05-09T05:14:28.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb4037040 con 0x7f2fcc0736b0 2024-05-09T05:14:28.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fcc1560d0 con 0x7f2fcc0736b0 2024-05-09T05:14:28.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.109+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fcc15f430 con 0x7f2fcc0736b0 2024-05-09T05:14:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.111+0000 7f2fd2605640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc15f670 0x7f2fcc15fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.111+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc1e7f50 con 0x7f2fcc15f670 2024-05-09T05:14:28.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.111+0000 7f2fcbfff640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc15f670 0x7f2fcc15fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.111+0000 7f2fcbfff640 1 -- 172.21.15.1:0/3089767585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2fcc050870 con 0x7f2fcc15f670 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.111+0000 7f2fcbfff640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc15f670 0x7f2fcc15fa80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000f60 tx=0x7f2fbc00e8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.111+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fbc019070 con 0x7f2fcc15f670 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.112+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2fbc0052e0 con 0x7f2fcc15f670 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.112+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc15f670 msgr2=0x7f2fcc15fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.112+0000 7f2fc97fa640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc15f670 0x7f2fcc15fa80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc000f60 tx=0x7f2fbc00e8b0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.112+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc15f670 msgr2=0x7f2fcc15fa80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.112+0000 7f2fc97fa640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc15f670 0x7f2fcc15fa80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.112+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2fb4044020 con 0x7f2fcc0736b0 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.113+0000 7f2fc97fa640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fac075cb0 0x7f2fac0781c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.113+0000 7f2fcbfff640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fac075cb0 0x7f2fac0781c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.113+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2fb40b7e30 con 0x7f2fcc0736b0 2024-05-09T05:14:28.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.114+0000 7f2fcbfff640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fac075cb0 0x7f2fac0781c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00ee00 tx=0x7f2fbc005000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.122+0000 7f2fd2605640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc157170 0x7f2fcc157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.122+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2fcc0c7e30 con 0x7f2fcc157170 2024-05-09T05:14:28.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.122+0000 7f2fd0ba2640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc157170 0x7f2fcc157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.122+0000 7f2fd0ba2640 1 -- 172.21.15.1:0/3089767585 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2fcc0c8580 con 0x7f2fcc157170 2024-05-09T05:14:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.122+0000 7f2fd0ba2640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc157170 0x7f2fcc157580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4007df0 tx=0x7f2fc400f520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fc4016420 con 0x7f2fcc157170 2024-05-09T05:14:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f2fc4016420 con 0x7f2fcc157170 2024-05-09T05:14:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc157170 msgr2=0x7f2fcc157580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fc97fa640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc157170 0x7f2fcc157580 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f2fc4007df0 tx=0x7f2fc400f520 comp rx=0 tx=0).stop 2024-05-09T05:14:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fc97fa640 1 -- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc157170 msgr2=0x7f2fcc157580 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:28.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fc97fa640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc157170 0x7f2fcc157580 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.123 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:28.123 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:28.123 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 68, 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 70318, 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:28.124 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:28.125 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:28.126 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:28.127 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:28.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.129 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:28.130 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:28.131 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.132 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.133 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.134 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:28.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:28.136 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:28.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fac075cb0 msgr2=0x7f2fac0781c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd2605640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2fac075cb0 0x7f2fac0781c0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f2fbc00ee00 tx=0x7f2fbc005000 comp rx=0 tx=0).stop 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 msgr2=0x7f2fcc1601f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd2605640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc1601f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2fb4009870 tx=0x7f2fb40046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd0ba2640 1 -- 172.21.15.1:0/3089767585 reap_dead start 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 shutdown_connections 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd2605640 1 --2- 172.21.15.1:0/3089767585 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2fcc0736b0 0x7f2fcc1601f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 >> 172.21.15.1:0/3089767585 conn(0x7f2fcc0ce180 msgr2=0x7f2fcc1541b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.123+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 shutdown_connections 2024-05-09T05:14:28.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.124+0000 7f2fd2605640 1 -- 172.21.15.1:0/3089767585 wait complete. 2024-05-09T05:14:28.138 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:14:28.139 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:14:28.139 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:14:28.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.260+0000 7fdf5df01640 1 Processor -- start 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.260+0000 7fdf5df01640 1 -- start start 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.261+0000 7fdf5df01640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fdf5816a010 0x7fdf580c8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.261+0000 7fdf5df01640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf580c8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.261+0000 7fdf5df01640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdf58073a80 con 0x7fdf5808d6a0 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.261+0000 7fdf5df01640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fdf5808dab0 con 0x7fdf5816a010 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.261+0000 7fdf5df01640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdf58074670 con 0x7fdf580c8840 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.262+0000 7fdf56ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fdf5816a010 0x7fdf580c8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.262+0000 7fdf57fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf580c8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.262+0000 7fdf577fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fdf5808d6a0 0x7fdf580c6940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:51760/0 (socket says 172.21.15.1:51760) 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.262+0000 7fdf577fe640 1 -- 172.21.15.1:0/486558710 learned_addr learned my addr 172.21.15.1:0/486558710 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:14:28.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.262+0000 7fdf57fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf580c8c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60196/0 (socket says 172.21.15.1:60196) 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.262+0000 7fdf56ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fdf5816a010 0x7fdf580c8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:55660/0 (socket says 172.21.15.1:55660) 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.262+0000 7fdf567fc640 1 -- 172.21.15.1:0/486558710 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1920841258 0 0) 0x7fdf58073a80 con 0x7fdf5808d6a0 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.338+0000 7fdf567fc640 1 -- 172.21.15.1:0/486558710 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fdf40003520 con 0x7fdf5808d6a0 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.338+0000 7fdf57fff640 1 -- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 msgr2=0x7fdf580c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.338+0000 7fdf57fff640 1 -- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 msgr2=0x7fdf580c8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.338+0000 7fdf57fff640 1 --2- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf580c8c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.338+0000 7fdf57fff640 1 --2- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf580c8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.338+0000 7fdf56ffd640 1 -- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 msgr2=0x7fdf580c8c50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.339+0000 7fdf56ffd640 1 --2- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf580c8c50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.339+0000 7fdf567fc640 1 -- 172.21.15.1:0/486558710 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1131873888 0 0) 0x7fdf40003520 con 0x7fdf5808d6a0 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.339+0000 7fdf56ffd640 1 -- 172.21.15.1:0/486558710 >> v1:172.21.15.196:6789/0 conn(0x7fdf5808d6a0 legacy=0x7fdf580c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.339+0000 7fdf56ffd640 1 -- 172.21.15.1:0/486558710 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf580c5240 con 0x7fdf5816a010 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.340+0000 7fdf56ffd640 1 --2- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816a010 0x7fdf580c8300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00dbc0 tx=0x7fdf4c02e410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=328dcb1a2bebbaab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.340+0000 7fdf567fc640 1 -- 172.21.15.1:0/486558710 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48001710 con 0x7fdf5816a010 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.340+0000 7fdf567fc640 1 -- 172.21.15.1:0/486558710 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf4c00d0d0 con 0x7fdf5816a010 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.340+0000 7fdf567fc640 1 -- 172.21.15.1:0/486558710 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4c00b670 con 0x7fdf5816a010 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.341+0000 7fdf5df01640 1 -- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816a010 msgr2=0x7fdf580c8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.341+0000 7fdf5df01640 1 --2- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816a010 0x7fdf580c8300 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c00dbc0 tx=0x7fdf4c02e410 comp rx=0 tx=0).stop 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.341+0000 7fdf5df01640 1 -- 172.21.15.1:0/486558710 shutdown_connections 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.341+0000 7fdf5df01640 1 --2- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf580c8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.341+0000 7fdf5df01640 1 --2- 172.21.15.1:0/486558710 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816a010 0x7fdf580c8300 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.341+0000 7fdf5df01640 1 -- 172.21.15.1:0/486558710 >> 172.21.15.1:0/486558710 conn(0x7fdf58070f40 msgr2=0x7fdf5806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.341+0000 7fdf5df01640 1 -- 172.21.15.1:0/486558710 shutdown_connections 2024-05-09T05:14:28.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.341+0000 7fdf5df01640 1 -- 172.21.15.1:0/486558710 wait complete. 2024-05-09T05:14:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.343+0000 7fdf5df01640 1 Processor -- start 2024-05-09T05:14:28.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.343+0000 7fdf5df01640 1 -- start start 2024-05-09T05:14:28.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf5df01640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5808d6a0 0x7fdf581ffd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf5df01640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf58203450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf5df01640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5816a010 0x7fdf581612c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf5df01640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdf58074300 con 0x7fdf5816a010 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf5df01640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdf58074020 con 0x7fdf5808d6a0 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf5df01640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdf5816a420 con 0x7fdf580c8840 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf57fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5816a010 0x7fdf581612c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf57fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5816a010 0x7fdf581612c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35968/0 (socket says 172.21.15.1:35968) 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.344+0000 7fdf57fff640 1 -- 172.21.15.1:0/173397321 learned_addr learned my addr 172.21.15.1:0/173397321 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf56ffd640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf58203450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf577fe640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5808d6a0 0x7fdf581ffd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf56ffd640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 msgr2=0x7fdf58203450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf56ffd640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 msgr2=0x7fdf58203450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:14:28.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf56ffd640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf58203450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf56ffd640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf58203450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf57fff640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 msgr2=0x7fdf58203450 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf57fff640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdf580c8840 0x7fdf58203450 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf57fff640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5808d6a0 msgr2=0x7fdf581ffd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf57fff640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5808d6a0 0x7fdf581ffd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf57fff640 1 -- 172.21.15.1:0/173397321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdf582002d0 con 0x7fdf5816a010 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf57fff640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5816a010 0x7fdf581612c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf50004630 tx=0x7fdf5000a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50019070 con 0x7fdf5816a010 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdf5000ae20 con 0x7fdf5816a010 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdf58203990 con 0x7fdf5816a010 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf50014690 con 0x7fdf5816a010 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.345+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdf581646f0 con 0x7fdf5816a010 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.346+0000 7fdf5df01640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf58169690 0x7fdf581ff540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.346+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdf58163ee0 con 0x7fdf58169690 2024-05-09T05:14:28.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.347+0000 7fdf577fe640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf58169690 0x7fdf581ff540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.347+0000 7fdf577fe640 1 -- 172.21.15.1:0/173397321 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdf582004a0 con 0x7fdf58169690 2024-05-09T05:14:28.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.347+0000 7fdf577fe640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf58169690 0x7fdf581ff540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdf580c55b0 tx=0x7fdf4800fb40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.347+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf48017020 con 0x7fdf58169690 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.348+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdf5001c070 con 0x7fdf5816a010 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.348+0000 7fdf3ffff640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf44076200 0x7fdf44078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.348+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdf500976b0 con 0x7fdf5816a010 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.348+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdf4800d6d0 con 0x7fdf58169690 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.348+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf58169690 msgr2=0x7fdf581ff540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.348+0000 7fdf3ffff640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf58169690 0x7fdf581ff540 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fdf580c55b0 tx=0x7fdf4800fb40 comp rx=0 tx=0).stop 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.348+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf58169690 msgr2=0x7fdf581ff540 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.348+0000 7fdf3ffff640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf58169690 0x7fdf581ff540 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.349+0000 7fdf56ffd640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf44076200 0x7fdf44078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.349+0000 7fdf56ffd640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf44076200 0x7fdf44078710 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c037000 tx=0x7fdf4c032000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.359+0000 7fdf5df01640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816cf60 0x7fdf5816d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.359+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdf5816d8b0 con 0x7fdf5816cf60 2024-05-09T05:14:28.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.359+0000 7fdf577fe640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816cf60 0x7fdf5816d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.359+0000 7fdf577fe640 1 -- 172.21.15.1:0/173397321 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdf58050870 con 0x7fdf5816cf60 2024-05-09T05:14:28.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf577fe640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816cf60 0x7fdf5816d370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf580c55b0 tx=0x7fdf48001d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdf4801e860 con 0x7fdf5816cf60 2024-05-09T05:14:28.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7fdf4801e860 con 0x7fdf5816cf60 2024-05-09T05:14:28.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816cf60 msgr2=0x7fdf5816d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf3ffff640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816cf60 0x7fdf5816d370 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf580c55b0 tx=0x7fdf48001d30 comp rx=0 tx=0).stop 2024-05-09T05:14:28.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf3ffff640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816cf60 msgr2=0x7fdf5816d370 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:28.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf3ffff640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdf5816cf60 0x7fdf5816d370 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.360 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:28.360 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 68, 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 130639, 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:28.361 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:28.362 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:28.363 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:28.364 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.365 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:28.366 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:28.367 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:28.368 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.369 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.370 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:14:28.371 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:28.372 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:28.373 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:28.373 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:28.373 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:28.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf44076200 msgr2=0x7fdf44078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.360+0000 7fdf5df01640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdf44076200 0x7fdf44078710 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fdf4c037000 tx=0x7fdf4c032000 comp rx=0 tx=0).stop 2024-05-09T05:14:28.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.361+0000 7fdf57fff640 1 -- 172.21.15.1:0/173397321 reap_dead start 2024-05-09T05:14:28.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.361+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5816a010 msgr2=0x7fdf581612c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.361+0000 7fdf5df01640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5816a010 0x7fdf581612c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fdf50004630 tx=0x7fdf5000a710 comp rx=0 tx=0).stop 2024-05-09T05:14:28.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.361+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 shutdown_connections 2024-05-09T05:14:28.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.361+0000 7fdf5df01640 1 --2- 172.21.15.1:0/173397321 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdf5816a010 0x7fdf581612c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.361+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 >> 172.21.15.1:0/173397321 conn(0x7fdf58070f40 msgr2=0x7fdf580cc2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:28.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.361+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 shutdown_connections 2024-05-09T05:14:28.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.361+0000 7fdf5df01640 1 -- 172.21.15.1:0/173397321 wait complete. 2024-05-09T05:14:28.375 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:14:28.375 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:14:28.376 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:14:28.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.506+0000 7f5dba346640 1 Processor -- start 2024-05-09T05:14:28.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.506+0000 7f5dba346640 1 -- start start 2024-05-09T05:14:28.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.507+0000 7f5dba346640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5db40c9340 0x7f5db415e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.507+0000 7f5dba346640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 0x7f5db4164f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.507+0000 7f5dba346640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5db40cb040 con 0x7f5db40c9a00 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.507+0000 7f5dba346640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f5db40c3c70 con 0x7f5db40c9340 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.507+0000 7f5dba346640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40c3f50 con 0x7f5db4072310 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.507+0000 7f5db3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5db40c9340 0x7f5db415e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.507+0000 7f5db3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5db40c9340 0x7f5db415e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35978/0 (socket says 172.21.15.1:35978) 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.507+0000 7f5db3fff640 1 -- 172.21.15.1:0/2436363482 learned_addr learned my addr 172.21.15.1:0/2436363482 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.508+0000 7f5db88e3640 1 --2- 172.21.15.1:0/2436363482 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 0x7f5db4164f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.508+0000 7f5db3fff640 1 -- 172.21.15.1:0/2436363482 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 msgr2=0x7f5db4164f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.508+0000 7f5db3fff640 1 --2- 172.21.15.1:0/2436363482 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 0x7f5db4164f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.508+0000 7f5db3fff640 1 -- 172.21.15.1:0/2436363482 >> v1:172.21.15.1:6789/0 conn(0x7f5db40c9a00 legacy=0x7f5db40c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.508+0000 7f5db3fff640 1 -- 172.21.15.1:0/2436363482 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db40c4230 con 0x7f5db40c9340 2024-05-09T05:14:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.509+0000 7f5db3fff640 1 --2- 172.21.15.1:0/2436363482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9340 0x7f5db415e490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009870 tx=0x7f5da402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbd26692f552b3ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.509+0000 7f5db2ffd640 1 -- 172.21.15.1:0/2436363482 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4037070 con 0x7f5db40c9340 2024-05-09T05:14:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.509+0000 7f5db2ffd640 1 -- 172.21.15.1:0/2436363482 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5da4032440 con 0x7f5db40c9340 2024-05-09T05:14:28.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.509+0000 7f5db2ffd640 1 -- 172.21.15.1:0/2436363482 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4032810 con 0x7f5db40c9340 2024-05-09T05:14:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.510+0000 7f5dba346640 1 -- 172.21.15.1:0/2436363482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9340 msgr2=0x7f5db415e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.510+0000 7f5dba346640 1 --2- 172.21.15.1:0/2436363482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9340 0x7f5db415e490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5da4009870 tx=0x7f5da402a270 comp rx=0 tx=0).stop 2024-05-09T05:14:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.510+0000 7f5dba346640 1 -- 172.21.15.1:0/2436363482 shutdown_connections 2024-05-09T05:14:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.510+0000 7f5dba346640 1 --2- 172.21.15.1:0/2436363482 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 0x7f5db4164f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.510+0000 7f5dba346640 1 --2- 172.21.15.1:0/2436363482 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9340 0x7f5db415e490 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.510+0000 7f5dba346640 1 -- 172.21.15.1:0/2436363482 >> 172.21.15.1:0/2436363482 conn(0x7f5db40747b0 msgr2=0x7f5db40ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.510+0000 7f5dba346640 1 -- 172.21.15.1:0/2436363482 shutdown_connections 2024-05-09T05:14:28.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.510+0000 7f5dba346640 1 -- 172.21.15.1:0/2436363482 wait complete. 2024-05-09T05:14:28.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.512+0000 7f5dba346640 1 Processor -- start 2024-05-09T05:14:28.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.512+0000 7f5dba346640 1 -- start start 2024-05-09T05:14:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5dba346640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 0x7f5db41f4960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5dba346640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db40c9340 0x7f5db41f1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5dba346640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9a00 0x7f5db4148140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5dba346640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5db40c5160 con 0x7f5db40c9a00 2024-05-09T05:14:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5dba346640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5db415df70 con 0x7f5db40c9340 2024-05-09T05:14:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5dba346640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40716a0 con 0x7f5db4072310 2024-05-09T05:14:28.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5db88e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9a00 0x7f5db4148140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5db88e3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9a00 0x7f5db4148140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35992/0 (socket says 172.21.15.1:35992) 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5db88e3640 1 -- 172.21.15.1:0/621937883 learned_addr learned my addr 172.21.15.1:0/621937883 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 0x7f5db41f4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.514+0000 7f5db37fe640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db40c9340 0x7f5db41f1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db88e3640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 msgr2=0x7f5db41f4960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db88e3640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4072310 0x7f5db41f4960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db88e3640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db40c9340 msgr2=0x7f5db41f1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db88e3640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5db40c9340 0x7f5db41f1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db88e3640 1 -- 172.21.15.1:0/621937883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5db41f4ea0 con 0x7f5db40c9a00 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db88e3640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9a00 0x7f5db4148140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5dac007d70 tx=0x7f5dac00e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac014420 con 0x7f5db40c9a00 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5dac019070 con 0x7f5db40c9a00 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5db41f1660 con 0x7f5db40c9a00 2024-05-09T05:14:28.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5dac014820 con 0x7f5db40c9a00 2024-05-09T05:14:28.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.515+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5db4158e20 con 0x7f5db40c9a00 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.516+0000 7f5dba346640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.516+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40ca880 con 0x7f5db4159020 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.516+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.516+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.516+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.516+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.517+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.517+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5dac0149c0 con 0x7f5db40c9a00 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.518+0000 7f5db0ff9640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5da0076300 0x7f5da0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.518+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5dac097f20 con 0x7f5db40c9a00 2024-05-09T05:14:28.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.518+0000 7f5db37fe640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5da0076300 0x7f5da0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.519+0000 7f5db37fe640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5da0076300 0x7f5da0078810 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5db414d600 tx=0x7f5da8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.718+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.718+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:14:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.718+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:14:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.718+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:28.718+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:14:29.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.119+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:29.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.119+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:14:29.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.119+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:14:29.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.119+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:29.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.119+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:14:29.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.920+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:29.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.920+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 21 2024-05-09T05:14:29.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.920+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:14:29.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.920+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:29.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:29.920+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:14:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.522+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.522+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5db4050870 con 0x7f5db4159020 2024-05-09T05:14:31.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.523+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5db40cf590 tx=0x7f5da4004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.523+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4037040 con 0x7f5db4159020 2024-05-09T05:14:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.523+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5da403c410 con 0x7f5db4159020 2024-05-09T05:14:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.523+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.523+0000 7f5db0ff9640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f5db40cf590 tx=0x7f5da4004730 comp rx=0 tx=0).stop 2024-05-09T05:14:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.523+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 msgr2=0x7f5db41f0a10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:31.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.523+0000 7f5db0ff9640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4159020 0x7f5db41f0a10 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.530+0000 7f5dba346640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4148680 0x7f5db415bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:31.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.530+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5db40c97e0 con 0x7f5db4148680 2024-05-09T05:14:31.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.530+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4148680 0x7f5db415bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.530+0000 7f5db3fff640 1 -- 172.21.15.1:0/621937883 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5db40c8c30 con 0x7f5db4148680 2024-05-09T05:14:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.531+0000 7f5db3fff640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4148680 0x7f5db415bdb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5da4000c00 tx=0x7f5da40046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.531+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5da4037040 con 0x7f5db4148680 2024-05-09T05:14:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.531+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4550 (secure 0 0 0) 0x7f5da404e870 con 0x7f5db4148680 2024-05-09T05:14:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.531+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4148680 msgr2=0x7f5db415bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.531+0000 7f5db0ff9640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4148680 0x7f5db415bdb0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f5da4000c00 tx=0x7f5da40046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.531+0000 7f5db0ff9640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4148680 msgr2=0x7f5db415bdb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:31.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.531+0000 7f5db0ff9640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5db4148680 0x7f5db415bdb0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 68, 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 10881, 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:14:31.532 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 5, 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:31.533 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:31.534 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:31.535 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:31.536 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:31.537 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:31.538 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:31.539 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.540 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:31.541 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:31.542 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:31.543 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:31.544 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:31.545 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:31.546 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:31.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5da0076300 msgr2=0x7f5da0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:31.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5da0076300 0x7f5da0078810 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5db414d600 tx=0x7f5da8009040 comp rx=0 tx=0).stop 2024-05-09T05:14:31.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9a00 msgr2=0x7f5db4148140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:31.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9a00 0x7f5db4148140 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5dac007d70 tx=0x7f5dac00e4b0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5db88e3640 1 -- 172.21.15.1:0/621937883 reap_dead start 2024-05-09T05:14:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 shutdown_connections 2024-05-09T05:14:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 --2- 172.21.15.1:0/621937883 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5db40c9a00 0x7f5db4148140 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 >> 172.21.15.1:0/621937883 conn(0x7f5db40747b0 msgr2=0x7f5db4153920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 shutdown_connections 2024-05-09T05:14:31.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.532+0000 7f5dba346640 1 -- 172.21.15.1:0/621937883 wait complete. 2024-05-09T05:14:31.548 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:14:31.548 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:14:31.549 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:14:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.680+0000 7fe322340640 1 Processor -- start 2024-05-09T05:14:31.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.680+0000 7fe322340640 1 -- start start 2024-05-09T05:14:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe322340640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe31c0ca340 0x7fe31c0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe322340640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe31c072120 0x7fe31c1547b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe322340640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe31c154cf0 con 0x7fe31c0c62e0 2024-05-09T05:14:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe322340640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe31c0c53f0 con 0x7fe31c0ca340 2024-05-09T05:14:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe322340640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe31c0c55b0 con 0x7fe31c072120 2024-05-09T05:14:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe31bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe31c0ca340 0x7fe31c0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe31bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe31c0ca340 0x7fe31c0c4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48774/0 (socket says 172.21.15.1:48774) 2024-05-09T05:14:31.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe31bfff640 1 -- 172.21.15.1:0/1034507561 learned_addr learned my addr 172.21.15.1:0/1034507561 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:31.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe313fff640 1 --2- 172.21.15.1:0/1034507561 >> v2:172.21.15.196:3300/0 conn(0x7fe31c072120 0x7fe31c1547b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:31.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe31bfff640 1 -- 172.21.15.1:0/1034507561 >> v2:172.21.15.196:3300/0 conn(0x7fe31c072120 msgr2=0x7fe31c1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:31.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe31bfff640 1 --2- 172.21.15.1:0/1034507561 >> v2:172.21.15.196:3300/0 conn(0x7fe31c072120 0x7fe31c1547b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe31bfff640 1 -- 172.21.15.1:0/1034507561 >> v1:172.21.15.196:6789/0 conn(0x7fe31c0c62e0 legacy=0x7fe31c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:31.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.682+0000 7fe31bfff640 1 -- 172.21.15.1:0/1034507561 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c15b110 con 0x7fe31c0ca340 2024-05-09T05:14:31.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.683+0000 7fe31bfff640 1 --2- 172.21.15.1:0/1034507561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c0ca340 0x7fe31c0c4eb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0b37a0 tx=0x7fe30c02a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff5acf1bc83f0a29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:31.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.683+0000 7fe31b7fe640 1 -- 172.21.15.1:0/1034507561 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c037070 con 0x7fe31c0ca340 2024-05-09T05:14:31.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.683+0000 7fe31b7fe640 1 -- 172.21.15.1:0/1034507561 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30c032440 con 0x7fe31c0ca340 2024-05-09T05:14:31.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.683+0000 7fe31b7fe640 1 -- 172.21.15.1:0/1034507561 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c032810 con 0x7fe31c0ca340 2024-05-09T05:14:31.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.684+0000 7fe322340640 1 -- 172.21.15.1:0/1034507561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c0ca340 msgr2=0x7fe31c0c4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:31.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.684+0000 7fe322340640 1 --2- 172.21.15.1:0/1034507561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c0ca340 0x7fe31c0c4eb0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0b37a0 tx=0x7fe30c02a1e0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.684+0000 7fe322340640 1 -- 172.21.15.1:0/1034507561 shutdown_connections 2024-05-09T05:14:31.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.684+0000 7fe322340640 1 --2- 172.21.15.1:0/1034507561 >> v2:172.21.15.196:3300/0 conn(0x7fe31c072120 0x7fe31c1547b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.684+0000 7fe322340640 1 --2- 172.21.15.1:0/1034507561 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c0ca340 0x7fe31c0c4eb0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.684+0000 7fe322340640 1 -- 172.21.15.1:0/1034507561 >> 172.21.15.1:0/1034507561 conn(0x7fe31c0731d0 msgr2=0x7fe31c0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:31.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.684+0000 7fe322340640 1 -- 172.21.15.1:0/1034507561 shutdown_connections 2024-05-09T05:14:31.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.684+0000 7fe322340640 1 -- 172.21.15.1:0/1034507561 wait complete. 2024-05-09T05:14:31.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.686+0000 7fe322340640 1 Processor -- start 2024-05-09T05:14:31.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.686+0000 7fe322340640 1 -- start start 2024-05-09T05:14:31.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe322340640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c072120 0x7fe31c15b700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:31.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe322340640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe31c0c62e0 0x7fe31c06d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:31.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe322340640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe31c0ca340 0x7fe31c159b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:31.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe322340640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c0c5ae0 con 0x7fe31c072120 2024-05-09T05:14:31.689 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe322340640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe31c0c5800 con 0x7fe31c0ca340 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe322340640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe31c0c5dc0 con 0x7fe31c0c62e0 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe31bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c072120 0x7fe31c15b700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe31bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c072120 0x7fe31c15b700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:48790/0 (socket says 172.21.15.1:48790) 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe31bfff640 1 -- 172.21.15.1:0/3263325752 learned_addr learned my addr 172.21.15.1:0/3263325752 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe313fff640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe31c0c62e0 0x7fe31c06d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe3208dd640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe31c0ca340 0x7fe31c159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe31bfff640 1 -- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe31c0c62e0 msgr2=0x7fe31c06d800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe31bfff640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe31c0c62e0 0x7fe31c06d800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe31bfff640 1 -- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe31c0ca340 msgr2=0x7fe31c159b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe31bfff640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe31c0ca340 0x7fe31c159b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.688+0000 7fe31bfff640 1 -- 172.21.15.1:0/3263325752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe31c1f5c40 con 0x7fe31c072120 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.689+0000 7fe31bfff640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c072120 0x7fe31c15b700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe30c02a220 tx=0x7fe30c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.689+0000 7fe3197fa640 1 -- 172.21.15.1:0/3263325752 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c037040 con 0x7fe31c072120 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.689+0000 7fe3197fa640 1 -- 172.21.15.1:0/3263325752 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe30c042070 con 0x7fe31c072120 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.689+0000 7fe3197fa640 1 -- 172.21.15.1:0/3263325752 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe30c030880 con 0x7fe31c072120 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.689+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe31c1f5eb0 con 0x7fe31c072120 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.689+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe31c1f6470 con 0x7fe31c072120 2024-05-09T05:14:31.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.690+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe31c050870 con 0x7fe31c072120 2024-05-09T05:14:31.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.691+0000 7fe3197fa640 1 -- 172.21.15.1:0/3263325752 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe30c030a20 con 0x7fe31c072120 2024-05-09T05:14:31.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.691+0000 7fe3197fa640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe2fc076300 0x7fe2fc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:31.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.692+0000 7fe3197fa640 1 -- 172.21.15.1:0/3263325752 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe30c0b7e90 con 0x7fe31c072120 2024-05-09T05:14:31.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.692+0000 7fe313fff640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe2fc076300 0x7fe2fc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:31.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.692+0000 7fe313fff640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe2fc076300 0x7fe2fc078810 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0c9ef0 tx=0x7fe304009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:31.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:31.696+0000 7fe3197fa640 1 -- 172.21.15.1:0/3263325752 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe30c081090 con 0x7fe31c072120 2024-05-09T05:14:32.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.030+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fe31c0cba10 con 0x7fe31c072120 2024-05-09T05:14:32.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.031+0000 7fe3197fa640 1 -- 172.21.15.1:0/3263325752 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fe30c085390 con 0x7fe31c072120 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe2fc076300 msgr2=0x7fe2fc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe2fc076300 0x7fe2fc078810 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe31c0c9ef0 tx=0x7fe304009190 comp rx=0 tx=0).stop 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c072120 msgr2=0x7fe31c15b700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c072120 0x7fe31c15b700 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fe30c02a220 tx=0x7fe30c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 shutdown_connections 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe2fc076300 0x7fe2fc078810 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe31c0ca340 0x7fe31c159b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe31c0c62e0 0x7fe31c06d800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:32.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 --2- 172.21.15.1:0/3263325752 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe31c072120 0x7fe31c15b700 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:32.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 >> 172.21.15.1:0/3263325752 conn(0x7fe31c0731d0 msgr2=0x7fe31c0761b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:32.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 shutdown_connections 2024-05-09T05:14:32.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:32.033+0000 7fe322340640 1 -- 172.21.15.1:0/3263325752 wait complete. 2024-05-09T05:14:32.045 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:14:33.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:14:33.525 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:14:42.046 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:14:42.047 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:14:42.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.164+0000 7f0176538640 1 Processor -- start 2024-05-09T05:14:42.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.164+0000 7f0176538640 1 -- start start 2024-05-09T05:14:42.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.166+0000 7f0176538640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f017006ce00 0x7f0170160750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.166+0000 7f0176538640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0170156f10 con 0x7f01700c4bc0 2024-05-09T05:14:42.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.166+0000 7f0176538640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f017006d2d0 con 0x7f01700c47b0 2024-05-09T05:14:42.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.166+0000 7f0176538640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f01700cbbf0 con 0x7f017006ce00 2024-05-09T05:14:42.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.167+0000 7f016ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f017006ce00 0x7f0170160750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.167+0000 7f016ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f017006ce00 0x7f0170160750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40294/0 (socket says 172.21.15.1:40294) 2024-05-09T05:14:42.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.167+0000 7f016ffff640 1 -- 172.21.15.1:0/4098287712 learned_addr learned my addr 172.21.15.1:0/4098287712 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:42.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.167+0000 7f0174ad5640 1 --1- 172.21.15.1:0/4098287712 >> v1:172.21.15.1:6789/0 conn(0x7f01700c4bc0 0x7f01701670d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51104/0 (socket says 172.21.15.1:51104) 2024-05-09T05:14:42.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.167+0000 7f016f7fe640 1 -- 172.21.15.1:0/4098287712 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2028891284 0 0) 0x7f017006d2d0 con 0x7f01700c47b0 2024-05-09T05:14:42.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.167+0000 7f016ffff640 1 -- 172.21.15.1:0/4098287712 >> v1:172.21.15.196:6789/0 conn(0x7f01700c47b0 legacy=0x7f01700ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.167+0000 7f016ffff640 1 -- 172.21.15.1:0/4098287712 >> v1:172.21.15.1:6789/0 conn(0x7f01700c4bc0 legacy=0x7f01701670d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.167+0000 7f016ffff640 1 -- 172.21.15.1:0/4098287712 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f017006d570 con 0x7f017006ce00 2024-05-09T05:14:42.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.168+0000 7f016ffff640 1 --2- 172.21.15.1:0/4098287712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f0170160750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f01700cbd70 tx=0x7f016002a370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f87df7964a390ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.168+0000 7f016f7fe640 1 -- 172.21.15.1:0/4098287712 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160037070 con 0x7f017006ce00 2024-05-09T05:14:42.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.168+0000 7f016f7fe640 1 -- 172.21.15.1:0/4098287712 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0160032640 con 0x7f017006ce00 2024-05-09T05:14:42.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.168+0000 7f016f7fe640 1 -- 172.21.15.1:0/4098287712 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160032a10 con 0x7f017006ce00 2024-05-09T05:14:42.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.169+0000 7f0176538640 1 -- 172.21.15.1:0/4098287712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 msgr2=0x7f0170160750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.169+0000 7f0176538640 1 --2- 172.21.15.1:0/4098287712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f0170160750 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f01700cbd70 tx=0x7f016002a370 comp rx=0 tx=0).stop 2024-05-09T05:14:42.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.169+0000 7f0176538640 1 -- 172.21.15.1:0/4098287712 shutdown_connections 2024-05-09T05:14:42.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.169+0000 7f0176538640 1 --2- 172.21.15.1:0/4098287712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f0170160750 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.169+0000 7f0176538640 1 -- 172.21.15.1:0/4098287712 >> 172.21.15.1:0/4098287712 conn(0x7f017006e400 msgr2=0x7f0170071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:42.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.169+0000 7f0176538640 1 -- 172.21.15.1:0/4098287712 shutdown_connections 2024-05-09T05:14:42.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.169+0000 7f0176538640 1 -- 172.21.15.1:0/4098287712 wait complete. 2024-05-09T05:14:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.171+0000 7f0176538640 1 Processor -- start 2024-05-09T05:14:42.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.172+0000 7f0176538640 1 -- start start 2024-05-09T05:14:42.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.173+0000 7f0176538640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f01701f2690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.173+0000 7f0176538640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01700c47b0 0x7f017015f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.173+0000 7f0176538640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01700c4bc0 0x7f01700c9750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.173+0000 7f0176538640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f01700cb880 con 0x7f017006ce00 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.173+0000 7f0176538640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f01700cb300 con 0x7f01700c47b0 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.173+0000 7f0176538640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f01700cc080 con 0x7f01700c4bc0 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f01701f2690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f01701f2690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40306/0 (socket says 172.21.15.1:40306) 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 -- 172.21.15.1:0/61982751 learned_addr learned my addr 172.21.15.1:0/61982751 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f01677fe640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01700c47b0 0x7f017015f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 -- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01700c4bc0 msgr2=0x7f01700c9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f0174ad5640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01700c4bc0 0x7f01700c9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01700c4bc0 0x7f01700c9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 -- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01700c47b0 msgr2=0x7f017015f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01700c47b0 0x7f017015f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 -- 172.21.15.1:0/61982751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01701f2bd0 con 0x7f017006ce00 2024-05-09T05:14:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f01677fe640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01700c47b0 0x7f017015f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:14:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f0174ad5640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01700c4bc0 0x7f01700c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:14:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.174+0000 7f016ffff640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f01701f2690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f016002a3b0 tx=0x7f01600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.175+0000 7f016d7fa640 1 -- 172.21.15.1:0/61982751 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160037040 con 0x7f017006ce00 2024-05-09T05:14:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.175+0000 7f016d7fa640 1 -- 172.21.15.1:0/61982751 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0160042070 con 0x7f017006ce00 2024-05-09T05:14:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.175+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01701e7b60 con 0x7f017006ce00 2024-05-09T05:14:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.175+0000 7f016d7fa640 1 -- 172.21.15.1:0/61982751 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0160030bb0 con 0x7f017006ce00 2024-05-09T05:14:42.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.175+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01701e7e00 con 0x7f017006ce00 2024-05-09T05:14:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.176+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01700ca4e0 con 0x7f017006ce00 2024-05-09T05:14:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.177+0000 7f016d7fa640 1 -- 172.21.15.1:0/61982751 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0160032500 con 0x7f017006ce00 2024-05-09T05:14:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.178+0000 7f016d7fa640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0150076300 0x7f0150078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.178+0000 7f016d7fa640 1 -- 172.21.15.1:0/61982751 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f01600b8170 con 0x7f017006ce00 2024-05-09T05:14:42.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.178+0000 7f01677fe640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0150076300 0x7f0150078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.179+0000 7f01677fe640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0150076300 0x7f0150078810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f01700c5a50 tx=0x7f0158009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.183+0000 7f016d7fa640 1 -- 172.21.15.1:0/61982751 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f01700ca4e0 con 0x7f017006ce00 2024-05-09T05:14:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.477+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f0170050870 con 0x7f017006ce00 2024-05-09T05:14:42.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.477+0000 7f016d7fa640 1 -- 172.21.15.1:0/61982751 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f0160081a50 con 0x7f017006ce00 2024-05-09T05:14:42.476 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0150076300 msgr2=0x7f0150078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0150076300 0x7f0150078810 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f01700c5a50 tx=0x7f0158009040 comp rx=0 tx=0).stop 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 msgr2=0x7f01701f2690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f01701f2690 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f016002a3b0 tx=0x7f01600046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 shutdown_connections 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0150076300 0x7f0150078810 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f01700c47b0 0x7f017015f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f01700c4bc0 0x7f01700c9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 --2- 172.21.15.1:0/61982751 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f017006ce00 0x7f01701f2690 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 >> 172.21.15.1:0/61982751 conn(0x7f017006e400 msgr2=0x7f0170074120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 shutdown_connections 2024-05-09T05:14:42.480 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.479+0000 7f0176538640 1 -- 172.21.15.1:0/61982751 wait complete. 2024-05-09T05:14:42.490 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:14:42.490 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:14:42.490 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:14:42.490 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:14:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.637+0000 7f4c7748b640 1 Processor -- start 2024-05-09T05:14:42.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.637+0000 7f4c7748b640 1 -- start start 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c7748b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c700cc970 0x7f4c7015b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c7748b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 0x7f4c70165030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c7748b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4c700cce20 con 0x7f4c700736b0 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c7748b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4c700cd100 con 0x7f4c700cc970 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c7748b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4c700cd3e0 con 0x7f4c700689e0 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c75227640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c700cc970 0x7f4c7015b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c75227640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4c700cc970 0x7f4c7015b0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40312/0 (socket says 172.21.15.1:40312) 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c75227640 1 -- 172.21.15.1:0/1510692988 learned_addr learned my addr 172.21.15.1:0/1510692988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c75a28640 1 --2- 172.21.15.1:0/1510692988 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 0x7f4c70165030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c75227640 1 -- 172.21.15.1:0/1510692988 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 msgr2=0x7f4c70165030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:42.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c75227640 1 --2- 172.21.15.1:0/1510692988 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 0x7f4c70165030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c75227640 1 -- 172.21.15.1:0/1510692988 >> v1:172.21.15.196:6789/0 conn(0x7f4c700736b0 legacy=0x7f4c700c7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.639+0000 7f4c75227640 1 -- 172.21.15.1:0/1510692988 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c7015a8c0 con 0x7f4c700cc970 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.640+0000 7f4c75227640 1 --2- 172.21.15.1:0/1510692988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700cc970 0x7f4c7015b0c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009870 tx=0x7f4c6c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2a03f4802b7dd69 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.640+0000 7f4c67fff640 1 -- 172.21.15.1:0/1510692988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c037070 con 0x7f4c700cc970 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.640+0000 7f4c67fff640 1 -- 172.21.15.1:0/1510692988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c6c032440 con 0x7f4c700cc970 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.640+0000 7f4c67fff640 1 -- 172.21.15.1:0/1510692988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c032810 con 0x7f4c700cc970 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.641+0000 7f4c7748b640 1 -- 172.21.15.1:0/1510692988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700cc970 msgr2=0x7f4c7015b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.641+0000 7f4c7748b640 1 --2- 172.21.15.1:0/1510692988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700cc970 0x7f4c7015b0c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4c6c009870 tx=0x7f4c6c02a270 comp rx=0 tx=0).stop 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.641+0000 7f4c7748b640 1 -- 172.21.15.1:0/1510692988 shutdown_connections 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.641+0000 7f4c7748b640 1 --2- 172.21.15.1:0/1510692988 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 0x7f4c70165030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.641+0000 7f4c7748b640 1 --2- 172.21.15.1:0/1510692988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700cc970 0x7f4c7015b0c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.641+0000 7f4c7748b640 1 -- 172.21.15.1:0/1510692988 >> 172.21.15.1:0/1510692988 conn(0x7f4c700ce180 msgr2=0x7f4c700ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.641+0000 7f4c7748b640 1 -- 172.21.15.1:0/1510692988 shutdown_connections 2024-05-09T05:14:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.641+0000 7f4c7748b640 1 -- 172.21.15.1:0/1510692988 wait complete. 2024-05-09T05:14:42.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.643+0000 7f4c7748b640 1 Processor -- start 2024-05-09T05:14:42.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.643+0000 7f4c7748b640 1 -- start start 2024-05-09T05:14:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c7748b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 0x7f4c701f1420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c7748b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700736b0 0x7f4c70144fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c7748b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c700cc970 0x7f4c701488d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c7748b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c70154d20 con 0x7f4c700736b0 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c7748b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7015b600 con 0x7f4c700cc970 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c7748b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4c7006c9c0 con 0x7f4c700689e0 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c74a26640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700736b0 0x7f4c70144fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c74a26640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700736b0 0x7f4c70144fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40328/0 (socket says 172.21.15.1:40328) 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c74a26640 1 -- 172.21.15.1:0/863042614 learned_addr learned my addr 172.21.15.1:0/863042614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c75a28640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c700cc970 0x7f4c701488d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c75227640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 0x7f4c701f1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c74a26640 1 -- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 msgr2=0x7f4c701f1420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c74a26640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4c700689e0 0x7f4c701f1420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c74a26640 1 -- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c700cc970 msgr2=0x7f4c701488d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c74a26640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4c700cc970 0x7f4c701488d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.646 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.645+0000 7f4c74a26640 1 -- 172.21.15.1:0/863042614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4c70145510 con 0x7f4c700736b0 2024-05-09T05:14:42.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.646+0000 7f4c74a26640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700736b0 0x7f4c70144fd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000c3f0 tx=0x7f4c6000c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.646+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c60017070 con 0x7f4c700736b0 2024-05-09T05:14:42.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.646+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4c600052d0 con 0x7f4c700736b0 2024-05-09T05:14:42.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.646+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4c70146920 con 0x7f4c700736b0 2024-05-09T05:14:42.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.646+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c60003830 con 0x7f4c700736b0 2024-05-09T05:14:42.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.646+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4c70147410 con 0x7f4c700736b0 2024-05-09T05:14:42.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.647+0000 7f4c7748b640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70146130 0x7f4c701447d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.647+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c7014f050 con 0x7f4c70146130 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.647+0000 7f4c75227640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70146130 0x7f4c701447d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.648+0000 7f4c75227640 1 -- 172.21.15.1:0/863042614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4c701466c0 con 0x7f4c70146130 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.648+0000 7f4c75227640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70146130 0x7f4c701447d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4c700734b0 tx=0x7f4c6c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.648+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c043070 con 0x7f4c70146130 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.648+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4c6001a070 con 0x7f4c700736b0 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.649+0000 7f4c65ffb640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4c540761d0 0x7f4c540786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.649+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4c60096880 con 0x7f4c700736b0 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.649+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4c6c03c640 con 0x7f4c70146130 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.649+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70146130 msgr2=0x7f4c701447d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.649+0000 7f4c65ffb640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70146130 0x7f4c701447d0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f4c700734b0 tx=0x7f4c6c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.649+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70146130 msgr2=0x7f4c701447d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:42.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.649+0000 7f4c65ffb640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c70146130 0x7f4c701447d0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.649+0000 7f4c75a28640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4c540761d0 0x7f4c540786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.650+0000 7f4c75a28640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4c540761d0 0x7f4c540786e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4c7015ed60 tx=0x7f4c68009780 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.656+0000 7f4c7748b640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c701439e0 0x7f4c70143df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.656+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4c701540c0 con 0x7f4c701439e0 2024-05-09T05:14:42.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.656+0000 7f4c75227640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c701439e0 0x7f4c70143df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.657+0000 7f4c75227640 1 -- 172.21.15.1:0/863042614 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4c70050870 con 0x7f4c701439e0 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.657+0000 7f4c75227640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c701439e0 0x7f4c70143df0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4c70149480 tx=0x7f4c6c009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.657+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4c6c037040 con 0x7f4c701439e0 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.657+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7f4c6c047e10 con 0x7f4c701439e0 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.657+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c701439e0 msgr2=0x7f4c70143df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.657+0000 7f4c65ffb640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c701439e0 0x7f4c70143df0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f4c70149480 tx=0x7f4c6c009240 comp rx=0 tx=0).stop 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.657+0000 7f4c65ffb640 1 -- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c701439e0 msgr2=0x7f4c70143df0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.657+0000 7f4c65ffb640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c701439e0 0x7f4c70143df0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.657 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 68, 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 84852, 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:42.658 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:42.659 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.660 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:42.661 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.662 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:42.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:42.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:42.665 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.666 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.667 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.668 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.669 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:42.670 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:42.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4c540761d0 msgr2=0x7f4c540786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c7748b640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4c540761d0 0x7f4c540786e0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f4c7015ed60 tx=0x7f4c68009780 comp rx=0 tx=0).stop 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700736b0 msgr2=0x7f4c70144fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c7748b640 1 --2- 172.21.15.1:0/863042614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4c700736b0 0x7f4c70144fd0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f4c6000c3f0 tx=0x7f4c6000c8b0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c75a28640 1 -- 172.21.15.1:0/863042614 reap_dead start 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c75a28640 1 -- 172.21.15.1:0/863042614 reap_dead start 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 shutdown_connections 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 >> 172.21.15.1:0/863042614 conn(0x7f4c700ce180 msgr2=0x7f4c700ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 shutdown_connections 2024-05-09T05:14:42.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.658+0000 7f4c7748b640 1 -- 172.21.15.1:0/863042614 wait complete. 2024-05-09T05:14:42.672 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:14:42.673 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:14:42.673 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:14:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.803+0000 7f858a8bd640 1 Processor -- start 2024-05-09T05:14:42.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.803+0000 7f858a8bd640 1 -- start start 2024-05-09T05:14:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.805+0000 7f858a8bd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f85840c6900 0x7f85841569e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.805+0000 7f858a8bd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f858415ca00 con 0x7f858406d800 2024-05-09T05:14:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.805+0000 7f858a8bd640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f858415cca0 con 0x7f8584072390 2024-05-09T05:14:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.805+0000 7f858a8bd640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8584156f20 con 0x7f85840c6900 2024-05-09T05:14:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.805+0000 7f8588e5a640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f858406d800 0x7f85840ca7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51106/0 (socket says 172.21.15.1:51106) 2024-05-09T05:14:42.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.805+0000 7f8588e5a640 1 -- 172.21.15.1:0/3561184407 learned_addr learned my addr 172.21.15.1:0/3561184407 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:14:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8583fff640 1 --2- 172.21.15.1:0/3561184407 >> v2:172.21.15.196:3300/0 conn(0x7f85840c6900 0x7f85841569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2203717608 0 0) 0x7f858415ca00 con 0x7f858406d800 2024-05-09T05:14:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f856c003520 con 0x7f858406d800 2024-05-09T05:14:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3072697716 0 0) 0x7f856c003520 con 0x7f858406d800 2024-05-09T05:14:42.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 >> v2:172.21.15.196:3300/0 conn(0x7f85840c6900 msgr2=0x7f85841569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 --2- 172.21.15.1:0/3561184407 >> v2:172.21.15.196:3300/0 conn(0x7f85840c6900 0x7f85841569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 >> v1:172.21.15.196:6789/0 conn(0x7f8584072390 legacy=0x7f8584070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f858415cf80 con 0x7f858406d800 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f857c001a70 con 0x7f858406d800 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 >> v1:172.21.15.1:6789/0 conn(0x7f858406d800 legacy=0x7f85840ca7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f856c003a50 0x7f856c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f856c004800 0x7f856c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f856c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8584156f20 con 0x7f856c005250 2024-05-09T05:14:42.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f856c00c1e0 con 0x7f856c003a50 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f856c00c3e0 con 0x7f856c004800 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8588e5a640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f856c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f8583fff640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f856c003a50 0x7f856c003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.806+0000 7f85837fe640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f856c004800 0x7f856c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.807+0000 7f8588e5a640 1 -- 172.21.15.1:0/3561184407 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f856c004800 msgr2=0x7f856c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.807+0000 7f8588e5a640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f856c004800 0x7f856c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.807+0000 7f8588e5a640 1 -- 172.21.15.1:0/3561184407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f856c003a50 msgr2=0x7f856c003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.807+0000 7f8588e5a640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f856c003a50 0x7f856c003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.807+0000 7f8588e5a640 1 -- 172.21.15.1:0/3561184407 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f856c003520 con 0x7f856c005250 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.807+0000 7f8588e5a640 1 -- 172.21.15.1:0/3561184407 reap_dead start 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.807+0000 7f8588e5a640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f856c00b9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f857c00a8c0 tx=0x7f856c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=634098dd3ee6f4ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c00a4e0 con 0x7f856c005250 2024-05-09T05:14:42.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857c02f070 con 0x7f856c005250 2024-05-09T05:14:42.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f8582ffd640 1 -- 172.21.15.1:0/3561184407 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857c02bc90 con 0x7f856c005250 2024-05-09T05:14:42.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f858a8bd640 1 -- 172.21.15.1:0/3561184407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 msgr2=0x7f856c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f858a8bd640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f856c00b9c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f857c00a8c0 tx=0x7f856c005830 comp rx=0 tx=0).stop 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f858a8bd640 1 -- 172.21.15.1:0/3561184407 shutdown_connections 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f858a8bd640 1 --2- 172.21.15.1:0/3561184407 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f856c00b9c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f858a8bd640 1 -- 172.21.15.1:0/3561184407 >> 172.21.15.1:0/3561184407 conn(0x7f85840d0470 msgr2=0x7f85840cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f858a8bd640 1 -- 172.21.15.1:0/3561184407 shutdown_connections 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.808+0000 7f858a8bd640 1 -- 172.21.15.1:0/3561184407 wait complete. 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.810+0000 7f858a8bd640 1 Processor -- start 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.810+0000 7f858a8bd640 1 -- start start 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f858a8bd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f85841f6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f858a8bd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584153860 0x7f85841e82c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f858a8bd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f858413ffc0 0x7f85841403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f858a8bd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f85841569e0 con 0x7f856c005250 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f858a8bd640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f858415d540 con 0x7f8584153860 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f858a8bd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8584156b60 con 0x7f858413ffc0 2024-05-09T05:14:42.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8583fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f85841f6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8583fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f85841f6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40350/0 (socket says 172.21.15.1:40350) 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8583fff640 1 -- 172.21.15.1:0/1532297046 learned_addr learned my addr 172.21.15.1:0/1532297046 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f85837fe640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584153860 0x7f85841e82c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8588e5a640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f858413ffc0 0x7f85841403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8583fff640 1 -- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f858413ffc0 msgr2=0x7f85841403d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8583fff640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f858413ffc0 0x7f85841403d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8583fff640 1 -- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584153860 msgr2=0x7f85841e82c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8583fff640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584153860 0x7f85841e82c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f8583fff640 1 -- 172.21.15.1:0/1532297046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85841f5ec0 con 0x7f856c005250 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.812+0000 7f85837fe640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584153860 0x7f85841e82c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.813+0000 7f8583fff640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f85841f6960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8574011910 tx=0x7f8574011dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.813+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f857400faf0 con 0x7f856c005250 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.813+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f857400fc90 con 0x7f856c005250 2024-05-09T05:14:42.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.813+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85840cdc10 con 0x7f856c005250 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.813+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8574014440 con 0x7f856c005250 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.813+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85841f6440 con 0x7f856c005250 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.814+0000 7f858a8bd640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85840ce8b0 0x7f8584068650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.814+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f85840cd9c0 con 0x7f85840ce8b0 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.814+0000 7f85837fe640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85840ce8b0 0x7f8584068650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.814+0000 7f85837fe640 1 -- 172.21.15.1:0/1532297046 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f85840cecc0 con 0x7f85840ce8b0 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.815+0000 7f85837fe640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85840ce8b0 0x7f8584068650 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f85840b5e20 tx=0x7f8578001660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.815+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8578013570 con 0x7f85840ce8b0 2024-05-09T05:14:42.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.815+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8574012040 con 0x7f856c005250 2024-05-09T05:14:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.816+0000 7f8580ff9640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85700761d0 0x7f85700786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.816+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8574097d60 con 0x7f856c005250 2024-05-09T05:14:42.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.816+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8578013710 con 0x7f85840ce8b0 2024-05-09T05:14:42.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.816+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85840ce8b0 msgr2=0x7f8584068650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.816+0000 7f8580ff9640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85840ce8b0 0x7f8584068650 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f85840b5e20 tx=0x7f8578001660 comp rx=0 tx=0).stop 2024-05-09T05:14:42.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.816+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85840ce8b0 msgr2=0x7f8584068650 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:42.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.816+0000 7f8580ff9640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85840ce8b0 0x7f8584068650 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.816+0000 7f8588e5a640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85700761d0 0x7f85700786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.817+0000 7f8588e5a640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85700761d0 0x7f85700786e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f856c00c1b0 tx=0x7f857c007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.826+0000 7f858a8bd640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584068b90 0x7f8584068fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.826+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f85840694e0 con 0x7f8584068b90 2024-05-09T05:14:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.826+0000 7f85837fe640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584068b90 0x7f8584068fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.827+0000 7f85837fe640 1 -- 172.21.15.1:0/1532297046 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8584050870 con 0x7f8584068b90 2024-05-09T05:14:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.827+0000 7f85837fe640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584068b90 0x7f8584068fa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8578017000 tx=0x7f8578001120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.827+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8578026410 con 0x7f8584068b90 2024-05-09T05:14:42.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.827+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4892 (secure 0 0 0) 0x7f8578014400 con 0x7f8584068b90 2024-05-09T05:14:42.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.827+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584068b90 msgr2=0x7f8584068fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.827+0000 7f8580ff9640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584068b90 0x7f8584068fa0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f8578017000 tx=0x7f8578001120 comp rx=0 tx=0).stop 2024-05-09T05:14:42.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.827+0000 7f8580ff9640 1 -- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584068b90 msgr2=0x7f8584068fa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:42.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.827+0000 7f8580ff9640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8584068b90 0x7f8584068fa0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 68, 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 145105, 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:14:42.828 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.829 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.830 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:42.831 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.832 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.833 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.834 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.835 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:42.836 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:42.837 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:42.837 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:42.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.837 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:42.837 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:42.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:42.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:42.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:42.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:42.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.842 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:42.842 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85700761d0 msgr2=0x7f85700786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f858a8bd640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85700761d0 0x7f85700786e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f856c00c1b0 tx=0x7f857c007000 comp rx=0 tx=0).stop 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 msgr2=0x7f85841f6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f858a8bd640 1 --2- 172.21.15.1:0/1532297046 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f856c005250 0x7f85841f6960 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f8574011910 tx=0x7f8574011dd0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f8588e5a640 1 -- 172.21.15.1:0/1532297046 reap_dead start 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f8588e5a640 1 -- 172.21.15.1:0/1532297046 reap_dead start 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 shutdown_connections 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 >> 172.21.15.1:0/1532297046 conn(0x7f85840d0470 msgr2=0x7f8584151010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:42.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 shutdown_connections 2024-05-09T05:14:42.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.828+0000 7f858a8bd640 1 -- 172.21.15.1:0/1532297046 wait complete. 2024-05-09T05:14:42.844 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:14:42.844 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:14:42.844 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:14:42.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.971+0000 7f63a1906640 1 Processor -- start 2024-05-09T05:14:42.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.971+0000 7f63a1906640 1 -- start start 2024-05-09T05:14:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.973+0000 7f63a1906640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f639c0c9340 0x7f639c15e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.973+0000 7f63a1906640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f639c0cb040 con 0x7f639c072310 2024-05-09T05:14:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.973+0000 7f63a1906640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f639c0c3c70 con 0x7f639c0c9a00 2024-05-09T05:14:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.973+0000 7f63a1906640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f639c0c3f50 con 0x7f639c0c9340 2024-05-09T05:14:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.973+0000 7f639affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f639c0c9340 0x7f639c15e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.973+0000 7f639affd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f639c0c9340 0x7f639c15e490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40358/0 (socket says 172.21.15.1:40358) 2024-05-09T05:14:42.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.973+0000 7f639affd640 1 -- 172.21.15.1:0/2304672532 learned_addr learned my addr 172.21.15.1:0/2304672532 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.973+0000 7f639b7fe640 1 --1- 172.21.15.1:0/2304672532 >> v1:172.21.15.1:6789/0 conn(0x7f639c072310 0x7f639c164f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51112/0 (socket says 172.21.15.1:51112) 2024-05-09T05:14:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.974+0000 7f639affd640 1 -- 172.21.15.1:0/2304672532 >> v1:172.21.15.196:6789/0 conn(0x7f639c0c9a00 legacy=0x7f639c0c7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.974+0000 7f639affd640 1 -- 172.21.15.1:0/2304672532 >> v1:172.21.15.1:6789/0 conn(0x7f639c072310 legacy=0x7f639c164f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.974+0000 7f639affd640 1 -- 172.21.15.1:0/2304672532 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c0c4230 con 0x7f639c0c9340 2024-05-09T05:14:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.974+0000 7f639affd640 1 --2- 172.21.15.1:0/2304672532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c0c9340 0x7f639c15e490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f638c009870 tx=0x7f638c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38bd97876b58e12b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f6399ffb640 1 -- 172.21.15.1:0/2304672532 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c037070 con 0x7f639c0c9340 2024-05-09T05:14:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f6399ffb640 1 -- 172.21.15.1:0/2304672532 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f638c02ae50 con 0x7f639c0c9340 2024-05-09T05:14:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f6399ffb640 1 -- 172.21.15.1:0/2304672532 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c032860 con 0x7f639c0c9340 2024-05-09T05:14:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f63a1906640 1 -- 172.21.15.1:0/2304672532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c0c9340 msgr2=0x7f639c15e490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f63a1906640 1 --2- 172.21.15.1:0/2304672532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c0c9340 0x7f639c15e490 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f638c009870 tx=0x7f638c02a300 comp rx=0 tx=0).stop 2024-05-09T05:14:42.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f63a1906640 1 -- 172.21.15.1:0/2304672532 shutdown_connections 2024-05-09T05:14:42.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f63a1906640 1 --2- 172.21.15.1:0/2304672532 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c0c9340 0x7f639c15e490 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f63a1906640 1 -- 172.21.15.1:0/2304672532 >> 172.21.15.1:0/2304672532 conn(0x7f639c0747b0 msgr2=0x7f639c0ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:42.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f63a1906640 1 -- 172.21.15.1:0/2304672532 shutdown_connections 2024-05-09T05:14:42.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.975+0000 7f63a1906640 1 -- 172.21.15.1:0/2304672532 wait complete. 2024-05-09T05:14:42.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.977+0000 7f63a1906640 1 Processor -- start 2024-05-09T05:14:42.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.977+0000 7f63a1906640 1 -- start start 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f63a1906640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c072310 0x7f639c1552b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f63a1906640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c0c9340 0x7f639c0c75c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f63a1906640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f639c0c9a00 0x7f639c0c6730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f63a1906640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f639c0c5160 con 0x7f639c072310 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f63a1906640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f639c15df70 con 0x7f639c0c9a00 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f63a1906640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f639c0716a0 con 0x7f639c0c9340 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f639affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c072310 0x7f639c1552b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f639affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c072310 0x7f639c1552b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40366/0 (socket says 172.21.15.1:40366) 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f639affd640 1 -- 172.21.15.1:0/4182900197 learned_addr learned my addr 172.21.15.1:0/4182900197 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f639b7fe640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f639c0c9a00 0x7f639c0c6730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.979+0000 7f639a7fc640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c0c9340 0x7f639c0c75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f639affd640 1 -- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c0c9340 msgr2=0x7f639c0c75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f639affd640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c0c9340 0x7f639c0c75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f639affd640 1 -- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f639c0c9a00 msgr2=0x7f639c0c6730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f639affd640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f639c0c9a00 0x7f639c0c6730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f639affd640 1 -- 172.21.15.1:0/4182900197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f639c1557f0 con 0x7f639c072310 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f639affd640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c072310 0x7f639c1552b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f638c000c00 tx=0x7f638c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c041070 con 0x7f639c072310 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f638c03c890 con 0x7f639c072310 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f639c1e4dc0 con 0x7f639c072310 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f638c037040 con 0x7f639c072310 2024-05-09T05:14:42.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.980+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f639c1e5380 con 0x7f639c072310 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.981+0000 7f63a1906640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c1e55c0 0x7f639c0c6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.981+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f639c1e4b70 con 0x7f639c1e55c0 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.981+0000 7f639a7fc640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c1e55c0 0x7f639c0c6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.982+0000 7f639a7fc640 1 -- 172.21.15.1:0/4182900197 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f639c1e59d0 con 0x7f639c1e55c0 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.982+0000 7f639a7fc640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c1e55c0 0x7f639c0c6c70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f639000e390 tx=0x7f639000e850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.982+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390019070 con 0x7f639c1e55c0 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.982+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f638c044070 con 0x7f639c072310 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.983+0000 7f6383fff640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6388076200 0x7f6388078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.983+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f638c0b7e90 con 0x7f639c072310 2024-05-09T05:14:42.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.983+0000 7f639b7fe640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6388076200 0x7f6388078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.983+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f63900052e0 con 0x7f639c1e55c0 2024-05-09T05:14:42.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.983+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c1e55c0 msgr2=0x7f639c0c6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.983+0000 7f6383fff640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c1e55c0 0x7f639c0c6c70 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f639000e390 tx=0x7f639000e850 comp rx=0 tx=0).stop 2024-05-09T05:14:42.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.983+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c1e55c0 msgr2=0x7f639c0c6c70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:42.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.983+0000 7f6383fff640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c1e55c0 0x7f639c0c6c70 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f639000e390 tx=0x7f639000e850 comp rx=0 tx=0).stop 2024-05-09T05:14:42.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.984+0000 7f639b7fe640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6388076200 0x7f6388078710 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6394007fd0 tx=0x7f6394007f40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.990+0000 7f63a1906640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c156080 0x7f639c156490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:42.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.990+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f639c15b7d0 con 0x7f639c156080 2024-05-09T05:14:42.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.991+0000 7f639a7fc640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c156080 0x7f639c156490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:42.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.991+0000 7f639a7fc640 1 -- 172.21.15.1:0/4182900197 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f639c0c71b0 con 0x7f639c156080 2024-05-09T05:14:42.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.991+0000 7f639a7fc640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c156080 0x7f639c156490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f639000e390 tx=0x7f6390000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:42.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.991+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6390007020 con 0x7f639c156080 2024-05-09T05:14:42.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f6390007020 con 0x7f639c156080 2024-05-09T05:14:42.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c156080 msgr2=0x7f639c156490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:42.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f6383fff640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c156080 0x7f639c156490 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f639000e390 tx=0x7f6390000ed0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f6383fff640 1 -- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c156080 msgr2=0x7f639c156490 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f6383fff640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f639c156080 0x7f639c156490 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f639000e390 tx=0x7f6390000ed0 comp rx=0 tx=0).stop 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 68, 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 22342, 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:14:42.992 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.993 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.994 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:42.995 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:42.996 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:42.997 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:42.998 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:42.999 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:43.000 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:43.001 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:43.002 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:43.003 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6388076200 msgr2=0x7f6388078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6388076200 0x7f6388078710 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f6394007fd0 tx=0x7f6394007f40 comp rx=0 tx=0).stop 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c072310 msgr2=0x7f639c1552b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f639b7fe640 1 -- 172.21.15.1:0/4182900197 reap_dead start 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c072310 0x7f639c1552b0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f638c000c00 tx=0x7f638c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 shutdown_connections 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 --2- 172.21.15.1:0/4182900197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f639c072310 0x7f639c1552b0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 >> 172.21.15.1:0/4182900197 conn(0x7f639c0747b0 msgr2=0x7f639c0cc6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 shutdown_connections 2024-05-09T05:14:43.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:42.992+0000 7f63a1906640 1 -- 172.21.15.1:0/4182900197 wait complete. 2024-05-09T05:14:43.006 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:14:43.006 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-05-09T05:14:43.006 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:14:43.006 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-05-09T05:14:43.006 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:14:43.006 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:14:43.006 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:14:43.019 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:14:43.019 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:14:43.019 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:14:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.140+0000 7fcaa2bfa640 1 Processor -- start 2024-05-09T05:14:43.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.141+0000 7fcaa2bfa640 1 -- start start 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.142+0000 7fcaa2bfa640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fca9c0d16b0 0x7fca9c0d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.142+0000 7fcaa2bfa640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca9c06f6d0 con 0x7fca9c15ca90 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.142+0000 7fcaa2bfa640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fca9c0cc4d0 con 0x7fca9c06f1b0 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.142+0000 7fcaa2bfa640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fca9c0cee50 con 0x7fca9c0d16b0 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fcaa1197640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fca9c15ca90 0x7fca9c15cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51120/0 (socket says 172.21.15.1:51120) 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fcaa1197640 1 -- 172.21.15.1:0/3037254592 learned_addr learned my addr 172.21.15.1:0/3037254592 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fcaa0996640 1 --2- 172.21.15.1:0/3037254592 >> v2:172.21.15.196:3300/0 conn(0x7fca9c0d16b0 0x7fca9c0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fca9b7fe640 1 -- 172.21.15.1:0/3037254592 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 431478142 0 0) 0x7fca9c06f6d0 con 0x7fca9c15ca90 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fca9b7fe640 1 -- 172.21.15.1:0/3037254592 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca8c003520 con 0x7fca9c15ca90 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fca9b7fe640 1 -- 172.21.15.1:0/3037254592 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 624369204 0 0) 0x7fca9c0cc4d0 con 0x7fca9c06f1b0 2024-05-09T05:14:43.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fca9b7fe640 1 -- 172.21.15.1:0/3037254592 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fca9c06f6d0 con 0x7fca9c06f1b0 2024-05-09T05:14:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fcaa0996640 1 -- 172.21.15.1:0/3037254592 >> v1:172.21.15.196:6789/0 conn(0x7fca9c06f1b0 legacy=0x7fca9c0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fcaa0996640 1 -- 172.21.15.1:0/3037254592 >> v1:172.21.15.1:6789/0 conn(0x7fca9c15ca90 legacy=0x7fca9c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.143+0000 7fcaa0996640 1 -- 172.21.15.1:0/3037254592 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c0cc7b0 con 0x7fca9c0d16b0 2024-05-09T05:14:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.144+0000 7fcaa0996640 1 --2- 172.21.15.1:0/3037254592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 0x7fca9c0d03d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fca900097a0 tx=0x7fca9002a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a83372e672ade3da server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.144+0000 7fca9b7fe640 1 -- 172.21.15.1:0/3037254592 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90037070 con 0x7fca9c0d16b0 2024-05-09T05:14:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.144+0000 7fca9b7fe640 1 -- 172.21.15.1:0/3037254592 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca90032640 con 0x7fca9c0d16b0 2024-05-09T05:14:43.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.144+0000 7fca9b7fe640 1 -- 172.21.15.1:0/3037254592 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90032a10 con 0x7fca9c0d16b0 2024-05-09T05:14:43.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.145+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/3037254592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 msgr2=0x7fca9c0d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:43.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.145+0000 7fcaa2bfa640 1 --2- 172.21.15.1:0/3037254592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 0x7fca9c0d03d0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fca900097a0 tx=0x7fca9002a2c0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.145+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/3037254592 shutdown_connections 2024-05-09T05:14:43.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.145+0000 7fcaa2bfa640 1 --2- 172.21.15.1:0/3037254592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 0x7fca9c0d03d0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.145+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/3037254592 >> 172.21.15.1:0/3037254592 conn(0x7fca9c0b3660 msgr2=0x7fca9c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.145+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/3037254592 shutdown_connections 2024-05-09T05:14:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.145+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/3037254592 wait complete. 2024-05-09T05:14:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.147+0000 7fcaa2bfa640 1 Processor -- start 2024-05-09T05:14:43.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.147+0000 7fcaa2bfa640 1 -- start start 2024-05-09T05:14:43.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.148+0000 7fcaa2bfa640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca9c06f1b0 0x7fca9c1f2580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.148+0000 7fcaa2bfa640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 0x7fca9c0b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.148+0000 7fcaa2bfa640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca9c15ca90 0x7fca9c153c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.148+0000 7fcaa2bfa640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c0ccd70 con 0x7fca9c06f1b0 2024-05-09T05:14:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.148+0000 7fcaa2bfa640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fca9c0cd0e0 con 0x7fca9c0d16b0 2024-05-09T05:14:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.148+0000 7fcaa2bfa640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fca9c0d0910 con 0x7fca9c15ca90 2024-05-09T05:14:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca9c06f1b0 0x7fca9c1f2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa1197640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca9c15ca90 msgr2=0x7fca9c153c10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:43.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa1197640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca9c15ca90 0x7fca9c153c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca9c06f1b0 0x7fca9c1f2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40382/0 (socket says 172.21.15.1:40382) 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 -- 172.21.15.1:0/665050396 learned_addr learned my addr 172.21.15.1:0/665050396 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fca9bfff640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 0x7fca9c0b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 -- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca9c15ca90 msgr2=0x7fca9c153c10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca9c15ca90 0x7fca9c153c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 -- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 msgr2=0x7fca9c0b45c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 0x7fca9c0b45c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 -- 172.21.15.1:0/665050396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca9c1f6650 con 0x7fca9c06f1b0 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fcaa0996640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca9c06f1b0 0x7fca9c1f2580 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fca900098c0 tx=0x7fca900046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.149+0000 7fca997fa640 1 -- 172.21.15.1:0/665050396 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90041030 con 0x7fca9c06f1b0 2024-05-09T05:14:43.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.150+0000 7fca997fa640 1 -- 172.21.15.1:0/665050396 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fca90032440 con 0x7fca9c06f1b0 2024-05-09T05:14:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.150+0000 7fca997fa640 1 -- 172.21.15.1:0/665050396 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca90037040 con 0x7fca9c06f1b0 2024-05-09T05:14:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.150+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca9c1f2ac0 con 0x7fca9c06f1b0 2024-05-09T05:14:43.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.150+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca9c1f5ad0 con 0x7fca9c06f1b0 2024-05-09T05:14:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.151+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fca9c050870 con 0x7fca9c06f1b0 2024-05-09T05:14:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.151+0000 7fca997fa640 1 -- 172.21.15.1:0/665050396 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fca90044070 con 0x7fca9c06f1b0 2024-05-09T05:14:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.152+0000 7fca997fa640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fca80076280 0x7fca80078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.152+0000 7fca997fa640 1 -- 172.21.15.1:0/665050396 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fca900b7c40 con 0x7fca9c06f1b0 2024-05-09T05:14:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.153+0000 7fca9bfff640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fca80076280 0x7fca80078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:43.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.153+0000 7fca9bfff640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fca80076280 0x7fca80078790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fca88001120 tx=0x7fca88009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:43.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.158+0000 7fca997fa640 1 -- 172.21.15.1:0/665050396 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fca90081520 con 0x7fca9c06f1b0 2024-05-09T05:14:43.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.538+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fca9c152cb0 con 0x7fca9c06f1b0 2024-05-09T05:14:43.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.538+0000 7fca997fa640 1 -- 172.21.15.1:0/665050396 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fca90080e20 con 0x7fca9c06f1b0 2024-05-09T05:14:43.538 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:43.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fca80076280 msgr2=0x7fca80078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fca80076280 0x7fca80078790 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fca88001120 tx=0x7fca88009040 comp rx=0 tx=0).stop 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca9c06f1b0 msgr2=0x7fca9c1f2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca9c06f1b0 0x7fca9c1f2580 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fca900098c0 tx=0x7fca900046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 shutdown_connections 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fca80076280 0x7fca80078790 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fca9c0d16b0 0x7fca9c0b45c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fca9c15ca90 0x7fca9c153c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 --2- 172.21.15.1:0/665050396 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fca9c06f1b0 0x7fca9c1f2580 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 >> 172.21.15.1:0/665050396 conn(0x7fca9c0b3660 msgr2=0x7fca9c0ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 shutdown_connections 2024-05-09T05:14:43.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:43.541+0000 7fcaa2bfa640 1 -- 172.21.15.1:0/665050396 wait complete. 2024-05-09T05:14:46.553 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:14:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.677+0000 7ffa8191f640 1 Processor -- start 2024-05-09T05:14:46.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.677+0000 7ffa8191f640 1 -- start start 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa8191f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ffa7c0d16b0 0x7ffa7c0d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa8191f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa7c15ca90 0x7ffa7c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa8191f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa7c06f6d0 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa8191f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ffa7c0cee50 con 0x7ffa7c15ca90 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa8191f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ffa7c0cd0e0 con 0x7ffa7c0d16b0 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa7b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa7c15ca90 0x7ffa7c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa7b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa7c15ca90 0x7ffa7c15cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40394/0 (socket says 172.21.15.1:40394) 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa7b7fe640 1 -- 172.21.15.1:0/1278849509 learned_addr learned my addr 172.21.15.1:0/1278849509 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:46.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.678+0000 7ffa7affd640 1 --2- 172.21.15.1:0/1278849509 >> v2:172.21.15.196:3300/0 conn(0x7ffa7c0d16b0 0x7ffa7c0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.679+0000 7ffa7b7fe640 1 -- 172.21.15.1:0/1278849509 >> v2:172.21.15.196:3300/0 conn(0x7ffa7c0d16b0 msgr2=0x7ffa7c0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.679+0000 7ffa7b7fe640 1 --2- 172.21.15.1:0/1278849509 >> v2:172.21.15.196:3300/0 conn(0x7ffa7c0d16b0 0x7ffa7c0d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.679+0000 7ffa7b7fe640 1 -- 172.21.15.1:0/1278849509 >> v1:172.21.15.1:6789/0 conn(0x7ffa7c06f1b0 legacy=0x7ffa7c0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.679+0000 7ffa7b7fe640 1 -- 172.21.15.1:0/1278849509 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7c0cc4d0 con 0x7ffa7c15ca90 2024-05-09T05:14:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.679+0000 7ffa7b7fe640 1 --2- 172.21.15.1:0/1278849509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c15ca90 0x7ffa7c15cea0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c06d120 tx=0x7ffa7402a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e44a43e5fe3d644 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa79ffb640 1 -- 172.21.15.1:0/1278849509 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74036070 con 0x7ffa7c15ca90 2024-05-09T05:14:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa79ffb640 1 -- 172.21.15.1:0/1278849509 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa74031600 con 0x7ffa7c15ca90 2024-05-09T05:14:46.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa79ffb640 1 -- 172.21.15.1:0/1278849509 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa740319f0 con 0x7ffa7c15ca90 2024-05-09T05:14:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa8191f640 1 -- 172.21.15.1:0/1278849509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c15ca90 msgr2=0x7ffa7c15cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa8191f640 1 --2- 172.21.15.1:0/1278849509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c15ca90 0x7ffa7c15cea0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c06d120 tx=0x7ffa7402a7c0 comp rx=0 tx=0).stop 2024-05-09T05:14:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa8191f640 1 -- 172.21.15.1:0/1278849509 shutdown_connections 2024-05-09T05:14:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa8191f640 1 --2- 172.21.15.1:0/1278849509 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c15ca90 0x7ffa7c15cea0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa8191f640 1 --2- 172.21.15.1:0/1278849509 >> v2:172.21.15.196:3300/0 conn(0x7ffa7c0d16b0 0x7ffa7c0d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa8191f640 1 -- 172.21.15.1:0/1278849509 >> 172.21.15.1:0/1278849509 conn(0x7ffa7c0b3660 msgr2=0x7ffa7c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa8191f640 1 -- 172.21.15.1:0/1278849509 shutdown_connections 2024-05-09T05:14:46.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.680+0000 7ffa8191f640 1 -- 172.21.15.1:0/1278849509 wait complete. 2024-05-09T05:14:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.682+0000 7ffa8191f640 1 Processor -- start 2024-05-09T05:14:46.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.682+0000 7ffa8191f640 1 -- start start 2024-05-09T05:14:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.683+0000 7ffa8191f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c06f1b0 0x7ffa7c0b45c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.683+0000 7ffa8191f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa7c0d16b0 0x7ffa7c0ca280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa8191f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa7c15ca90 0x7ffa7c0b65e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa8191f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c0ccd70 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa8191f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffa7c0cca90 con 0x7ffa7c0d16b0 2024-05-09T05:14:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa8191f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffa7c0d0910 con 0x7ffa7c15ca90 2024-05-09T05:14:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7b7fe640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa7c15ca90 msgr2=0x7ffa7c0b65e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:46.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c06f1b0 0x7ffa7c0b45c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7b7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa7c15ca90 0x7ffa7c0b65e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c06f1b0 0x7ffa7c0b45c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:40410/0 (socket says 172.21.15.1:40410) 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 -- 172.21.15.1:0/996509422 learned_addr learned my addr 172.21.15.1:0/996509422 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7a7fc640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa7c0d16b0 0x7ffa7c0ca280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 -- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa7c15ca90 msgr2=0x7ffa7c0b65e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa7c15ca90 0x7ffa7c0b65e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 -- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa7c0d16b0 msgr2=0x7ffa7c0ca280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa7c0d16b0 0x7ffa7c0ca280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 -- 172.21.15.1:0/996509422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7c1f2560 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa7affd640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c06f1b0 0x7ffa7c0b45c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c00d620 tx=0x7ffa6c00dae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa63fff640 1 -- 172.21.15.1:0/996509422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c00b620 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.684+0000 7ffa63fff640 1 -- 172.21.15.1:0/996509422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa6c019070 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.685+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa7c1f2840 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.685+0000 7ffa63fff640 1 -- 172.21.15.1:0/996509422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6c00ba00 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.685+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa7c1f2e00 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.686+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa7c050870 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.687+0000 7ffa63fff640 1 -- 172.21.15.1:0/996509422 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa6c00bba0 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.688+0000 7ffa63fff640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa680762b0 0x7ffa680787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.688+0000 7ffa63fff640 1 -- 172.21.15.1:0/996509422 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ffa6c098320 con 0x7ffa7c06f1b0 2024-05-09T05:14:46.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.688+0000 7ffa7a7fc640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa680762b0 0x7ffa680787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:46.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.689+0000 7ffa7a7fc640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa680762b0 0x7ffa680787c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c0ce840 tx=0x7ffa70009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:46.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:46.692+0000 7ffa63fff640 1 -- 172.21.15.1:0/996509422 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffa6c061ce0 con 0x7ffa7c06f1b0 2024-05-09T05:14:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.045+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffa7c0ce870 con 0x7ffa7c06f1b0 2024-05-09T05:14:47.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.046+0000 7ffa63fff640 1 -- 172.21.15.1:0/996509422 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ffa6c0615e0 con 0x7ffa7c06f1b0 2024-05-09T05:14:47.045 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa680762b0 msgr2=0x7ffa680787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa680762b0 0x7ffa680787c0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c0ce840 tx=0x7ffa70009040 comp rx=0 tx=0).stop 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c06f1b0 msgr2=0x7ffa7c0b45c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c06f1b0 0x7ffa7c0b45c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c00d620 tx=0x7ffa6c00dae0 comp rx=0 tx=0).stop 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 shutdown_connections 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa680762b0 0x7ffa680787c0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa7c0d16b0 0x7ffa7c0ca280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa7c15ca90 0x7ffa7c0b65e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:47.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 --2- 172.21.15.1:0/996509422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa7c06f1b0 0x7ffa7c0b45c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.048+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 >> 172.21.15.1:0/996509422 conn(0x7ffa7c0b3660 msgr2=0x7ffa7c0cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.049+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 shutdown_connections 2024-05-09T05:14:47.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:47.049+0000 7ffa8191f640 1 -- 172.21.15.1:0/996509422 wait complete. 2024-05-09T05:14:50.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:14:50.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.185+0000 7ff1a752d640 1 Processor -- start 2024-05-09T05:14:50.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.185+0000 7ff1a752d640 1 -- start start 2024-05-09T05:14:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.186+0000 7ff1a752d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1a015a980 0x7ff1a015ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.186+0000 7ff1a752d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1a015b2d0 con 0x7ff1a00c96a0 2024-05-09T05:14:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.186+0000 7ff1a752d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1a0074cc0 con 0x7ff1a0142970 2024-05-09T05:14:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.186+0000 7ff1a752d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff1a0159590 con 0x7ff1a015a980 2024-05-09T05:14:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.186+0000 7ff1a5aca640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1a015a980 0x7ff1a015ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.186+0000 7ff1a5aca640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1a015a980 0x7ff1a015ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43346/0 (socket says 172.21.15.1:43346) 2024-05-09T05:14:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.186+0000 7ff1a5aca640 1 -- 172.21.15.1:0/2952415315 learned_addr learned my addr 172.21.15.1:0/2952415315 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:50.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.187+0000 7ff1a5aca640 1 -- 172.21.15.1:0/2952415315 >> v1:172.21.15.196:6789/0 conn(0x7ff1a0142970 legacy=0x7ff1a0074510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:50.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.187+0000 7ff1a5aca640 1 -- 172.21.15.1:0/2952415315 >> v1:172.21.15.1:6789/0 conn(0x7ff1a00c96a0 legacy=0x7ff1a00ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:50.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.187+0000 7ff1a5aca640 1 -- 172.21.15.1:0/2952415315 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a0159870 con 0x7ff1a015a980 2024-05-09T05:14:50.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.188+0000 7ff1a5aca640 1 --2- 172.21.15.1:0/2952415315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a015a980 0x7ff1a015ad90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff198009e50 tx=0x7ff19802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe3dbf5496c5a47f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:50.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.188+0000 7ff197fff640 1 -- 172.21.15.1:0/2952415315 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff198036070 con 0x7ff1a015a980 2024-05-09T05:14:50.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.188+0000 7ff197fff640 1 -- 172.21.15.1:0/2952415315 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff198031600 con 0x7ff1a015a980 2024-05-09T05:14:50.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.188+0000 7ff197fff640 1 -- 172.21.15.1:0/2952415315 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1980319f0 con 0x7ff1a015a980 2024-05-09T05:14:50.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.189+0000 7ff1a752d640 1 -- 172.21.15.1:0/2952415315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a015a980 msgr2=0x7ff1a015ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:50.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.189+0000 7ff1a752d640 1 --2- 172.21.15.1:0/2952415315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a015a980 0x7ff1a015ad90 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7ff198009e50 tx=0x7ff19802a7c0 comp rx=0 tx=0).stop 2024-05-09T05:14:50.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.189+0000 7ff1a752d640 1 -- 172.21.15.1:0/2952415315 shutdown_connections 2024-05-09T05:14:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.189+0000 7ff1a752d640 1 --2- 172.21.15.1:0/2952415315 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a015a980 0x7ff1a015ad90 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.189+0000 7ff1a752d640 1 -- 172.21.15.1:0/2952415315 >> 172.21.15.1:0/2952415315 conn(0x7ff1a006e400 msgr2=0x7ff1a0071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.189+0000 7ff1a752d640 1 -- 172.21.15.1:0/2952415315 shutdown_connections 2024-05-09T05:14:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.189+0000 7ff1a752d640 1 -- 172.21.15.1:0/2952415315 wait complete. 2024-05-09T05:14:50.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.191+0000 7ff1a752d640 1 Processor -- start 2024-05-09T05:14:50.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.191+0000 7ff1a752d640 1 -- start start 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.192+0000 7ff1a752d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a00c96a0 0x7ff1a01f0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.192+0000 7ff1a752d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1a015a980 0x7ff1a00cb490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.192+0000 7ff1a752d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1a013f1c0 0x7ff1a0141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.192+0000 7ff1a752d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a00cefb0 con 0x7ff1a00c96a0 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.192+0000 7ff1a752d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff1a0159e30 con 0x7ff1a013f1c0 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.192+0000 7ff1a752d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff1a00cf130 con 0x7ff1a015a980 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a4ac8640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1a015a980 msgr2=0x7ff1a00cb490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a00c96a0 0x7ff1a01f0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a4ac8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1a015a980 0x7ff1a00cb490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a00c96a0 0x7ff1a01f0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43352/0 (socket says 172.21.15.1:43352) 2024-05-09T05:14:50.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 -- 172.21.15.1:0/117871658 learned_addr learned my addr 172.21.15.1:0/117871658 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a5aca640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1a013f1c0 0x7ff1a0141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 -- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1a015a980 msgr2=0x7ff1a00cb490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1a015a980 0x7ff1a00cb490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 -- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1a013f1c0 msgr2=0x7ff1a0141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1a013f1c0 0x7ff1a0141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 -- 172.21.15.1:0/117871658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1a00cb9d0 con 0x7ff1a00c96a0 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a5aca640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1a013f1c0 0x7ff1a0141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a52c9640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a00c96a0 0x7ff1a01f0ce0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00c3d0 tx=0x7ff19c00c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff195ffb640 1 -- 172.21.15.1:0/117871658 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c017070 con 0x7ff1a00c96a0 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff195ffb640 1 -- 172.21.15.1:0/117871658 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff19c0052d0 con 0x7ff1a00c96a0 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff195ffb640 1 -- 172.21.15.1:0/117871658 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff19c003830 con 0x7ff1a00c96a0 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.193+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1a0158980 con 0x7ff1a00c96a0 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.194+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1a00ca9b0 con 0x7ff1a00c96a0 2024-05-09T05:14:50.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.195+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1a0050870 con 0x7ff1a00c96a0 2024-05-09T05:14:50.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.195+0000 7ff195ffb640 1 -- 172.21.15.1:0/117871658 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff19c01a070 con 0x7ff1a00c96a0 2024-05-09T05:14:50.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.196+0000 7ff195ffb640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff184076280 0x7ff184078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:50.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.196+0000 7ff195ffb640 1 -- 172.21.15.1:0/117871658 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff19c0969a0 con 0x7ff1a00c96a0 2024-05-09T05:14:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.196+0000 7ff1a4ac8640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff184076280 0x7ff184078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.197+0000 7ff1a4ac8640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff184076280 0x7ff184078790 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff1a01ede90 tx=0x7ff190007a10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:50.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.201+0000 7ff195ffb640 1 -- 172.21.15.1:0/117871658 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff19c060280 con 0x7ff1a00c96a0 2024-05-09T05:14:50.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.532+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff1a00c9480 con 0x7ff1a00c96a0 2024-05-09T05:14:50.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.533+0000 7ff195ffb640 1 -- 172.21.15.1:0/117871658 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff19c068270 con 0x7ff1a00c96a0 2024-05-09T05:14:50.532 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":68,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:50.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.535+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff184076280 msgr2=0x7ff184078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:50.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.535+0000 7ff1a752d640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff184076280 0x7ff184078790 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7ff1a01ede90 tx=0x7ff190007a10 comp rx=0 tx=0).stop 2024-05-09T05:14:50.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.535+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a00c96a0 msgr2=0x7ff1a01f0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:50.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.535+0000 7ff1a752d640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a00c96a0 0x7ff1a01f0ce0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff19c00c3d0 tx=0x7ff19c00c890 comp rx=0 tx=0).stop 2024-05-09T05:14:50.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.535+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 shutdown_connections 2024-05-09T05:14:50.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.536+0000 7ff1a752d640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff184076280 0x7ff184078790 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:50.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.536+0000 7ff1a752d640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1a013f1c0 0x7ff1a0141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:50.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.536+0000 7ff1a752d640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1a015a980 0x7ff1a00cb490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:50.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.536+0000 7ff1a752d640 1 --2- 172.21.15.1:0/117871658 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1a00c96a0 0x7ff1a01f0ce0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:50.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.536+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 >> 172.21.15.1:0/117871658 conn(0x7ff1a006e400 msgr2=0x7ff1a00cc7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:50.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.536+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 shutdown_connections 2024-05-09T05:14:50.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:50.536+0000 7ff1a752d640 1 -- 172.21.15.1:0/117871658 wait complete. 2024-05-09T05:14:53.548 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:14:53.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.672+0000 7f8ff1b00640 1 Processor -- start 2024-05-09T05:14:53.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.672+0000 7f8ff1b00640 1 -- start start 2024-05-09T05:14:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.673+0000 7f8ff1b00640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8ff1b00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 0x7f8fec0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8ff1b00640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8ff1b00640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8fec0cd0e0 con 0x7f8fec0d16b0 2024-05-09T05:14:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8ff1b00640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8fec06f6d0 con 0x7f8fec15ca90 2024-05-09T05:14:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8ff1b00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8fec0cc4d0 con 0x7f8fec06f1b0 2024-05-09T05:14:53.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8feb7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:53.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8fe2ffd640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 msgr2=0x7f8fec0ce740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:53.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8fe2ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 0x7f8fec0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:53.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8feb7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43368/0 (socket says 172.21.15.1:43368) 2024-05-09T05:14:53.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 learned_addr learned my addr 172.21.15.1:0/3630397559 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:53.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:14:53.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:53.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:53.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:53.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:53.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:14:53.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:53.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:53.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.674+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 msgr2=0x7f8fec0ce740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 0x7f8fec0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:53.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:14:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:53.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:53.875+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:14:54.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 msgr2=0x7f8fec0ce740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 0x7f8fec0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:54.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:54.276+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:14:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 msgr2=0x7f8fec0ce740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 0x7f8fec0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:14:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:14:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:55.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:14:55.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:14:55.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:14:55.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:14:55.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:55.077+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:14:56.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:56.674+0000 7f8fe9ffb640 1 -- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 msgr2=0x7f8fec0ce740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:56.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:56.674+0000 7f8fe9ffb640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fec06f1b0 0x7f8fec0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:56.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:56.674+0000 7f8fe9ffb640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 msgr2=0x7f8fec15cea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:56.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:56.674+0000 7f8fe9ffb640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.196:3300/0 conn(0x7f8fec15ca90 0x7f8fec15cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:56.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:56.674+0000 7f8fe9ffb640 1 -- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 msgr2=0x7f8fec0d03d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:56.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:56.674+0000 7f8fe9ffb640 1 --2- 172.21.15.1:0/3630397559 >> v2:172.21.15.1:3300/0 conn(0x7f8fec0d16b0 0x7f8fec0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:56.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:56.674+0000 7f8fe9ffb640 1 -- 172.21.15.1:0/3630397559 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fec0cc7b0 con 0x7f8fcc003670 2024-05-09T05:14:56.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:56.674+0000 7f8fe9ffb640 1 -- 172.21.15.1:0/3630397559 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8fcc00c180 con 0x7f8fcc007400 2024-05-09T05:14:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.916+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1141781915 0 0) 0x7f8fcc00c180 con 0x7f8fcc007400 2024-05-09T05:14:57.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.916+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fd8003520 con 0x7f8fcc007400 2024-05-09T05:14:57.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.919+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2940911269 0 0) 0x7f8fec0cc7b0 con 0x7f8fcc003670 2024-05-09T05:14:57.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.919+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8fcc00c180 con 0x7f8fcc003670 2024-05-09T05:14:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3810085320 0 0) 0x7f8fcc00c180 con 0x7f8fcc003670 2024-05-09T05:14:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 >> v1:172.21.15.196:6789/0 conn(0x7f8fcc007400 legacy=0x7f8fcc009910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fcc00d3e0 con 0x7f8fcc003670 2024-05-09T05:14:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f8fdc00f040 con 0x7f8fcc003670 2024-05-09T05:14:57.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 >> v1:172.21.15.1:6789/0 conn(0x7f8fcc003670 legacy=0x7f8fcc005b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8febfff640 1 -- 172.21.15.1:0/3630397559 reap_dead start 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd8003b40 0x7f8fd8003f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fd8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd80099b0 0x7f8fd800bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd800c3c0 con 0x7f8fd8004800 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8fd800c570 con 0x7f8fd80099b0 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8fd800c770 con 0x7f8fd8003b40 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feb7fe640 1 -- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd8003b40 msgr2=0x7f8fd8003f70 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd8003b40 0x7f8fd8003f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fd8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:57.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.920+0000 7f8febfff640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd80099b0 0x7f8fd800bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.921+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd8003b40 msgr2=0x7f8fd8003f70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.921+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd8003b40 0x7f8fd8003f70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.921+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd80099b0 msgr2=0x7f8fd800bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:14:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.921+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd80099b0 0x7f8fd800bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.921+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/3630397559 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fcc00c180 con 0x7f8fd8004800 2024-05-09T05:14:57.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.922+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fd8004c10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4007d10 tx=0x7f8fd8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3d83f8f650c37c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:57.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.923+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4007a40 con 0x7f8fd8004800 2024-05-09T05:14:57.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.923+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd4032630 con 0x7f8fd8004800 2024-05-09T05:14:57.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.923+0000 7f8feaffd640 1 -- 172.21.15.1:0/3630397559 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd402ae60 con 0x7f8fd8004800 2024-05-09T05:14:57.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 -- 172.21.15.1:0/3630397559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 msgr2=0x7f8fd8004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:57.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fd8004c10 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4007d10 tx=0x7f8fd8005830 comp rx=0 tx=0).stop 2024-05-09T05:14:57.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 -- 172.21.15.1:0/3630397559 shutdown_connections 2024-05-09T05:14:57.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd80099b0 0x7f8fd800bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:57.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fd8004c10 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:57.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/3630397559 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd8003b40 0x7f8fd8003f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:57.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 -- 172.21.15.1:0/3630397559 >> 172.21.15.1:0/3630397559 conn(0x7f8fec0b3660 msgr2=0x7f8fec0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:57.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 -- 172.21.15.1:0/3630397559 shutdown_connections 2024-05-09T05:14:57.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.924+0000 7f8ff1b00640 1 -- 172.21.15.1:0/3630397559 wait complete. 2024-05-09T05:14:57.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.926+0000 7f8ff1b00640 1 Processor -- start 2024-05-09T05:14:57.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.926+0000 7f8ff1b00640 1 -- start start 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8ff1b00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd8003b40 0x7f8fec0c8fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8ff1b00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fec0ca540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8ff1b00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd80099b0 0x7f8fec14d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8ff1b00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8fec0d03d0 con 0x7f8fd8004800 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8ff1b00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8fec0ccd70 con 0x7f8fd8003b40 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8ff1b00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8fec0d0550 con 0x7f8fd80099b0 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8febfff640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd80099b0 msgr2=0x7f8fec14d0b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8fe2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fec0ca540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.927+0000 7f8febfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd80099b0 0x7f8fec14d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fec0ca540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43398/0 (socket says 172.21.15.1:43398) 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/2522374153 learned_addr learned my addr 172.21.15.1:0/2522374153 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd8003b40 0x7f8fec0c8fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd80099b0 msgr2=0x7f8fec14d0b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd80099b0 0x7f8fec14d0b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd8003b40 msgr2=0x7f8fec0c8fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd8003b40 0x7f8fec0c8fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe2ffd640 1 -- 172.21.15.1:0/2522374153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8fec15bd80 con 0x7f8fd8004800 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe2ffd640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fec0ca540 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4007d10 tx=0x7f8fd4005f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe8ff9640 1 -- 172.21.15.1:0/2522374153 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4043070 con 0x7f8fd8004800 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe8ff9640 1 -- 172.21.15.1:0/2522374153 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8fd4038040 con 0x7f8fd8004800 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8fe8ff9640 1 -- 172.21.15.1:0/2522374153 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8fd4007a40 con 0x7f8fd8004800 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.928+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8fec15c3e0 con 0x7f8fd8004800 2024-05-09T05:14:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.929+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8fec14ff30 con 0x7f8fd8004800 2024-05-09T05:14:57.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.930+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8fec050870 con 0x7f8fd8004800 2024-05-09T05:14:57.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.931+0000 7f8fe8ff9640 1 -- 172.21.15.1:0/2522374153 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8fd4046070 con 0x7f8fd8004800 2024-05-09T05:14:57.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.931+0000 7f8fe8ff9640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8fcc076d10 0x7f8fcc079220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:57.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.931+0000 7f8fe8ff9640 1 -- 172.21.15.1:0/2522374153 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8fd40b77b0 con 0x7f8fd8004800 2024-05-09T05:14:57.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.932+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8fcc076d10 0x7f8fcc079220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:57.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.932+0000 7f8feb7fe640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8fcc076d10 0x7f8fcc079220 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc000c00 tx=0x7f8fdc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:57.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:57.937+0000 7f8fe8ff9640 1 -- 172.21.15.1:0/2522374153 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8fd4080ad0 con 0x7f8fd8004800 2024-05-09T05:14:58.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.250+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8fec1e7590 con 0x7f8fd8004800 2024-05-09T05:14:58.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.250+0000 7f8fe8ff9640 1 -- 172.21.15.1:0/2522374153 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f8fd4084dd0 con 0x7f8fd8004800 2024-05-09T05:14:58.250 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":72,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:14:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8fcc076d10 msgr2=0x7f8fcc079220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8fcc076d10 0x7f8fcc079220 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8fdc000c00 tx=0x7f8fdc008040 comp rx=0 tx=0).stop 2024-05-09T05:14:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 msgr2=0x7f8fec0ca540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fec0ca540 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f8fd4007d10 tx=0x7f8fd4005f60 comp rx=0 tx=0).stop 2024-05-09T05:14:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 shutdown_connections 2024-05-09T05:14:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8fcc076d10 0x7f8fcc079220 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8fd80099b0 0x7f8fec14d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8fd8004800 0x7f8fec0ca540 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 --2- 172.21.15.1:0/2522374153 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8fd8003b40 0x7f8fec0c8fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 >> 172.21.15.1:0/2522374153 conn(0x7f8fec0b3660 msgr2=0x7f8fec0ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:58.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 shutdown_connections 2024-05-09T05:14:58.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.253+0000 7f8ff1b00640 1 -- 172.21.15.1:0/2522374153 wait complete. 2024-05-09T05:14:58.265 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:14:58.265 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:14:58.266 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:14:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.391+0000 7f7d734d1640 1 Processor -- start 2024-05-09T05:14:58.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.392+0000 7f7d734d1640 1 -- start start 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.393+0000 7f7d734d1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d6c0cae10 0x7f7d6c153730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.393+0000 7f7d734d1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d6c0c8a50 con 0x7f7d6c0ca970 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.393+0000 7f7d734d1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d6c0c8d10 con 0x7f7d6c0c67b0 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.393+0000 7f7d734d1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7d6c0c9d00 con 0x7f7d6c0cae10 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.393+0000 7f7d70a6c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d6c0cae10 0x7f7d6c153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.393+0000 7f7d70a6c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d6c0cae10 0x7f7d6c153730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43410/0 (socket says 172.21.15.1:43410) 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.393+0000 7f7d70a6c640 1 -- 172.21.15.1:0/2438016088 learned_addr learned my addr 172.21.15.1:0/2438016088 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.393+0000 7f7d7126d640 1 --1- 172.21.15.1:0/2438016088 >> v1:172.21.15.1:6789/0 conn(0x7f7d6c0ca970 0x7f7d6c15c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53840/0 (socket says 172.21.15.1:53840) 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.394+0000 7f7d70a6c640 1 -- 172.21.15.1:0/2438016088 >> v1:172.21.15.196:6789/0 conn(0x7f7d6c0c67b0 legacy=0x7f7d6c163d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.394+0000 7f7d70a6c640 1 -- 172.21.15.1:0/2438016088 >> v1:172.21.15.1:6789/0 conn(0x7f7d6c0ca970 legacy=0x7f7d6c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.394+0000 7f7d70a6c640 1 -- 172.21.15.1:0/2438016088 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c0c5e10 con 0x7f7d6c0cae10 2024-05-09T05:14:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.395+0000 7f7d70a6c640 1 --2- 172.21.15.1:0/2438016088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0cae10 0x7f7d6c153730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009870 tx=0x7f7d5c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16a148aa92ce99a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.395+0000 7f7d63fff640 1 -- 172.21.15.1:0/2438016088 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c037070 con 0x7f7d6c0cae10 2024-05-09T05:14:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.395+0000 7f7d63fff640 1 -- 172.21.15.1:0/2438016088 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5c02ae50 con 0x7f7d6c0cae10 2024-05-09T05:14:58.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.395+0000 7f7d63fff640 1 -- 172.21.15.1:0/2438016088 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c032860 con 0x7f7d6c0cae10 2024-05-09T05:14:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.396+0000 7f7d734d1640 1 -- 172.21.15.1:0/2438016088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0cae10 msgr2=0x7f7d6c153730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.396+0000 7f7d734d1640 1 --2- 172.21.15.1:0/2438016088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0cae10 0x7f7d6c153730 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009870 tx=0x7f7d5c02a300 comp rx=0 tx=0).stop 2024-05-09T05:14:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.396+0000 7f7d734d1640 1 -- 172.21.15.1:0/2438016088 shutdown_connections 2024-05-09T05:14:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.396+0000 7f7d734d1640 1 --2- 172.21.15.1:0/2438016088 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0cae10 0x7f7d6c153730 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.396+0000 7f7d734d1640 1 -- 172.21.15.1:0/2438016088 >> 172.21.15.1:0/2438016088 conn(0x7f7d6c0ce910 msgr2=0x7f7d6c0ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.396+0000 7f7d734d1640 1 -- 172.21.15.1:0/2438016088 shutdown_connections 2024-05-09T05:14:58.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.396+0000 7f7d734d1640 1 -- 172.21.15.1:0/2438016088 wait complete. 2024-05-09T05:14:58.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.397+0000 7f7d734d1640 1 Processor -- start 2024-05-09T05:14:58.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.398+0000 7f7d734d1640 1 -- start start 2024-05-09T05:14:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d734d1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d6c0c67b0 0x7f7d6c1f0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d734d1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0ca970 0x7f7d6c14ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d734d1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d6c0cae10 0x7f7d6c146da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d734d1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c0c63d0 con 0x7f7d6c0ca970 2024-05-09T05:14:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d734d1640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c0c9870 con 0x7f7d6c0cae10 2024-05-09T05:14:58.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d734d1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d6c0c9500 con 0x7f7d6c0c67b0 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d7126d640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d6c0c67b0 msgr2=0x7f7d6c1f0ca0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d70a6c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0ca970 0x7f7d6c14ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d7126d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d6c0c67b0 0x7f7d6c1f0ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d70a6c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0ca970 0x7f7d6c14ba20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43418/0 (socket says 172.21.15.1:43418) 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d70a6c640 1 -- 172.21.15.1:0/4249173264 learned_addr learned my addr 172.21.15.1:0/4249173264 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d71a6e640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d6c0cae10 0x7f7d6c146da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d70a6c640 1 -- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d6c0c67b0 msgr2=0x7f7d6c1f0ca0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d70a6c640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d6c0c67b0 0x7f7d6c1f0ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d70a6c640 1 -- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d6c0cae10 msgr2=0x7f7d6c146da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d70a6c640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d6c0cae10 0x7f7d6c146da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.399+0000 7f7d70a6c640 1 -- 172.21.15.1:0/4249173264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d6c14eb90 con 0x7f7d6c0ca970 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.400+0000 7f7d70a6c640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0ca970 0x7f7d6c14ba20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009870 tx=0x7f7d5c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.400+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c041070 con 0x7f7d6c0ca970 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.400+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d5c03c870 con 0x7f7d6c0ca970 2024-05-09T05:14:58.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.400+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d5c037040 con 0x7f7d6c0ca970 2024-05-09T05:14:58.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.400+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d6c1f41b0 con 0x7f7d6c0ca970 2024-05-09T05:14:58.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.400+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d6c1f4770 con 0x7f7d6c0ca970 2024-05-09T05:14:58.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.401+0000 7f7d734d1640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c1472e0 0x7f7d6c1f4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.401+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c1f1670 con 0x7f7d6c1472e0 2024-05-09T05:14:58.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.401+0000 7f7d7126d640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c1472e0 0x7f7d6c1f4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.402+0000 7f7d7126d640 1 -- 172.21.15.1:0/4249173264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d6c1f11e0 con 0x7f7d6c1472e0 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.402+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7d5c044070 con 0x7f7d6c0ca970 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.402+0000 7f7d7126d640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c1472e0 0x7f7d6c1f4940 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d68001080 tx=0x7f7d6800c810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.402+0000 7f7d61ffb640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d50076280 0x7f7d50078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.403+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7d5c0b7ff0 con 0x7f7d6c0ca970 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.403+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680074e0 con 0x7f7d6c1472e0 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.403+0000 7f7d71a6e640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d50076280 0x7f7d50078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.403+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7d68007680 con 0x7f7d6c1472e0 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.403+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c1472e0 msgr2=0x7f7d6c1f4940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.403+0000 7f7d61ffb640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c1472e0 0x7f7d6c1f4940 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d68001080 tx=0x7f7d6800c810 comp rx=0 tx=0).stop 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.403+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c1472e0 msgr2=0x7f7d6c1f4940 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:58.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.403+0000 7f7d61ffb640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c1472e0 0x7f7d6c1f4940 secure :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f7d68001080 tx=0x7f7d6800c810 comp rx=0 tx=0).stop 2024-05-09T05:14:58.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.404+0000 7f7d71a6e640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d50076280 0x7f7d50078790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7d640012e0 tx=0x7f7d64009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.410+0000 7f7d734d1640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c14c690 0x7f7d6c14caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.410+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d6c14cfe0 con 0x7f7d6c14c690 2024-05-09T05:14:58.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.410+0000 7f7d7126d640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c14c690 0x7f7d6c14caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.411+0000 7f7d7126d640 1 -- 172.21.15.1:0/4249173264 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7d6c1f4e80 con 0x7f7d6c14c690 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.411+0000 7f7d7126d640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c14c690 0x7f7d6c14caa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7d68001080 tx=0x7f7d680010b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.411+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d680074e0 con 0x7f7d6c14c690 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.411+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5052 (secure 0 0 0) 0x7f7d68019880 con 0x7f7d6c14c690 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.411+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c14c690 msgr2=0x7f7d6c14caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.411+0000 7f7d61ffb640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c14c690 0x7f7d6c14caa0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f7d68001080 tx=0x7f7d680010b0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.411+0000 7f7d61ffb640 1 -- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c14c690 msgr2=0x7f7d6c14caa0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.411+0000 7f7d61ffb640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c14c690 0x7f7d6c14caa0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:58.412 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 72, 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 100606, 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:58.413 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:58.414 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:58.415 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:58.416 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.417 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:58.418 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:58.419 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:58.420 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:58.421 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:58.422 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:14:58.423 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:14:58.424 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.425 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:58.425 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d50076280 msgr2=0x7f7d50078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d734d1640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d50076280 0x7f7d50078790 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f7d640012e0 tx=0x7f7d64009040 comp rx=0 tx=0).stop 2024-05-09T05:14:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0ca970 msgr2=0x7f7d6c14ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d734d1640 1 --2- 172.21.15.1:0/4249173264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d6c0ca970 0x7f7d6c14ba20 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c009870 tx=0x7f7d5c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d71a6e640 1 -- 172.21.15.1:0/4249173264 reap_dead start 2024-05-09T05:14:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d71a6e640 1 -- 172.21.15.1:0/4249173264 reap_dead start 2024-05-09T05:14:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 shutdown_connections 2024-05-09T05:14:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 >> 172.21.15.1:0/4249173264 conn(0x7f7d6c0ce910 msgr2=0x7f7d6c151980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 shutdown_connections 2024-05-09T05:14:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.412+0000 7f7d734d1640 1 -- 172.21.15.1:0/4249173264 wait complete. 2024-05-09T05:14:58.427 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:14:58.427 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:14:58.427 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:14:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.555+0000 7f5723346640 1 Processor -- start 2024-05-09T05:14:58.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.555+0000 7f5723346640 1 -- start start 2024-05-09T05:14:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.556+0000 7f5723346640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 0x7f571c0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.556+0000 7f5723346640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f571c071be0 0x7f571c0c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.556+0000 7f5723346640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f571c156dd0 con 0x7f571c15d130 2024-05-09T05:14:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.556+0000 7f5723346640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f571c15d540 con 0x7f571c071be0 2024-05-09T05:14:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.556+0000 7f5723346640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f571c0c3ce0 con 0x7f571c1569c0 2024-05-09T05:14:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57218e3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f571c071be0 0x7f571c0c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.557 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57208e1640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 msgr2=0x7f571c0716a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57218e3640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f571c071be0 0x7f571c0c48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43434/0 (socket says 172.21.15.1:43434) 2024-05-09T05:14:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57218e3640 1 -- 172.21.15.1:0/4293388754 learned_addr learned my addr 172.21.15.1:0/4293388754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57208e1640 1 --2- 172.21.15.1:0/4293388754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 0x7f571c0716a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57218e3640 1 -- 172.21.15.1:0/4293388754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 msgr2=0x7f571c0716a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:14:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57218e3640 1 --2- 172.21.15.1:0/4293388754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 0x7f571c0716a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57218e3640 1 -- 172.21.15.1:0/4293388754 >> v1:172.21.15.1:6789/0 conn(0x7f571c15d130 legacy=0x7f571c06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.558 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.557+0000 7f57218e3640 1 -- 172.21.15.1:0/4293388754 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571c0c3fc0 con 0x7f571c071be0 2024-05-09T05:14:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.558+0000 7f57218e3640 1 --2- 172.21.15.1:0/4293388754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c0c48c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5714009e50 tx=0x7f571402a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=781a9abed4e23ba7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.558+0000 7f5713fff640 1 -- 172.21.15.1:0/4293388754 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5714036070 con 0x7f571c071be0 2024-05-09T05:14:58.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.558+0000 7f5713fff640 1 -- 172.21.15.1:0/4293388754 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5714031600 con 0x7f571c071be0 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.558+0000 7f5713fff640 1 -- 172.21.15.1:0/4293388754 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57140319d0 con 0x7f571c071be0 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.559+0000 7f5723346640 1 -- 172.21.15.1:0/4293388754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 msgr2=0x7f571c0c48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.559+0000 7f5723346640 1 --2- 172.21.15.1:0/4293388754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c0c48c0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5714009e50 tx=0x7f571402a730 comp rx=0 tx=0).stop 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.559+0000 7f5723346640 1 -- 172.21.15.1:0/4293388754 shutdown_connections 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.559+0000 7f5723346640 1 --2- 172.21.15.1:0/4293388754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 0x7f571c0716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.559+0000 7f5723346640 1 --2- 172.21.15.1:0/4293388754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c0c48c0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.559+0000 7f5723346640 1 -- 172.21.15.1:0/4293388754 >> 172.21.15.1:0/4293388754 conn(0x7f571c0747b0 msgr2=0x7f571c0ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.559+0000 7f5723346640 1 -- 172.21.15.1:0/4293388754 shutdown_connections 2024-05-09T05:14:58.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.559+0000 7f5723346640 1 -- 172.21.15.1:0/4293388754 wait complete. 2024-05-09T05:14:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.561+0000 7f5723346640 1 Processor -- start 2024-05-09T05:14:58.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.561+0000 7f5723346640 1 -- start start 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f5723346640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c1e5b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f5723346640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 0x7f571c1f2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f5723346640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c15d130 0x7f571c15a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f5723346640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f571c06e1a0 con 0x7f571c071be0 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f5723346640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f571c06e780 con 0x7f571c15d130 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f5723346640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f571c0cac90 con 0x7f571c1569c0 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f57210e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c1e5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f57208e1640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 msgr2=0x7f571c1f2980 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.562+0000 7f57210e2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c1e5b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43436/0 (socket says 172.21.15.1:43436) 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57210e2640 1 -- 172.21.15.1:0/3618546669 learned_addr learned my addr 172.21.15.1:0/3618546669 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57208e1640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 0x7f571c1f2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57218e3640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c15d130 0x7f571c15a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57210e2640 1 -- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 msgr2=0x7f571c1f2980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57210e2640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f571c1569c0 0x7f571c1f2980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57210e2640 1 -- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c15d130 msgr2=0x7f571c15a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57210e2640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c15d130 0x7f571c15a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57210e2640 1 -- 172.21.15.1:0/3618546669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f571c1f2ec0 con 0x7f571c071be0 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f57210e2640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c1e5b30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f571800c3d0 tx=0x7f571800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718017070 con 0x7f571c071be0 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57180052d0 con 0x7f571c071be0 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5718003830 con 0x7f571c071be0 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.563+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f571c158640 con 0x7f571c071be0 2024-05-09T05:14:58.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.564+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f571c158f00 con 0x7f571c071be0 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.564+0000 7f5723346640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c1e7a40 0x7f571c159140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.564+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f571c1f3390 con 0x7f571c1e7a40 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.565+0000 7f57208e1640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c1e7a40 0x7f571c159140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.565+0000 7f57208e1640 1 -- 172.21.15.1:0/3618546669 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f571c050870 con 0x7f571c1e7a40 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.565+0000 7f57208e1640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c1e7a40 0x7f571c159140 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f571c1e8b30 tx=0x7f570c00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.565+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c003e10 con 0x7f571c1e7a40 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f571801a070 con 0x7f571c071be0 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f5711ffb640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5700076200 0x7f5700078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f57180969e0 con 0x7f571c071be0 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f570c015070 con 0x7f571c1e7a40 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c1e7a40 msgr2=0x7f571c159140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f5711ffb640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c1e7a40 0x7f571c159140 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f571c1e8b30 tx=0x7f570c00b8c0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f57218e3640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5700076200 0x7f5700078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c1e7a40 msgr2=0x7f571c159140 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.566+0000 7f5711ffb640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c1e7a40 0x7f571c159140 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.568 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.567+0000 7f57218e3640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5700076200 0x7f5700078710 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5714033000 tx=0x7f5714009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.577+0000 7f5723346640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c14e1a0 0x7f571c14e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:14:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.577+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f571c156ff0 con 0x7f571c14e1a0 2024-05-09T05:14:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.577+0000 7f57208e1640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c14e1a0 0x7f571c14e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:14:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.578+0000 7f57208e1640 1 -- 172.21.15.1:0/3618546669 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f571c15a120 con 0x7f571c14e1a0 2024-05-09T05:14:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.578+0000 7f57208e1640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c14e1a0 0x7f571c14e5b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f570c014000 tx=0x7f570c005a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:14:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.578+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570c023410 con 0x7f571c14e1a0 2024-05-09T05:14:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.578+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f570c010400 con 0x7f571c14e1a0 2024-05-09T05:14:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.578+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c14e1a0 msgr2=0x7f571c14e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.578+0000 7f5711ffb640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c14e1a0 0x7f571c14e5b0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f570c014000 tx=0x7f570c005a20 comp rx=0 tx=0).stop 2024-05-09T05:14:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.578+0000 7f5711ffb640 1 -- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c14e1a0 msgr2=0x7f571c14e5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:14:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.578+0000 7f5711ffb640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f571c14e1a0 0x7f571c14e5b0 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f570c014000 tx=0x7f570c005a20 comp rx=0 tx=0).stop 2024-05-09T05:14:58.579 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:14:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:58.579 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 72, 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 160857, 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:58.580 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:58.581 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:14:58.582 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:14:58.583 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:14:58.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:14:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:14:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:14:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:14:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:14:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:14:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:14:58.592 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:14:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5700076200 msgr2=0x7f5700078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5700076200 0x7f5700078710 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5714033000 tx=0x7f5714009000 comp rx=0 tx=0).stop 2024-05-09T05:14:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 msgr2=0x7f571c1e5b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:14:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c1e5b30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f571800c3d0 tx=0x7f571800c890 comp rx=0 tx=0).stop 2024-05-09T05:14:58.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f57218e3640 1 -- 172.21.15.1:0/3618546669 reap_dead start 2024-05-09T05:14:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 shutdown_connections 2024-05-09T05:14:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 --2- 172.21.15.1:0/3618546669 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f571c071be0 0x7f571c1e5b30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:14:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 >> 172.21.15.1:0/3618546669 conn(0x7f571c0747b0 msgr2=0x7f571c0c52f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:14:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 shutdown_connections 2024-05-09T05:14:58.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:14:58.579+0000 7f5723346640 1 -- 172.21.15.1:0/3618546669 wait complete. 2024-05-09T05:14:58.594 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:14:58.594 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:15:03.794 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:15:03.797 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:15:18.595 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:15:18.596 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-05-09T05:15:18.596 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T05:15:18.596 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T05:15:18.599 INFO:tasks.ceph.mon.c:Started 2024-05-09T05:15:18.599 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:15:18.600 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:15:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.718+0000 7f168090d640 1 Processor -- start 2024-05-09T05:15:18.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.718+0000 7f168090d640 1 -- start start 2024-05-09T05:15:18.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f168090d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f167c06f1b0 0x7f167c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:18.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f168090d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 0x7f167c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:18.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f168090d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f167c06f6d0 con 0x7f167c0d16b0 2024-05-09T05:15:18.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f168090d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f167c0cd0e0 con 0x7f167c06f1b0 2024-05-09T05:15:18.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f168090d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f167c0cc4d0 con 0x7f167c15ca90 2024-05-09T05:15:18.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f1673fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f167c06f1b0 0x7f167c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:18.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f167ad9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 0x7f167c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f167a59c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f167c0d16b0 0x7f167c0d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:58852/0 (socket says 172.21.15.1:58852) 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.719+0000 7f167a59c640 1 -- 172.21.15.1:0/2576879091 learned_addr learned my addr 172.21.15.1:0/2576879091 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.720+0000 7f167ad9d640 1 -- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 msgr2=0x7f167c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.720+0000 7f167ad9d640 1 -- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 msgr2=0x7f167c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.720+0000 7f167ad9d640 1 --2- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 0x7f167c15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.720+0000 7f167ad9d640 1 --2- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 0x7f167c15cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.720+0000 7f1673fff640 1 -- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 msgr2=0x7f167c15cea0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.720+0000 7f1673fff640 1 --2- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 0x7f167c15cea0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.720+0000 7f1673fff640 1 -- 172.21.15.1:0/2576879091 >> v1:172.21.15.196:6789/0 conn(0x7f167c0d16b0 legacy=0x7f167c0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:18.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.720+0000 7f1673fff640 1 -- 172.21.15.1:0/2576879091 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c0cc7b0 con 0x7f167c06f1b0 2024-05-09T05:15:18.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.721+0000 7f1673fff640 1 --2- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 0x7f167c0ce740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f16640097a0 tx=0x7f166402a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaf7c15b339231dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:18.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.721+0000 7f1679d9b640 1 -- 172.21.15.1:0/2576879091 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1664037070 con 0x7f167c06f1b0 2024-05-09T05:15:18.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.721+0000 7f1679d9b640 1 -- 172.21.15.1:0/2576879091 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1664032440 con 0x7f167c06f1b0 2024-05-09T05:15:18.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.721+0000 7f1679d9b640 1 -- 172.21.15.1:0/2576879091 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1664032810 con 0x7f167c06f1b0 2024-05-09T05:15:18.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.721+0000 7f168090d640 1 -- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 msgr2=0x7f167c0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:18.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.721+0000 7f168090d640 1 --2- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 0x7f167c0ce740 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f16640097a0 tx=0x7f166402a210 comp rx=0 tx=0).stop 2024-05-09T05:15:18.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.722+0000 7f168090d640 1 -- 172.21.15.1:0/2576879091 shutdown_connections 2024-05-09T05:15:18.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.722+0000 7f168090d640 1 --2- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c15ca90 0x7f167c15cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:18.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.722+0000 7f168090d640 1 --2- 172.21.15.1:0/2576879091 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 0x7f167c0ce740 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:18.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.722+0000 7f168090d640 1 -- 172.21.15.1:0/2576879091 >> 172.21.15.1:0/2576879091 conn(0x7f167c0b3660 msgr2=0x7f167c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:18.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.722+0000 7f168090d640 1 -- 172.21.15.1:0/2576879091 shutdown_connections 2024-05-09T05:15:18.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.722+0000 7f168090d640 1 -- 172.21.15.1:0/2576879091 wait complete. 2024-05-09T05:15:18.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.724+0000 7f168090d640 1 Processor -- start 2024-05-09T05:15:18.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.724+0000 7f168090d640 1 -- start start 2024-05-09T05:15:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f168090d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 0x7f167c1f3b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f168090d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c0d16b0 0x7f167c159fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f168090d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f167c15ca90 0x7f167c151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f168090d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f167c15d3e0 con 0x7f167c15ca90 2024-05-09T05:15:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f168090d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f167c0ccd70 con 0x7f167c06f1b0 2024-05-09T05:15:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f168090d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f167c0cec80 con 0x7f167c0d16b0 2024-05-09T05:15:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f167c15ca90 0x7f167c151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:18.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167ad9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f167c15ca90 0x7f167c151e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50166/0 (socket says 172.21.15.1:50166) 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167ad9d640 1 -- 172.21.15.1:0/3319759296 learned_addr learned my addr 172.21.15.1:0/3319759296 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167a59c640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 0x7f167c1f3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f1673fff640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c0d16b0 0x7f167c159fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167ad9d640 1 -- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c0d16b0 msgr2=0x7f167c159fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167ad9d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c0d16b0 0x7f167c159fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167ad9d640 1 -- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 msgr2=0x7f167c1f3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167ad9d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 0x7f167c1f3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.725+0000 7f167ad9d640 1 -- 172.21.15.1:0/3319759296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f167c157780 con 0x7f167c15ca90 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.726+0000 7f167ad9d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f167c15ca90 0x7f167c151e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1674007e30 tx=0x7f167400e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.726+0000 7f1672ffd640 1 -- 172.21.15.1:0/3319759296 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1674014420 con 0x7f167c15ca90 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.726+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f167c157a60 con 0x7f167c15ca90 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.726+0000 7f1672ffd640 1 -- 172.21.15.1:0/3319759296 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1674019070 con 0x7f167c15ca90 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.726+0000 7f1672ffd640 1 -- 172.21.15.1:0/3319759296 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1674014820 con 0x7f167c15ca90 2024-05-09T05:15:18.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.726+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f167c158020 con 0x7f167c15ca90 2024-05-09T05:15:18.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.727+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f167c050870 con 0x7f167c15ca90 2024-05-09T05:15:18.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.728+0000 7f1672ffd640 1 -- 172.21.15.1:0/3319759296 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f16740149c0 con 0x7f167c15ca90 2024-05-09T05:15:18.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.728+0000 7f1672ffd640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f165c076300 0x7f165c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:18.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.728+0000 7f1672ffd640 1 -- 172.21.15.1:0/3319759296 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1674097d50 con 0x7f167c15ca90 2024-05-09T05:15:18.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.729+0000 7f167a59c640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f165c076300 0x7f165c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:18.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.729+0000 7f167a59c640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f165c076300 0x7f165c078810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f166c000f60 tx=0x7f166c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:18.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:18.735+0000 7f1672ffd640 1 -- 172.21.15.1:0/3319759296 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f167409b030 con 0x7f167c15ca90 2024-05-09T05:15:19.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.095+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f167c15d1c0 con 0x7f167c15ca90 2024-05-09T05:15:19.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.095+0000 7f1672ffd640 1 -- 172.21.15.1:0/3319759296 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f1674061630 con 0x7f167c15ca90 2024-05-09T05:15:19.094 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":72,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:15:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f165c076300 msgr2=0x7f165c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f165c076300 0x7f165c078810 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f166c000f60 tx=0x7f166c009040 comp rx=0 tx=0).stop 2024-05-09T05:15:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f167c15ca90 msgr2=0x7f167c151e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f167c15ca90 0x7f167c151e60 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1674007e30 tx=0x7f167400e4b0 comp rx=0 tx=0).stop 2024-05-09T05:15:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 shutdown_connections 2024-05-09T05:15:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f165c076300 0x7f165c078810 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f167c15ca90 0x7f167c151e60 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:19.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f167c0d16b0 0x7f167c159fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 --2- 172.21.15.1:0/3319759296 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f167c06f1b0 0x7f167c1f3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.097+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 >> 172.21.15.1:0/3319759296 conn(0x7f167c0b3660 msgr2=0x7f167c154d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.098+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 shutdown_connections 2024-05-09T05:15:19.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:19.098+0000 7f168090d640 1 -- 172.21.15.1:0/3319759296 wait complete. 2024-05-09T05:15:22.112 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:15:22.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.236+0000 7f9a9f166640 1 Processor -- start 2024-05-09T05:15:22.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.236+0000 7f9a9f166640 1 -- start start 2024-05-09T05:15:22.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.237+0000 7f9a9f166640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 0x7f9a98156a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.237+0000 7f9a9f166640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f9a980c9490 0x7f9a980c98a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.237+0000 7f9a9f166640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9a9815c800 con 0x7f9a980716a0 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.237+0000 7f9a9f166640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f9a98156fb0 con 0x7f9a980c9490 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.237+0000 7f9a9f166640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9a980caa80 con 0x7f9a9806c6f0 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.237+0000 7f9a9d703640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9a980716a0 0x7f9a9815fdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54510/0 (socket says 172.21.15.1:54510) 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a9d703640 1 -- 172.21.15.1:0/3360144422 learned_addr learned my addr 172.21.15.1:0/3360144422 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a97fff640 1 --2- 172.21.15.1:0/3360144422 >> v2:172.21.15.196:3300/0 conn(0x7f9a980c9490 0x7f9a980c98a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a9cf02640 1 --2- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 0x7f9a98156a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a977fe640 1 -- 172.21.15.1:0/3360144422 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2335163847 0 0) 0x7f9a9815c800 con 0x7f9a980716a0 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a977fe640 1 -- 172.21.15.1:0/3360144422 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9a88003520 con 0x7f9a980716a0 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a9cf02640 1 -- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 msgr2=0x7f9a98156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a9cf02640 1 -- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 msgr2=0x7f9a98156a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:22.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a9cf02640 1 --2- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 0x7f9a98156a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:22.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a9cf02640 1 --2- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 0x7f9a98156a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:22.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a97fff640 1 -- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 msgr2=0x7f9a98156a70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:15:22.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a97fff640 1 --2- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 0x7f9a98156a70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a97fff640 1 -- 172.21.15.1:0/3360144422 >> v1:172.21.15.1:6789/0 conn(0x7f9a980716a0 legacy=0x7f9a9815fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:22.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a977fe640 1 -- 172.21.15.1:0/3360144422 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3940873905 0 0) 0x7f9a88003520 con 0x7f9a980716a0 2024-05-09T05:15:22.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.238+0000 7f9a97fff640 1 -- 172.21.15.1:0/3360144422 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9815cac0 con 0x7f9a980c9490 2024-05-09T05:15:22.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.239+0000 7f9a97fff640 1 --2- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a980c9490 0x7f9a980c98a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a840097a0 tx=0x7f9a8402a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e395b047629fa3b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:22.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.239+0000 7f9a977fe640 1 -- 172.21.15.1:0/3360144422 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84037070 con 0x7f9a980c9490 2024-05-09T05:15:22.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a977fe640 1 -- 172.21.15.1:0/3360144422 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a8402aba0 con 0x7f9a980c9490 2024-05-09T05:15:22.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a977fe640 1 -- 172.21.15.1:0/3360144422 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a90001a70 con 0x7f9a980c9490 2024-05-09T05:15:22.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a9f166640 1 -- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a980c9490 msgr2=0x7f9a980c98a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:22.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a9f166640 1 --2- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a980c9490 0x7f9a980c98a0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9a840097a0 tx=0x7f9a8402a110 comp rx=0 tx=0).stop 2024-05-09T05:15:22.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a9f166640 1 -- 172.21.15.1:0/3360144422 shutdown_connections 2024-05-09T05:15:22.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a9f166640 1 --2- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a980c9490 0x7f9a980c98a0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a9f166640 1 --2- 172.21.15.1:0/3360144422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a9806c6f0 0x7f9a98156a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a9f166640 1 -- 172.21.15.1:0/3360144422 >> 172.21.15.1:0/3360144422 conn(0x7f9a980cea90 msgr2=0x7f9a980ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:22.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a9f166640 1 -- 172.21.15.1:0/3360144422 shutdown_connections 2024-05-09T05:15:22.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.240+0000 7f9a9f166640 1 -- 172.21.15.1:0/3360144422 wait complete. 2024-05-09T05:15:22.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.242+0000 7f9a9f166640 1 Processor -- start 2024-05-09T05:15:22.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.242+0000 7f9a9f166640 1 -- start start 2024-05-09T05:15:22.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9f166640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a9806c6f0 0x7f9a98157950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:22.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9f166640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a980716a0 0x7f9a98155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:22.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9f166640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 0x7f9a98145880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:22.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9f166640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9815d360 con 0x7f9a980716a0 2024-05-09T05:15:22.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9f166640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9a9815d080 con 0x7f9a9806c6f0 2024-05-09T05:15:22.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9f166640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9a980cae60 con 0x7f9a980c9490 2024-05-09T05:15:22.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a980716a0 0x7f9a98155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a97fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a980716a0 0x7f9a98155890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53810/0 (socket says 172.21.15.1:53810) 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a97fff640 1 -- 172.21.15.1:0/1159404580 learned_addr learned my addr 172.21.15.1:0/1159404580 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9cf02640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a9806c6f0 0x7f9a98157950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9d703640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 0x7f9a98145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9d703640 1 -- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 msgr2=0x7f9a98145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9d703640 1 -- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 msgr2=0x7f9a98145880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9d703640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 0x7f9a98145880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a9d703640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 0x7f9a98145880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a97fff640 1 -- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 msgr2=0x7f9a98145880 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a97fff640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 0x7f9a98145880 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a97fff640 1 -- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a9806c6f0 msgr2=0x7f9a98157950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a97fff640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a9806c6f0 0x7f9a98157950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.244+0000 7f9a97fff640 1 -- 172.21.15.1:0/1159404580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9a9815a070 con 0x7f9a980716a0 2024-05-09T05:15:22.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.245+0000 7f9a97fff640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a980716a0 0x7f9a98155890 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9a840098c0 tx=0x7f9a840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.245+0000 7f9a957fa640 1 -- 172.21.15.1:0/1159404580 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84042020 con 0x7f9a980716a0 2024-05-09T05:15:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.245+0000 7f9a957fa640 1 -- 172.21.15.1:0/1159404580 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9a84003a40 con 0x7f9a980716a0 2024-05-09T05:15:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.245+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9a98157e90 con 0x7f9a980716a0 2024-05-09T05:15:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.245+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9a98159ab0 con 0x7f9a980716a0 2024-05-09T05:15:22.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.245+0000 7f9a957fa640 1 -- 172.21.15.1:0/1159404580 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9a84037040 con 0x7f9a980716a0 2024-05-09T05:15:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.246+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9a98050870 con 0x7f9a980716a0 2024-05-09T05:15:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.247+0000 7f9a957fa640 1 -- 172.21.15.1:0/1159404580 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9a84045020 con 0x7f9a980716a0 2024-05-09T05:15:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.247+0000 7f9a957fa640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a7c0762d0 0x7f9a7c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.247+0000 7f9a957fa640 1 -- 172.21.15.1:0/1159404580 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9a840b7c90 con 0x7f9a980716a0 2024-05-09T05:15:22.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.248+0000 7f9a9cf02640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a7c0762d0 0x7f9a7c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:22.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.248+0000 7f9a9cf02640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a7c0762d0 0x7f9a7c0787e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c006fd0 tx=0x7f9a8c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:22.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.253+0000 7f9a957fa640 1 -- 172.21.15.1:0/1159404580 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9a8404c030 con 0x7f9a980716a0 2024-05-09T05:15:22.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.538+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9a980b4ad0 con 0x7f9a980716a0 2024-05-09T05:15:22.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.539+0000 7f9a957fa640 1 -- 172.21.15.1:0/1159404580 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f9a84081580 con 0x7f9a980716a0 2024-05-09T05:15:22.538 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":72,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:15:22.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a7c0762d0 msgr2=0x7f9a7c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a7c0762d0 0x7f9a7c0787e0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f9a8c006fd0 tx=0x7f9a8c008040 comp rx=0 tx=0).stop 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a980716a0 msgr2=0x7f9a98155890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a980716a0 0x7f9a98155890 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f9a840098c0 tx=0x7f9a840046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 shutdown_connections 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9a7c0762d0 0x7f9a7c0787e0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9a980c9490 0x7f9a98145880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9a980716a0 0x7f9a98155890 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 --2- 172.21.15.1:0/1159404580 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9a9806c6f0 0x7f9a98157950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 >> 172.21.15.1:0/1159404580 conn(0x7f9a980cea90 msgr2=0x7f9a980ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.541+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 shutdown_connections 2024-05-09T05:15:22.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:22.542+0000 7f9a9f166640 1 -- 172.21.15.1:0/1159404580 wait complete. 2024-05-09T05:15:25.553 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:15:25.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.681+0000 7f773359e640 1 Processor -- start 2024-05-09T05:15:25.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.682+0000 7f773359e640 1 -- start start 2024-05-09T05:15:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.683+0000 7f773359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f773415c790 0x7f77341605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.683+0000 7f773359e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77340c8450 con 0x7f77340705e0 2024-05-09T05:15:25.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.683+0000 7f773359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f77340c8730 con 0x7f77340c5a50 2024-05-09T05:15:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.683+0000 7f773359e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f773406fe50 con 0x7f773415c790 2024-05-09T05:15:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.683+0000 7f773259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f773415c790 0x7f77341605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.683+0000 7f773259c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f773415c790 0x7f77341605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53822/0 (socket says 172.21.15.1:53822) 2024-05-09T05:15:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.683+0000 7f773259c640 1 -- 172.21.15.1:0/3964307425 learned_addr learned my addr 172.21.15.1:0/3964307425 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.684+0000 7f773259c640 1 -- 172.21.15.1:0/3964307425 >> v1:172.21.15.196:6789/0 conn(0x7f77340c5a50 legacy=0x7f7734167030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.684+0000 7f773259c640 1 -- 172.21.15.1:0/3964307425 >> v1:172.21.15.1:6789/0 conn(0x7f77340705e0 legacy=0x7f77340c72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:25.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.684+0000 7f773259c640 1 -- 172.21.15.1:0/3964307425 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f773415fda0 con 0x7f773415c790 2024-05-09T05:15:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.684+0000 7f773259c640 1 --2- 172.21.15.1:0/3964307425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773415c790 0x7f77341605a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f773406ffd0 tx=0x7f772402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c94084b631bcda server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.685+0000 7f773159a640 1 -- 172.21.15.1:0/3964307425 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724037070 con 0x7f773415c790 2024-05-09T05:15:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.685+0000 7f773159a640 1 -- 172.21.15.1:0/3964307425 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7724032440 con 0x7f773415c790 2024-05-09T05:15:25.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.685+0000 7f773159a640 1 -- 172.21.15.1:0/3964307425 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7724032810 con 0x7f773415c790 2024-05-09T05:15:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.686+0000 7f773359e640 1 -- 172.21.15.1:0/3964307425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773415c790 msgr2=0x7f77341605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.686+0000 7f773359e640 1 --2- 172.21.15.1:0/3964307425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773415c790 0x7f77341605a0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f773406ffd0 tx=0x7f772402a250 comp rx=0 tx=0).stop 2024-05-09T05:15:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.686+0000 7f773359e640 1 -- 172.21.15.1:0/3964307425 shutdown_connections 2024-05-09T05:15:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.686+0000 7f773359e640 1 --2- 172.21.15.1:0/3964307425 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f773415c790 0x7f77341605a0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.686+0000 7f773359e640 1 -- 172.21.15.1:0/3964307425 >> 172.21.15.1:0/3964307425 conn(0x7f7734073b70 msgr2=0x7f7734073f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.686+0000 7f773359e640 1 -- 172.21.15.1:0/3964307425 shutdown_connections 2024-05-09T05:15:25.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.686+0000 7f773359e640 1 -- 172.21.15.1:0/3964307425 wait complete. 2024-05-09T05:15:25.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.688+0000 7f773359e640 1 Processor -- start 2024-05-09T05:15:25.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.688+0000 7f773359e640 1 -- start start 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f773359e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77340705e0 0x7f773415a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f773359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340c5a50 0x7f77341f2c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f773359e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f773415c790 0x7f77340c9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f773359e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f77340c8a10 con 0x7f77340c5a50 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f773359e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7734160110 con 0x7f77340705e0 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f773359e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7734160ae0 con 0x7f773415c790 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340c5a50 0x7f77341f2c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340c5a50 0x7f77341f2c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53828/0 (socket says 172.21.15.1:53828) 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 -- 172.21.15.1:0/2271183795 learned_addr learned my addr 172.21.15.1:0/2271183795 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:25.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7732d9d640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f773415c790 0x7f77340c9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f773259c640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77340705e0 0x7f773415a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 -- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f773415c790 msgr2=0x7f77340c9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f773415c790 0x7f77340c9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 -- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77340705e0 msgr2=0x7f773415a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77340705e0 0x7f773415a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 -- 172.21.15.1:0/2271183795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77341f3190 con 0x7f77340c5a50 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f773259c640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77340705e0 0x7f773415a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7732d9d640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f773415c790 0x7f77340c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7731d9b640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340c5a50 0x7f77341f2c50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f772800c3a0 tx=0x7f772800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.690+0000 7f7722ffd640 1 -- 172.21.15.1:0/2271183795 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728017070 con 0x7f77340c5a50 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.691+0000 7f7722ffd640 1 -- 172.21.15.1:0/2271183795 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77280052d0 con 0x7f77340c5a50 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.691+0000 7f7722ffd640 1 -- 172.21.15.1:0/2271183795 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7728003830 con 0x7f77340c5a50 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.691+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77341545c0 con 0x7f77340c5a50 2024-05-09T05:15:25.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.691+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7734154b80 con 0x7f77340c5a50 2024-05-09T05:15:25.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.692+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7734076f80 con 0x7f77340c5a50 2024-05-09T05:15:25.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.693+0000 7f7722ffd640 1 -- 172.21.15.1:0/2271183795 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f772801a070 con 0x7f77340c5a50 2024-05-09T05:15:25.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.693+0000 7f7722ffd640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f77140762d0 0x7f77140787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:25.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.694+0000 7f7722ffd640 1 -- 172.21.15.1:0/2271183795 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7728096990 con 0x7f77340c5a50 2024-05-09T05:15:25.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.694+0000 7f773259c640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f77140762d0 0x7f77140787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:25.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.694+0000 7f773259c640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f77140762d0 0x7f77140787e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7724034000 tx=0x7f772402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:25.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:25.699+0000 7f7722ffd640 1 -- 172.21.15.1:0/2271183795 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7728060270 con 0x7f77340c5a50 2024-05-09T05:15:26.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.027+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f77340703c0 con 0x7f77340c5a50 2024-05-09T05:15:26.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.027+0000 7f7722ffd640 1 -- 172.21.15.1:0/2271183795 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f7728068260 con 0x7f77340c5a50 2024-05-09T05:15:26.026 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:15:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.029+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f77140762d0 msgr2=0x7f77140787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f77140762d0 0x7f77140787e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f7724034000 tx=0x7f772402e000 comp rx=0 tx=0).stop 2024-05-09T05:15:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340c5a50 msgr2=0x7f77341f2c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340c5a50 0x7f77341f2c50 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f772800c3a0 tx=0x7f772800c860 comp rx=0 tx=0).stop 2024-05-09T05:15:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 shutdown_connections 2024-05-09T05:15:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f77140762d0 0x7f77140787e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f773415c790 0x7f77340c9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f77340c5a50 0x7f77341f2c50 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 --2- 172.21.15.1:0/2271183795 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f77340705e0 0x7f773415a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 >> 172.21.15.1:0/2271183795 conn(0x7f7734073b70 msgr2=0x7f77341537a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 shutdown_connections 2024-05-09T05:15:26.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.030+0000 7f773359e640 1 -- 172.21.15.1:0/2271183795 wait complete. 2024-05-09T05:15:26.040 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:15:26.040 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:15:26.040 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:15:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.163+0000 7f57115d5640 1 Processor -- start 2024-05-09T05:15:26.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.163+0000 7f57115d5640 1 -- start start 2024-05-09T05:15:26.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f57115d5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c0c5da0 0x7f570c0c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f57115d5640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f570c153650 0x7f570c0cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f57115d5640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f570c159520 con 0x7f570c06c6f0 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f57115d5640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f570c0c64c0 con 0x7f570c153650 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f57115d5640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f570c0c67a0 con 0x7f570c0c5da0 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f570c06c6f0 0x7f570c159fa0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54524/0 (socket says 172.21.15.1:54524) 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570affd640 1 -- 172.21.15.1:0/734309369 learned_addr learned my addr 172.21.15.1:0/734309369 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570b7fe640 1 --2- 172.21.15.1:0/734309369 >> v2:172.21.15.196:3300/0 conn(0x7f570c153650 0x7f570c0cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570a7fc640 1 --2- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c0c5da0 0x7f570c0c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f5709ffb640 1 -- 172.21.15.1:0/734309369 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2737881675 0 0) 0x7f56fc001110 con 0x7f570c06c6f0 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f5709ffb640 1 -- 172.21.15.1:0/734309369 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f56f4003520 con 0x7f570c06c6f0 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570b7fe640 1 -- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c0c5da0 msgr2=0x7f570c0c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570b7fe640 1 --2- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c0c5da0 0x7f570c0c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570b7fe640 1 -- 172.21.15.1:0/734309369 >> v1:172.21.15.1:6789/0 conn(0x7f570c06c6f0 legacy=0x7f570c159fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570b7fe640 1 -- 172.21.15.1:0/734309369 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c0c6a80 con 0x7f570c153650 2024-05-09T05:15:26.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.165+0000 7f570a7fc640 1 --2- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c0c5da0 0x7f570c0c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.166+0000 7f570b7fe640 1 --2- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f570c153650 0x7f570c0cc1f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5704009d90 tx=0x7f570402a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cca06edfbf1fcd7d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.167+0000 7f5709ffb640 1 -- 172.21.15.1:0/734309369 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704007740 con 0x7f570c153650 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.167+0000 7f5709ffb640 1 -- 172.21.15.1:0/734309369 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f57040078e0 con 0x7f570c153650 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.167+0000 7f5709ffb640 1 -- 172.21.15.1:0/734309369 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5704007c90 con 0x7f570c153650 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.167+0000 7f57115d5640 1 -- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f570c153650 msgr2=0x7f570c0cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.167+0000 7f57115d5640 1 --2- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f570c153650 0x7f570c0cc1f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f5704009d90 tx=0x7f570402a6f0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.168+0000 7f57115d5640 1 -- 172.21.15.1:0/734309369 shutdown_connections 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.168+0000 7f57115d5640 1 --2- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f570c153650 0x7f570c0cc1f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.168+0000 7f57115d5640 1 --2- 172.21.15.1:0/734309369 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c0c5da0 0x7f570c0c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.168+0000 7f57115d5640 1 -- 172.21.15.1:0/734309369 >> 172.21.15.1:0/734309369 conn(0x7f570c0ce180 msgr2=0x7f570c0ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.168+0000 7f57115d5640 1 -- 172.21.15.1:0/734309369 shutdown_connections 2024-05-09T05:15:26.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.168+0000 7f57115d5640 1 -- 172.21.15.1:0/734309369 wait complete. 2024-05-09T05:15:26.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.170+0000 7f57115d5640 1 Processor -- start 2024-05-09T05:15:26.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.170+0000 7f57115d5640 1 -- start start 2024-05-09T05:15:26.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f57115d5640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f570c06c6f0 0x7f570c15b830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f57115d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c0c5da0 0x7f570c148200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f57115d5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c153650 0x7f570c0736b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f57115d5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f570c0c61b0 con 0x7f570c0c5da0 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f57115d5640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f570c159a80 con 0x7f570c06c6f0 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f57115d5640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f570c0c83c0 con 0x7f570c153650 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f570a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c0c5da0 0x7f570c148200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f570a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c0c5da0 0x7f570c148200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53834/0 (socket says 172.21.15.1:53834) 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f570a7fc640 1 -- 172.21.15.1:0/1713742040 learned_addr learned my addr 172.21.15.1:0/1713742040 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f570affd640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f570c06c6f0 0x7f570c15b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.172+0000 7f570b7fe640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c153650 0x7f570c0736b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f570a7fc640 1 -- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c153650 msgr2=0x7f570c0736b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f570a7fc640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f570c153650 0x7f570c0736b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f570a7fc640 1 -- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f570c06c6f0 msgr2=0x7f570c15b830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f570a7fc640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f570c06c6f0 0x7f570c15b830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f570a7fc640 1 -- 172.21.15.1:0/1713742040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f570c15bd70 con 0x7f570c0c5da0 2024-05-09T05:15:26.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f570a7fc640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c0c5da0 0x7f570c148200 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f570000e990 tx=0x7f570000ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5700017070 con 0x7f570c0c5da0 2024-05-09T05:15:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f570000c9e0 con 0x7f570c0c5da0 2024-05-09T05:15:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f570c1e68f0 con 0x7f570c0c5da0 2024-05-09T05:15:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f570000cdb0 con 0x7f570c0c5da0 2024-05-09T05:15:26.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.173+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f570c15d730 con 0x7f570c0c5da0 2024-05-09T05:15:26.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.174+0000 7f57115d5640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15a6e0 0x7f570c1e2950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.174+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f570c15c240 con 0x7f570c15a6e0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.174+0000 7f570affd640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15a6e0 0x7f570c1e2950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.175+0000 7f570affd640 1 -- 172.21.15.1:0/1713742040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f570c15ace0 con 0x7f570c15a6e0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.175+0000 7f570affd640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15a6e0 0x7f570c1e2950 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00e4d0 tx=0x7f56fc00e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.175+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f570001a070 con 0x7f570c0c5da0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.175+0000 7f56f37fe640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56f80762d0 0x7f56f80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5700097520 con 0x7f570c0c5da0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f570b7fe640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56f80762d0 0x7f56f80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc019070 con 0x7f570c15a6e0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f56fc005ac0 con 0x7f570c15a6e0 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15a6e0 msgr2=0x7f570c1e2950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f56f37fe640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15a6e0 0x7f570c1e2950 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00e4d0 tx=0x7f56fc00e990 comp rx=0 tx=0).stop 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15a6e0 msgr2=0x7f570c1e2950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:26.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f56f37fe640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15a6e0 0x7f570c1e2950 secure :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f56fc00e4d0 tx=0x7f56fc00e990 comp rx=0 tx=0).stop 2024-05-09T05:15:26.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.176+0000 7f570b7fe640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56f80762d0 0x7f56f80787e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5704034000 tx=0x7f570402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.183+0000 7f57115d5640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15b2b0 0x7f570c073bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.183+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f570c050870 con 0x7f570c15b2b0 2024-05-09T05:15:26.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.183+0000 7f570affd640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15b2b0 0x7f570c073bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.183+0000 7f570affd640 1 -- 172.21.15.1:0/1713742040 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f570c0cccf0 con 0x7f570c15b2b0 2024-05-09T05:15:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f570affd640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15b2b0 0x7f570c073bf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f56fc001eb0 tx=0x7f56fc0057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56fc019030 con 0x7f570c15b2b0 2024-05-09T05:15:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7f56fc0075d0 con 0x7f570c15b2b0 2024-05-09T05:15:26.183 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15b2b0 msgr2=0x7f570c073bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f56f37fe640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15b2b0 0x7f570c073bf0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f56fc001eb0 tx=0x7f56fc0057f0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f56f37fe640 1 -- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15b2b0 msgr2=0x7f570c073bf0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f56f37fe640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c15b2b0 0x7f570c073bf0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 74, 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 128379, 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:15:26.184 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.185 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.186 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:26.187 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.188 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:15:26.190 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.192 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.194 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.195 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.196 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:26.197 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.197 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.197 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:15:26.197 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:15:26.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56f80762d0 msgr2=0x7f56f80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f57115d5640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56f80762d0 0x7f56f80787e0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f5704034000 tx=0x7f570402f000 comp rx=0 tx=0).stop 2024-05-09T05:15:26.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c0c5da0 msgr2=0x7f570c148200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f57115d5640 1 --2- 172.21.15.1:0/1713742040 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f570c0c5da0 0x7f570c148200 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f570000e990 tx=0x7f570000ee50 comp rx=0 tx=0).stop 2024-05-09T05:15:26.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.184+0000 7f570b7fe640 1 -- 172.21.15.1:0/1713742040 reap_dead start 2024-05-09T05:15:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.185+0000 7f570b7fe640 1 -- 172.21.15.1:0/1713742040 reap_dead start 2024-05-09T05:15:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.185+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 shutdown_connections 2024-05-09T05:15:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.185+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 >> 172.21.15.1:0/1713742040 conn(0x7f570c0ce180 msgr2=0x7f570c06cb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.185+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 shutdown_connections 2024-05-09T05:15:26.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.185+0000 7f57115d5640 1 -- 172.21.15.1:0/1713742040 wait complete. 2024-05-09T05:15:26.199 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:15:26.199 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:15:26.199 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:15:26.332 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.333+0000 7fd36194d640 1 Processor -- start 2024-05-09T05:15:26.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.333+0000 7fd36194d640 1 -- start start 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd36194d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd35c15d130 0x7fd35c1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd36194d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd35c06c6f0 0x7fd35c0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd36194d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd35c0c3ce0 con 0x7fd35c071be0 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd36194d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd35c15d540 con 0x7fd35c06c6f0 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd36194d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd35c0c3fc0 con 0x7fd35c15d130 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd35a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd35c06c6f0 0x7fd35c0716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd35a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd35c06c6f0 0x7fd35c0716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53848/0 (socket says 172.21.15.1:53848) 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd35a7fc640 1 -- 172.21.15.1:0/3933691988 learned_addr learned my addr 172.21.15.1:0/3933691988 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd35affd640 1 --2- 172.21.15.1:0/3933691988 >> v2:172.21.15.196:3300/0 conn(0x7fd35c15d130 0x7fd35c1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd35a7fc640 1 -- 172.21.15.1:0/3933691988 >> v2:172.21.15.196:3300/0 conn(0x7fd35c15d130 msgr2=0x7fd35c1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:26.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd35a7fc640 1 --2- 172.21.15.1:0/3933691988 >> v2:172.21.15.196:3300/0 conn(0x7fd35c15d130 0x7fd35c1569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.335+0000 7fd35a7fc640 1 -- 172.21.15.1:0/3933691988 >> v1:172.21.15.196:6789/0 conn(0x7fd35c071be0 legacy=0x7fd35c0c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.336+0000 7fd35a7fc640 1 -- 172.21.15.1:0/3933691988 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35c06e4a0 con 0x7fd35c06c6f0 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.336+0000 7fd35a7fc640 1 --2- 172.21.15.1:0/3933691988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c06c6f0 0x7fd35c0716a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0cf650 tx=0x7fd35002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=254764b24f2adafd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd359ffb640 1 -- 172.21.15.1:0/3933691988 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350037070 con 0x7fd35c06c6f0 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd359ffb640 1 -- 172.21.15.1:0/3933691988 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd350032440 con 0x7fd35c06c6f0 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd359ffb640 1 -- 172.21.15.1:0/3933691988 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350032810 con 0x7fd35c06c6f0 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd36194d640 1 -- 172.21.15.1:0/3933691988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c06c6f0 msgr2=0x7fd35c0716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd36194d640 1 --2- 172.21.15.1:0/3933691988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c06c6f0 0x7fd35c0716a0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd35c0cf650 tx=0x7fd35002a270 comp rx=0 tx=0).stop 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd36194d640 1 -- 172.21.15.1:0/3933691988 shutdown_connections 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd36194d640 1 --2- 172.21.15.1:0/3933691988 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c06c6f0 0x7fd35c0716a0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd36194d640 1 --2- 172.21.15.1:0/3933691988 >> v2:172.21.15.196:3300/0 conn(0x7fd35c15d130 0x7fd35c1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.337+0000 7fd36194d640 1 -- 172.21.15.1:0/3933691988 >> 172.21.15.1:0/3933691988 conn(0x7fd35c0747b0 msgr2=0x7fd35c0ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.338+0000 7fd36194d640 1 -- 172.21.15.1:0/3933691988 shutdown_connections 2024-05-09T05:15:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.338+0000 7fd36194d640 1 -- 172.21.15.1:0/3933691988 wait complete. 2024-05-09T05:15:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.339+0000 7fd36194d640 1 Processor -- start 2024-05-09T05:15:26.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.339+0000 7fd36194d640 1 -- start start 2024-05-09T05:15:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd36194d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd35c06c6f0 0x7fd35c1f37a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd36194d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c071be0 0x7fd35c1e7b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd36194d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c15d130 0x7fd35c0cd820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd36194d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c06cb00 con 0x7fd35c071be0 2024-05-09T05:15:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd36194d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c06e1a0 con 0x7fd35c15d130 2024-05-09T05:15:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd36194d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd35c06cc80 con 0x7fd35c06c6f0 2024-05-09T05:15:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c071be0 0x7fd35c1e7b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c071be0 0x7fd35c1e7b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53862/0 (socket says 172.21.15.1:53862) 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 -- 172.21.15.1:0/1616771978 learned_addr learned my addr 172.21.15.1:0/1616771978 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35affd640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd35c06c6f0 0x7fd35c1f37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35b7fe640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c15d130 0x7fd35c0cd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 -- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd35c06c6f0 msgr2=0x7fd35c1f37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd35c06c6f0 0x7fd35c1f37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 -- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c15d130 msgr2=0x7fd35c0cd820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c15d130 0x7fd35c0cd820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 -- 172.21.15.1:0/1616771978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35c1e80d0 con 0x7fd35c071be0 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.341+0000 7fd35a7fc640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c071be0 0x7fd35c1e7b90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd35002a2b0 tx=0x7fd3500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.342+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350041030 con 0x7fd35c071be0 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.342+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35c0ce8d0 con 0x7fd35c071be0 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.342+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd350030440 con 0x7fd35c071be0 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.342+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd350037040 con 0x7fd35c071be0 2024-05-09T05:15:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.342+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd35c1e70b0 con 0x7fd35c071be0 2024-05-09T05:15:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.343+0000 7fd36194d640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0cdd60 0x7fd35c0c8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.343+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c1e85a0 con 0x7fd35c0cdd60 2024-05-09T05:15:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.343+0000 7fd35affd640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0cdd60 0x7fd35c0c8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.343+0000 7fd35affd640 1 -- 172.21.15.1:0/1616771978 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd35c0ce640 con 0x7fd35c0cdd60 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.344+0000 7fd35affd640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0cdd60 0x7fd35c0c8130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd34c011890 tx=0x7fd34c011d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.344+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c019070 con 0x7fd35c0cdd60 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.344+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd350044070 con 0x7fd35c071be0 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.344+0000 7fd343fff640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3480761d0 0x7fd3480786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.345+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd3500b7d10 con 0x7fd35c071be0 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.345+0000 7fd35b7fe640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3480761d0 0x7fd3480786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.345+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd34c004240 con 0x7fd35c0cdd60 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.345+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0cdd60 msgr2=0x7fd35c0c8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.345+0000 7fd343fff640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0cdd60 0x7fd35c0c8130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd34c011890 tx=0x7fd34c011d50 comp rx=0 tx=0).stop 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.345+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0cdd60 msgr2=0x7fd35c0c8130 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.345+0000 7fd343fff640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0cdd60 0x7fd35c0c8130 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.345+0000 7fd35b7fe640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3480761d0 0x7fd3480786e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd3540012e0 tx=0x7fd354009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.351+0000 7fd36194d640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0c8670 0x7fd35c1f6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.352+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd35c050870 con 0x7fd35c0c8670 2024-05-09T05:15:26.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.352+0000 7fd35affd640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0c8670 0x7fd35c1f6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.352+0000 7fd35affd640 1 -- 172.21.15.1:0/1616771978 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd35c0cc1b0 con 0x7fd35c0c8670 2024-05-09T05:15:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.352+0000 7fd35affd640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0c8670 0x7fd35c1f6360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd34c011890 tx=0x7fd34c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34c01e450 con 0x7fd35c0c8670 2024-05-09T05:15:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7fd34c0283f0 con 0x7fd35c0c8670 2024-05-09T05:15:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0c8670 msgr2=0x7fd35c1f6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd343fff640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0c8670 0x7fd35c1f6360 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fd34c011890 tx=0x7fd34c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd343fff640 1 -- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0c8670 msgr2=0x7fd35c1f6360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd343fff640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd35c0c8670 0x7fd35c1f6360 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 74, 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 188632, 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:26.353 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.354 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.355 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:15:26.356 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.357 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.358 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:15:26.359 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.360 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.361 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.362 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:15:26.363 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.364 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.365 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.366 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.366 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:15:26.366 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3480761d0 msgr2=0x7fd3480786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3480761d0 0x7fd3480786e0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fd3540012e0 tx=0x7fd354009040 comp rx=0 tx=0).stop 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c071be0 msgr2=0x7fd35c1e7b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c071be0 0x7fd35c1e7b90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd35002a2b0 tx=0x7fd3500046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd35b7fe640 1 -- 172.21.15.1:0/1616771978 reap_dead start 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 shutdown_connections 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 --2- 172.21.15.1:0/1616771978 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd35c071be0 0x7fd35c1e7b90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 >> 172.21.15.1:0/1616771978 conn(0x7fd35c0747b0 msgr2=0x7fd35c0c4750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 shutdown_connections 2024-05-09T05:15:26.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.353+0000 7fd36194d640 1 -- 172.21.15.1:0/1616771978 wait complete. 2024-05-09T05:15:26.368 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:15:26.368 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:15:26.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:15:26.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.489+0000 7fa063ce0640 1 Processor -- start 2024-05-09T05:15:26.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.489+0000 7fa063ce0640 1 -- start start 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.490+0000 7fa063ce0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 0x7fa05c0745d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.490+0000 7fa063ce0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa05c0c5510 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.490+0000 7fa063ce0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa05c153c10 con 0x7fa05c0c9410 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.490+0000 7fa063ce0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa05c073e10 con 0x7fa05c153800 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.490+0000 7fa061a7c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa05c0c7ac0 0x7fa05c0c4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54540/0 (socket says 172.21.15.1:54540) 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.490+0000 7fa061a7c640 1 -- 172.21.15.1:0/554850487 learned_addr learned my addr 172.21.15.1:0/554850487 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.490+0000 7fa06127b640 1 --2- 172.21.15.1:0/554850487 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 0x7fa05c0745d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.491+0000 7fa060a7a640 1 -- 172.21.15.1:0/554850487 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1962335717 0 0) 0x7fa05c0c5510 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.491+0000 7fa060a7a640 1 -- 172.21.15.1:0/554850487 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa048003520 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.491+0000 7fa060a7a640 1 -- 172.21.15.1:0/554850487 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3910398050 0 0) 0x7fa05c153c10 con 0x7fa05c0c9410 2024-05-09T05:15:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.491+0000 7fa06127b640 1 -- 172.21.15.1:0/554850487 >> v1:172.21.15.196:6789/0 conn(0x7fa05c0c9410 legacy=0x7fa05c163e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.491+0000 7fa06127b640 1 -- 172.21.15.1:0/554850487 >> v1:172.21.15.1:6789/0 conn(0x7fa05c0c7ac0 legacy=0x7fa05c0c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.491+0000 7fa06127b640 1 -- 172.21.15.1:0/554850487 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05c0c5b00 con 0x7fa05c153800 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.492+0000 7fa06127b640 1 --2- 172.21.15.1:0/554850487 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 0x7fa05c0745d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0097d0 tx=0x7fa04c02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c940e247e08f9fbd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.492+0000 7fa060a7a640 1 -- 172.21.15.1:0/554850487 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c037070 con 0x7fa05c153800 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.492+0000 7fa060a7a640 1 -- 172.21.15.1:0/554850487 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa04c032440 con 0x7fa05c153800 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.492+0000 7fa060a7a640 1 -- 172.21.15.1:0/554850487 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c0327f0 con 0x7fa05c153800 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.493+0000 7fa063ce0640 1 -- 172.21.15.1:0/554850487 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 msgr2=0x7fa05c0745d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.493+0000 7fa063ce0640 1 --2- 172.21.15.1:0/554850487 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 0x7fa05c0745d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fa04c0097d0 tx=0x7fa04c02a210 comp rx=0 tx=0).stop 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.493+0000 7fa063ce0640 1 -- 172.21.15.1:0/554850487 shutdown_connections 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.493+0000 7fa063ce0640 1 --2- 172.21.15.1:0/554850487 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 0x7fa05c0745d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.493+0000 7fa063ce0640 1 -- 172.21.15.1:0/554850487 >> 172.21.15.1:0/554850487 conn(0x7fa05c070f40 msgr2=0x7fa05c06c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.493+0000 7fa063ce0640 1 -- 172.21.15.1:0/554850487 shutdown_connections 2024-05-09T05:15:26.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.493+0000 7fa063ce0640 1 -- 172.21.15.1:0/554850487 wait complete. 2024-05-09T05:15:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.495+0000 7fa063ce0640 1 Processor -- start 2024-05-09T05:15:26.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.495+0000 7fa063ce0640 1 -- start start 2024-05-09T05:15:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa063ce0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa05c0c7ac0 0x7fa05c14e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa063ce0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa05c0c9410 0x7fa05c14b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa063ce0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 0x7fa05c1e4210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa063ce0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c0c63a0 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa063ce0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa05c0c60c0 con 0x7fa05c0c9410 2024-05-09T05:15:26.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa063ce0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa05c0740b0 con 0x7fa05c153800 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa061a7c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa05c0c7ac0 0x7fa05c14e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa061a7c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa05c0c7ac0 0x7fa05c14e380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53868/0 (socket says 172.21.15.1:53868) 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa061a7c640 1 -- 172.21.15.1:0/3196519647 learned_addr learned my addr 172.21.15.1:0/3196519647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa06227d640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 0x7fa05c1e4210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.497+0000 7fa06127b640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa05c0c9410 0x7fa05c14b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa061a7c640 1 -- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 msgr2=0x7fa05c1e4210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa061a7c640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c153800 0x7fa05c1e4210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa061a7c640 1 -- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa05c0c9410 msgr2=0x7fa05c14b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa061a7c640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa05c0c9410 0x7fa05c14b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa061a7c640 1 -- 172.21.15.1:0/3196519647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa05c14e8c0 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa061a7c640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa05c0c7ac0 0x7fa05c14e380 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa05800c440 tx=0x7fa05800c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058017070 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa058005ce0 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa05c0cd650 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa058007270 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.498+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa05c1e36b0 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.499+0000 7fa063ce0640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c14ea90 0x7fa05c1e4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.499+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa05c0cd400 con 0x7fa05c14ea90 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.500+0000 7fa06127b640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c14ea90 0x7fa05c1e4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.500+0000 7fa06127b640 1 -- 172.21.15.1:0/3196519647 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa05c050870 con 0x7fa05c14ea90 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.500+0000 7fa06127b640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c14ea90 0x7fa05c1e4750 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0c9f60 tx=0x7fa04c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.500+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa05801a070 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa0527fc640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa0400762d0 0x7fa0400787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa058096900 con 0x7fa05c0c7ac0 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c042070 con 0x7fa05c14ea90 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa06227d640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa0400762d0 0x7fa0400787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa04c03e660 con 0x7fa05c14ea90 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c14ea90 msgr2=0x7fa05c1e4750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa0527fc640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c14ea90 0x7fa05c1e4750 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fa05c0c9f60 tx=0x7fa04c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c14ea90 msgr2=0x7fa05c1e4750 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:26.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.501+0000 7fa0527fc640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c14ea90 0x7fa05c1e4750 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.502+0000 7fa06227d640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa0400762d0 0x7fa0400787e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa054007fd0 tx=0x7fa054009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.513+0000 7fa063ce0640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c0cb700 0x7fa05c1e1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.513+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa05c073bf0 con 0x7fa05c0cb700 2024-05-09T05:15:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.513+0000 7fa06127b640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c0cb700 0x7fa05c1e1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.514+0000 7fa06127b640 1 -- 172.21.15.1:0/3196519647 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa05c1e5890 con 0x7fa05c0cb700 2024-05-09T05:15:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.514+0000 7fa06127b640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c0cb700 0x7fa05c1e1850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009720 tx=0x7fa04c009190 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.514+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa04c037040 con 0x7fa05c0cb700 2024-05-09T05:15:26.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.514+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7fa04c03cd80 con 0x7fa05c0cb700 2024-05-09T05:15:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.514+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c0cb700 msgr2=0x7fa05c1e1850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.514+0000 7fa0527fc640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c0cb700 0x7fa05c1e1850 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa04c009720 tx=0x7fa04c009190 comp rx=0 tx=0).stop 2024-05-09T05:15:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.514+0000 7fa0527fc640 1 -- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c0cb700 msgr2=0x7fa05c1e1850 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:26.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.514+0000 7fa0527fc640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa05c0cb700 0x7fa05c1e1850 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.514 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:15:26.514 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:26.514 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:26.514 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 74, 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 7809, 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.515 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.516 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:15:26.517 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:26.518 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:15:26.519 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.520 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.521 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.522 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.523 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:15:26.524 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:26.525 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:26.526 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:15:26.526 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:15:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa0400762d0 msgr2=0x7fa0400787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa063ce0640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa0400762d0 0x7fa0400787e0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fa054007fd0 tx=0x7fa054009040 comp rx=0 tx=0).stop 2024-05-09T05:15:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa05c0c7ac0 msgr2=0x7fa05c14e380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa063ce0640 1 --2- 172.21.15.1:0/3196519647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa05c0c7ac0 0x7fa05c14e380 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa05800c440 tx=0x7fa05800c900 comp rx=0 tx=0).stop 2024-05-09T05:15:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa06227d640 1 -- 172.21.15.1:0/3196519647 reap_dead start 2024-05-09T05:15:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa06227d640 1 -- 172.21.15.1:0/3196519647 reap_dead start 2024-05-09T05:15:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 shutdown_connections 2024-05-09T05:15:26.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 >> 172.21.15.1:0/3196519647 conn(0x7fa05c070f40 msgr2=0x7fa05c0745d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 shutdown_connections 2024-05-09T05:15:26.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.515+0000 7fa063ce0640 1 -- 172.21.15.1:0/3196519647 wait complete. 2024-05-09T05:15:26.528 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:15:26.528 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:15:26.528 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:15:26.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.649+0000 7f29dcab8640 1 Processor -- start 2024-05-09T05:15:26.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.649+0000 7f29dcab8640 1 -- start start 2024-05-09T05:15:26.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29dcab8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d8160750 0x7f29d80ce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29dcab8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29d80cede0 0x7f29d81670d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29dcab8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f29d8156f10 con 0x7f29d806ce00 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29dcab8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f29d80c4ec0 con 0x7f29d80cede0 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29dcab8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f29d80cbbf0 con 0x7f29d8160750 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29d6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29d80cede0 0x7f29d81670d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29d6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f29d80cede0 0x7f29d81670d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53874/0 (socket says 172.21.15.1:53874) 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29d6d9d640 1 -- 172.21.15.1:0/4276786478 learned_addr learned my addr 172.21.15.1:0/4276786478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.651+0000 7f29d5d9b640 1 --2- 172.21.15.1:0/4276786478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d8160750 0x7f29d80ce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.652+0000 7f29d6d9d640 1 -- 172.21.15.1:0/4276786478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d8160750 msgr2=0x7f29d80ce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:26.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.652+0000 7f29d6d9d640 1 --2- 172.21.15.1:0/4276786478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d8160750 0x7f29d80ce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.652+0000 7f29d6d9d640 1 -- 172.21.15.1:0/4276786478 >> v1:172.21.15.196:6789/0 conn(0x7f29d806ce00 legacy=0x7f29d80c47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.652+0000 7f29d6d9d640 1 -- 172.21.15.1:0/4276786478 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d806d2d0 con 0x7f29d80cede0 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.652+0000 7f29d5d9b640 1 --2- 172.21.15.1:0/4276786478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d8160750 0x7f29d80ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.653+0000 7f29d6d9d640 1 --2- 172.21.15.1:0/4276786478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d81670d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f29d0009e80 tx=0x7f29d002a7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc54527816c9987 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.653+0000 7f29d559a640 1 -- 172.21.15.1:0/4276786478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d0036070 con 0x7f29d80cede0 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.653+0000 7f29d559a640 1 -- 172.21.15.1:0/4276786478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29d0031600 con 0x7f29d80cede0 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.653+0000 7f29d559a640 1 -- 172.21.15.1:0/4276786478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d00319d0 con 0x7f29d80cede0 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.653+0000 7f29dcab8640 1 -- 172.21.15.1:0/4276786478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 msgr2=0x7f29d81670d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.653+0000 7f29dcab8640 1 --2- 172.21.15.1:0/4276786478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d81670d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f29d0009e80 tx=0x7f29d002a7e0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.654+0000 7f29dcab8640 1 -- 172.21.15.1:0/4276786478 shutdown_connections 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.654+0000 7f29dcab8640 1 --2- 172.21.15.1:0/4276786478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d81670d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.654+0000 7f29dcab8640 1 --2- 172.21.15.1:0/4276786478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d8160750 0x7f29d80ce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.654+0000 7f29dcab8640 1 -- 172.21.15.1:0/4276786478 >> 172.21.15.1:0/4276786478 conn(0x7f29d806e400 msgr2=0x7f29d8071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.654+0000 7f29dcab8640 1 -- 172.21.15.1:0/4276786478 shutdown_connections 2024-05-09T05:15:26.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.654+0000 7f29dcab8640 1 -- 172.21.15.1:0/4276786478 wait complete. 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.656+0000 7f29dcab8640 1 Processor -- start 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.656+0000 7f29dcab8640 1 -- start start 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29dcab8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d806ce00 0x7f29d81f3080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29dcab8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d8155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29dcab8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29d8160750 0x7f29d80c6fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29dcab8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f29d80cb300 con 0x7f29d80cede0 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29dcab8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f29d806d570 con 0x7f29d8160750 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29dcab8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f29d80cb880 con 0x7f29d806ce00 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29d5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d8155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29d5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d8155a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53880/0 (socket says 172.21.15.1:53880) 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29d5d9b640 1 -- 172.21.15.1:0/2894120695 learned_addr learned my addr 172.21.15.1:0/2894120695 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29d6d9d640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29d8160750 0x7f29d80c6fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.657+0000 7f29d659c640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d806ce00 0x7f29d81f3080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29d5d9b640 1 -- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d806ce00 msgr2=0x7f29d81f3080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29d5d9b640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d806ce00 0x7f29d81f3080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29d5d9b640 1 -- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29d8160750 msgr2=0x7f29d80c6fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29d5d9b640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29d8160750 0x7f29d80c6fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29d5d9b640 1 -- 172.21.15.1:0/2894120695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29d81f35c0 con 0x7f29d80cede0 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29d5d9b640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d8155a30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00d800 tx=0x7f29cc00dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29c6ffd640 1 -- 172.21.15.1:0/2894120695 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc004030 con 0x7f29d80cede0 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29c6ffd640 1 -- 172.21.15.1:0/2894120695 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f29cc0041d0 con 0x7f29d80cede0 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29d80c64a0 con 0x7f29d80cede0 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29c6ffd640 1 -- 172.21.15.1:0/2894120695 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29cc0096a0 con 0x7f29d80cede0 2024-05-09T05:15:26.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.658+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29d81f68d0 con 0x7f29d80cede0 2024-05-09T05:15:26.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.659+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29d81f2950 con 0x7f29d80cede0 2024-05-09T05:15:26.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.660+0000 7f29c6ffd640 1 -- 172.21.15.1:0/2894120695 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f29cc009840 con 0x7f29d80cede0 2024-05-09T05:15:26.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.661+0000 7f29c6ffd640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f29b80762d0 0x7f29b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:26.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.661+0000 7f29c6ffd640 1 -- 172.21.15.1:0/2894120695 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f29cc097880 con 0x7f29d80cede0 2024-05-09T05:15:26.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.661+0000 7f29d659c640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f29b80762d0 0x7f29b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:26.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.662+0000 7f29d659c640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f29b80762d0 0x7f29b80787e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f29c8000f60 tx=0x7f29c8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:26.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.666+0000 7f29c6ffd640 1 -- 172.21.15.1:0/2894120695 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f29cc060a80 con 0x7f29d80cede0 2024-05-09T05:15:26.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.971+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f29d80caa30 con 0x7f29d80cede0 2024-05-09T05:15:26.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.972+0000 7f29c6ffd640 1 -- 172.21.15.1:0/2894120695 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f29cc064d80 con 0x7f29d80cede0 2024-05-09T05:15:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.974+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f29b80762d0 msgr2=0x7f29b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.974+0000 7f29dcab8640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f29b80762d0 0x7f29b80787e0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f29c8000f60 tx=0x7f29c8009040 comp rx=0 tx=0).stop 2024-05-09T05:15:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.974+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 msgr2=0x7f29d8155a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:26.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.974+0000 7f29dcab8640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d8155a30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f29cc00d800 tx=0x7f29cc00dcc0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.974+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 shutdown_connections 2024-05-09T05:15:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.975+0000 7f29dcab8640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f29b80762d0 0x7f29b80787e0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.975+0000 7f29dcab8640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f29d8160750 0x7f29d80c6fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.975+0000 7f29dcab8640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f29d80cede0 0x7f29d8155a30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.975+0000 7f29dcab8640 1 --2- 172.21.15.1:0/2894120695 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f29d806ce00 0x7f29d81f3080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.975+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 >> 172.21.15.1:0/2894120695 conn(0x7f29d806e400 msgr2=0x7f29d8075f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.975+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 shutdown_connections 2024-05-09T05:15:26.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:26.975+0000 7f29dcab8640 1 -- 172.21.15.1:0/2894120695 wait complete. 2024-05-09T05:15:26.987 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:15:34.048 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:15:34.051 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:15:36.988 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:15:36.989 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:15:37.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.111+0000 7fd67b2a8640 1 Processor -- start 2024-05-09T05:15:37.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.111+0000 7fd67b2a8640 1 -- start start 2024-05-09T05:15:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.112+0000 7fd67b2a8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd6740d16b0 0x7fd6740d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.112+0000 7fd67b2a8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd67406f1b0 0x7fd6740ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd67b2a8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd67406f6d0 con 0x7fd67415ca90 2024-05-09T05:15:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd67b2a8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd6740cd0e0 con 0x7fd67406f1b0 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd67b2a8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd6740cc4d0 con 0x7fd6740d16b0 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd678843640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd67406f1b0 0x7fd6740ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd678843640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd67406f1b0 0x7fd6740ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42738/0 (socket says 172.21.15.1:42738) 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd678843640 1 -- 172.21.15.1:0/2523268285 learned_addr learned my addr 172.21.15.1:0/2523268285 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd679044640 1 --2- 172.21.15.1:0/2523268285 >> v2:172.21.15.196:3300/0 conn(0x7fd6740d16b0 0x7fd6740d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd678843640 1 -- 172.21.15.1:0/2523268285 >> v2:172.21.15.196:3300/0 conn(0x7fd6740d16b0 msgr2=0x7fd6740d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd678843640 1 --2- 172.21.15.1:0/2523268285 >> v2:172.21.15.196:3300/0 conn(0x7fd6740d16b0 0x7fd6740d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd678843640 1 -- 172.21.15.1:0/2523268285 >> v1:172.21.15.196:6789/0 conn(0x7fd67415ca90 legacy=0x7fd67415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.113+0000 7fd678843640 1 -- 172.21.15.1:0/2523268285 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6740cc7b0 con 0x7fd67406f1b0 2024-05-09T05:15:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.114+0000 7fd678843640 1 --2- 172.21.15.1:0/2523268285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740ce740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd67406d120 tx=0x7fd66402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5089d1b34e26d199 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.114+0000 7fd66bfff640 1 -- 172.21.15.1:0/2523268285 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664037070 con 0x7fd67406f1b0 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.114+0000 7fd66bfff640 1 -- 172.21.15.1:0/2523268285 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd66402ae50 con 0x7fd67406f1b0 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.114+0000 7fd66bfff640 1 -- 172.21.15.1:0/2523268285 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd664032860 con 0x7fd67406f1b0 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.115+0000 7fd67b2a8640 1 -- 172.21.15.1:0/2523268285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 msgr2=0x7fd6740ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.115+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/2523268285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740ce740 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7fd67406d120 tx=0x7fd66402a300 comp rx=0 tx=0).stop 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.115+0000 7fd67b2a8640 1 -- 172.21.15.1:0/2523268285 shutdown_connections 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.115+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/2523268285 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740ce740 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.115+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/2523268285 >> v2:172.21.15.196:3300/0 conn(0x7fd6740d16b0 0x7fd6740d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.115+0000 7fd67b2a8640 1 -- 172.21.15.1:0/2523268285 >> 172.21.15.1:0/2523268285 conn(0x7fd6740b3660 msgr2=0x7fd6740b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.115+0000 7fd67b2a8640 1 -- 172.21.15.1:0/2523268285 shutdown_connections 2024-05-09T05:15:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.115+0000 7fd67b2a8640 1 -- 172.21.15.1:0/2523268285 wait complete. 2024-05-09T05:15:37.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.117+0000 7fd67b2a8640 1 Processor -- start 2024-05-09T05:15:37.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.118+0000 7fd67b2a8640 1 -- start start 2024-05-09T05:15:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd67b2a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740cb840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd67b2a8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6740d16b0 0x7fd6741f2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd67b2a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd67415ca90 0x7fd6741517b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd67b2a8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd6740d0910 con 0x7fd67406f1b0 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd67b2a8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd6740ccd70 con 0x7fd6740d16b0 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd67b2a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd6740cec80 con 0x7fd67415ca90 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd679044640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd679044640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740cb840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42746/0 (socket says 172.21.15.1:42746) 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd679044640 1 -- 172.21.15.1:0/3112804692 learned_addr learned my addr 172.21.15.1:0/3112804692 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd678843640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6740d16b0 0x7fd6741f2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.119+0000 7fd679845640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd67415ca90 0x7fd6741517b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd679044640 1 -- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd67415ca90 msgr2=0x7fd6741517b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd679044640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd67415ca90 0x7fd6741517b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd679044640 1 -- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6740d16b0 msgr2=0x7fd6741f2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd679044640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6740d16b0 0x7fd6741f2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd679044640 1 -- 172.21.15.1:0/3112804692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd6741f5f40 con 0x7fd67406f1b0 2024-05-09T05:15:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd679044640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740cb840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd67000d600 tx=0x7fd67000dac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd669ffb640 1 -- 172.21.15.1:0/3112804692 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6700125f0 con 0x7fd67406f1b0 2024-05-09T05:15:37.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd669ffb640 1 -- 172.21.15.1:0/3112804692 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd670019070 con 0x7fd67406f1b0 2024-05-09T05:15:37.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd669ffb640 1 -- 172.21.15.1:0/3112804692 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd6700129f0 con 0x7fd67406f1b0 2024-05-09T05:15:37.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd6741f6220 con 0x7fd67406f1b0 2024-05-09T05:15:37.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.120+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd6741f67e0 con 0x7fd67406f1b0 2024-05-09T05:15:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.121+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd674050870 con 0x7fd67406f1b0 2024-05-09T05:15:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.122+0000 7fd669ffb640 1 -- 172.21.15.1:0/3112804692 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd670012b90 con 0x7fd67406f1b0 2024-05-09T05:15:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.123+0000 7fd669ffb640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd6580762d0 0x7fd6580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.123+0000 7fd669ffb640 1 -- 172.21.15.1:0/3112804692 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd6700979c0 con 0x7fd67406f1b0 2024-05-09T05:15:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.123+0000 7fd678843640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd6580762d0 0x7fd6580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.124+0000 7fd678843640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd6580762d0 0x7fd6580787e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd664034000 tx=0x7fd66402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.128+0000 7fd669ffb640 1 -- 172.21.15.1:0/3112804692 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd670099e20 con 0x7fd67406f1b0 2024-05-09T05:15:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.450+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd67415cfb0 con 0x7fd67406f1b0 2024-05-09T05:15:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.450+0000 7fd669ffb640 1 -- 172.21.15.1:0/3112804692 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd670061330 con 0x7fd67406f1b0 2024-05-09T05:15:37.449 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":74,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:15:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd6580762d0 msgr2=0x7fd6580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd6580762d0 0x7fd6580787e0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fd664034000 tx=0x7fd66402e000 comp rx=0 tx=0).stop 2024-05-09T05:15:37.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 msgr2=0x7fd6740cb840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740cb840 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fd67000d600 tx=0x7fd67000dac0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 shutdown_connections 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd6580762d0 0x7fd6580787e0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd6740d16b0 0x7fd6741f2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd67415ca90 0x7fd6741517b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 --2- 172.21.15.1:0/3112804692 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd67406f1b0 0x7fd6740cb840 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 >> 172.21.15.1:0/3112804692 conn(0x7fd6740b3660 msgr2=0x7fd6740d0170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 shutdown_connections 2024-05-09T05:15:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.453+0000 7fd67b2a8640 1 -- 172.21.15.1:0/3112804692 wait complete. 2024-05-09T05:15:37.463 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:15:37.463 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:15:37.464 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:15:37.464 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:15:37.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.596+0000 7f66b4f84640 1 Processor -- start 2024-05-09T05:15:37.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.596+0000 7f66b4f84640 1 -- start start 2024-05-09T05:15:37.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66b4f84640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f66b00c46e0 0x7f66b0159650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66b4f84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c4af0 0x7f66b0159b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66b4f84640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66b00c3be0 con 0x7f66b00c89a0 2024-05-09T05:15:37.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66b4f84640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f66b00cb040 con 0x7f66b00c46e0 2024-05-09T05:15:37.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66b4f84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f66b00c9290 con 0x7f66b00c4af0 2024-05-09T05:15:37.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66aed9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c4af0 0x7f66b0159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66a7fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f66b00c46e0 0x7f66b0159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66aed9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c4af0 0x7f66b0159b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:55104/0 (socket says 172.21.15.1:55104) 2024-05-09T05:15:37.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.598+0000 7f66aed9d640 1 -- 172.21.15.1:0/3838097647 learned_addr learned my addr 172.21.15.1:0/3838097647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.599+0000 7f66a7fff640 1 -- 172.21.15.1:0/3838097647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c4af0 msgr2=0x7f66b0159b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.599+0000 7f66a7fff640 1 --2- 172.21.15.1:0/3838097647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c4af0 0x7f66b0159b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.599+0000 7f66a7fff640 1 -- 172.21.15.1:0/3838097647 >> v1:172.21.15.196:6789/0 conn(0x7f66b00c89a0 legacy=0x7f66b00ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.599+0000 7f66a7fff640 1 -- 172.21.15.1:0/3838097647 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b006e890 con 0x7f66b00c46e0 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.599+0000 7f66aed9d640 1 --2- 172.21.15.1:0/3838097647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c4af0 0x7f66b0159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.600+0000 7f66a7fff640 1 --2- 172.21.15.1:0/3838097647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b00c46e0 0x7f66b0159650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f66980097a0 tx=0x7f669802a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1670f4b249de6bc5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.600+0000 7f66add9b640 1 -- 172.21.15.1:0/3838097647 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698037070 con 0x7f66b00c46e0 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.600+0000 7f66add9b640 1 -- 172.21.15.1:0/3838097647 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6698032440 con 0x7f66b00c46e0 2024-05-09T05:15:37.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.600+0000 7f66add9b640 1 -- 172.21.15.1:0/3838097647 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698032810 con 0x7f66b00c46e0 2024-05-09T05:15:37.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.600+0000 7f66b4f84640 1 -- 172.21.15.1:0/3838097647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b00c46e0 msgr2=0x7f66b0159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.600+0000 7f66b4f84640 1 --2- 172.21.15.1:0/3838097647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b00c46e0 0x7f66b0159650 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f66980097a0 tx=0x7f669802a180 comp rx=0 tx=0).stop 2024-05-09T05:15:37.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.601+0000 7f66b4f84640 1 -- 172.21.15.1:0/3838097647 shutdown_connections 2024-05-09T05:15:37.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.601+0000 7f66b4f84640 1 --2- 172.21.15.1:0/3838097647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c4af0 0x7f66b0159b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.601+0000 7f66b4f84640 1 --2- 172.21.15.1:0/3838097647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b00c46e0 0x7f66b0159650 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.601+0000 7f66b4f84640 1 -- 172.21.15.1:0/3838097647 >> 172.21.15.1:0/3838097647 conn(0x7f66b00747b0 msgr2=0x7f66b00ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:37.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.601+0000 7f66b4f84640 1 -- 172.21.15.1:0/3838097647 shutdown_connections 2024-05-09T05:15:37.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.601+0000 7f66b4f84640 1 -- 172.21.15.1:0/3838097647 wait complete. 2024-05-09T05:15:37.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.603+0000 7f66b4f84640 1 Processor -- start 2024-05-09T05:15:37.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.603+0000 7f66b4f84640 1 -- start start 2024-05-09T05:15:37.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66b4f84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c46e0 0x7f66b01ee260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66b4f84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00c4af0 0x7f66b00cdc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66b4f84640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b00c89a0 0x7f66b0150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66b4f84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f66b00c94e0 con 0x7f66b00c4af0 2024-05-09T05:15:37.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66b4f84640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f66b00c4140 con 0x7f66b00c89a0 2024-05-09T05:15:37.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66b4f84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f66b00c97c0 con 0x7f66b00c46e0 2024-05-09T05:15:37.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00c4af0 0x7f66b00cdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66a7fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00c4af0 0x7f66b00cdc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42754/0 (socket says 172.21.15.1:42754) 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66a7fff640 1 -- 172.21.15.1:0/2832162633 learned_addr learned my addr 172.21.15.1:0/2832162633 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66aed9d640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b00c89a0 0x7f66b0150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.605+0000 7f66ae59c640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c46e0 0x7f66b01ee260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a7fff640 1 -- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c46e0 msgr2=0x7f66b01ee260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a7fff640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c46e0 0x7f66b01ee260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a7fff640 1 -- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b00c89a0 msgr2=0x7f66b0150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a7fff640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66b00c89a0 0x7f66b0150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a7fff640 1 -- 172.21.15.1:0/2832162633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66b00ce1b0 con 0x7f66b00c4af0 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66ae59c640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66b00c46e0 0x7f66b01ee260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a7fff640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00c4af0 0x7f66b00cdc70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f66980098c0 tx=0x7f66980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6698037040 con 0x7f66b00c4af0 2024-05-09T05:15:37.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6698030500 con 0x7f66b00c4af0 2024-05-09T05:15:37.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66980308d0 con 0x7f66b00c4af0 2024-05-09T05:15:37.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66b00ccbd0 con 0x7f66b00c4af0 2024-05-09T05:15:37.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.606+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66b00cd190 con 0x7f66b00c4af0 2024-05-09T05:15:37.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.607+0000 7f66b4f84640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00cd360 0x7f66b01e1b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.607+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f66b00ce3c0 con 0x7f66b00cd360 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.607+0000 7f66ae59c640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00cd360 0x7f66b01e1b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.608+0000 7f66ae59c640 1 -- 172.21.15.1:0/2832162633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f66b0050870 con 0x7f66b00cd360 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.608+0000 7f66ae59c640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00cd360 0x7f66b01e1b00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f66a000e4d0 tx=0x7f66a000e990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.608+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a00052e0 con 0x7f66b00cd360 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.608+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6698044020 con 0x7f66b00c4af0 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.609+0000 7f66a6ffd640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66900761d0 0x7f66900786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.609+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f66980b7890 con 0x7f66b00c4af0 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.609+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f66a0018070 con 0x7f66b00cd360 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.609+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00cd360 msgr2=0x7f66b01e1b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.609+0000 7f66a6ffd640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00cd360 0x7f66b01e1b00 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f66a000e4d0 tx=0x7f66a000e990 comp rx=0 tx=0).stop 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.609+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00cd360 msgr2=0x7f66b01e1b00 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.609+0000 7f66a6ffd640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00cd360 0x7f66b01e1b00 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.609+0000 7f66aed9d640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66900761d0 0x7f66900786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.610+0000 7f66aed9d640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66900761d0 0x7f66900786e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f66b0150ec0 tx=0x7f66a800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.616+0000 7f66b4f84640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b01e2040 0x7f66b01f17a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.616+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f66b01df800 con 0x7f66b01e2040 2024-05-09T05:15:37.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.617+0000 7f66ae59c640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b01e2040 0x7f66b01f17a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.617+0000 7f66ae59c640 1 -- 172.21.15.1:0/2832162633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f66b00ca360 con 0x7f66b01e2040 2024-05-09T05:15:37.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.617+0000 7f66ae59c640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b01e2040 0x7f66b01f17a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f66a000e4d0 tx=0x7f66a0000ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.617+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66a0006d20 con 0x7f66b01e2040 2024-05-09T05:15:37.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7f66a0006d20 con 0x7f66b01e2040 2024-05-09T05:15:37.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b01e2040 msgr2=0x7f66b01f17a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66a6ffd640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b01e2040 0x7f66b01f17a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f66a000e4d0 tx=0x7f66a0000ed0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66a6ffd640 1 -- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b01e2040 msgr2=0x7f66b01f17a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:37.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66a6ffd640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b01e2040 0x7f66b01f17a0 secure :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f66a000e4d0 tx=0x7f66a0000ed0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 74, 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 139812, 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:15:37.618 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 14, 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.619 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.620 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:37.621 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:15:37.622 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:15:37.622 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:15:37.622 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:15:37.622 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:15:37.622 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:15:37.622 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:15:37.622 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:37.623 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:15:37.623 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.623 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.623 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.623 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.623 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.623 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:37.624 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.625 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.626 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.627 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.628 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.629 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.630 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.631 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.632 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.633 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:37.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.633 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:15:37.633 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:15:37.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66900761d0 msgr2=0x7f66900786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66b4f84640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66900761d0 0x7f66900786e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f66b0150ec0 tx=0x7f66a800b040 comp rx=0 tx=0).stop 2024-05-09T05:15:37.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00c4af0 msgr2=0x7f66b00cdc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66b4f84640 1 --2- 172.21.15.1:0/2832162633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66b00c4af0 0x7f66b00cdc70 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f66980098c0 tx=0x7f66980046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66aed9d640 1 -- 172.21.15.1:0/2832162633 reap_dead start 2024-05-09T05:15:37.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66aed9d640 1 -- 172.21.15.1:0/2832162633 reap_dead start 2024-05-09T05:15:37.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 shutdown_connections 2024-05-09T05:15:37.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 >> 172.21.15.1:0/2832162633 conn(0x7f66b00747b0 msgr2=0x7f66b0152190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:37.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 shutdown_connections 2024-05-09T05:15:37.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.618+0000 7f66b4f84640 1 -- 172.21.15.1:0/2832162633 wait complete. 2024-05-09T05:15:37.635 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:15:37.635 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:15:37.636 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:15:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.772+0000 7f2181ec0640 1 Processor -- start 2024-05-09T05:15:37.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.772+0000 7f2181ec0640 1 -- start start 2024-05-09T05:15:37.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f2181ec0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c0b4ec0 0x7f217c0cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f2181ec0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f217c160460 0x7f217c160870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f2181ec0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f217c15c7b0 con 0x7f217c0ce010 2024-05-09T05:15:37.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f2181ec0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f217c15d0c0 con 0x7f217c160460 2024-05-09T05:15:37.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f2181ec0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f217c0cca50 con 0x7f217c0b4ec0 2024-05-09T05:15:37.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f217c0ce010 0x7f217c15fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54486/0 (socket says 172.21.15.1:54486) 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217affd640 1 -- 172.21.15.1:0/2891976092 learned_addr learned my addr 172.21.15.1:0/2891976092 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217bfff640 1 --2- 172.21.15.1:0/2891976092 >> v2:172.21.15.196:3300/0 conn(0x7f217c160460 0x7f217c160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217b7fe640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c0b4ec0 0x7f217c0cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 407010147 0 0) 0x7f217c15c7b0 con 0x7f217c0ce010 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2164003520 con 0x7f217c0ce010 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2883393707 0 0) 0x7f2164003520 con 0x7f217c0ce010 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c0b4ec0 msgr2=0x7f217c0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217a7fc640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c0b4ec0 0x7f217c0cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 >> v2:172.21.15.196:3300/0 conn(0x7f217c160460 msgr2=0x7f217c160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217a7fc640 1 --2- 172.21.15.1:0/2891976092 >> v2:172.21.15.196:3300/0 conn(0x7f217c160460 0x7f217c160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.774+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c0ccd30 con 0x7f217c0ce010 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f21700016f0 con 0x7f217c0ce010 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 >> v1:172.21.15.1:6789/0 conn(0x7f217c0ce010 legacy=0x7f217c15fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217a7fc640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2164003a00 0x7f2164003e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217a7fc640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164004800 0x7f2164004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217a7fc640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c0b4ec0 0x7f2164005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f217c15d0c0 con 0x7f2164003a00 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f217c0cca50 con 0x7f2164004800 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f216400bb20 con 0x7f217c0b4ec0 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217b7fe640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2164003a00 0x7f2164003e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217bfff640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c0b4ec0 0x7f2164005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217affd640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164004800 0x7f2164004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217b7fe640 1 -- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c0b4ec0 msgr2=0x7f2164005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217b7fe640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c0b4ec0 0x7f2164005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217b7fe640 1 -- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164004800 msgr2=0x7f2164004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217b7fe640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164004800 0x7f2164004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217b7fe640 1 -- 172.21.15.1:0/2891976092 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2164003520 con 0x7f2164003a00 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.775+0000 7f217affd640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164004800 0x7f2164004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.776+0000 7f217b7fe640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2164003a00 0x7f2164003e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f216c009ac0 tx=0x7f2164005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8b2b588d69165d7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.776+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c036070 con 0x7f2164003a00 2024-05-09T05:15:37.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.776+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f216c031400 con 0x7f2164003a00 2024-05-09T05:15:37.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.776+0000 7f217a7fc640 1 -- 172.21.15.1:0/2891976092 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c009690 con 0x7f2164003a00 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.777+0000 7f2181ec0640 1 -- 172.21.15.1:0/2891976092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2164003a00 msgr2=0x7f2164003e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.777+0000 7f2181ec0640 1 --2- 172.21.15.1:0/2891976092 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2164003a00 0x7f2164003e90 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f216c009ac0 tx=0x7f2164005830 comp rx=0 tx=0).stop 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.777+0000 7f217bfff640 1 -- 172.21.15.1:0/2891976092 reap_dead start 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.777+0000 7f2181ec0640 1 -- 172.21.15.1:0/2891976092 shutdown_connections 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.777+0000 7f2181ec0640 1 -- 172.21.15.1:0/2891976092 >> 172.21.15.1:0/2891976092 conn(0x7f217c0d2360 msgr2=0x7f217c0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.777+0000 7f2181ec0640 1 -- 172.21.15.1:0/2891976092 shutdown_connections 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.777+0000 7f2181ec0640 1 -- 172.21.15.1:0/2891976092 wait complete. 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.779+0000 7f2181ec0640 1 Processor -- start 2024-05-09T05:15:37.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.779+0000 7f2181ec0640 1 -- start start 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.780+0000 7f2181ec0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164003a00 0x7f217c1f3020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.780+0000 7f2181ec0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f217c0ca6f0 0x7f217c0c90b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.780+0000 7f2181ec0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c15d970 0x7f217c15dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.780+0000 7f2181ec0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f217c0cc750 con 0x7f217c0ca6f0 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.780+0000 7f2181ec0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f217c15cd50 con 0x7f2164003a00 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.780+0000 7f2181ec0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f217c156ae0 con 0x7f217c15d970 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.780+0000 7f217affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f217c0ca6f0 0x7f217c0c90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f217c0ca6f0 0x7f217c0c90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42790/0 (socket says 172.21.15.1:42790) 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217affd640 1 -- 172.21.15.1:0/1093449483 learned_addr learned my addr 172.21.15.1:0/1093449483 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217b7fe640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164003a00 0x7f217c1f3020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217bfff640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c15d970 0x7f217c15dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217affd640 1 -- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c15d970 msgr2=0x7f217c15dd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217affd640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f217c15d970 0x7f217c15dd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217affd640 1 -- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164003a00 msgr2=0x7f217c1f3020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217affd640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2164003a00 0x7f217c1f3020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217affd640 1 -- 172.21.15.1:0/1093449483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f217c1f3560 con 0x7f217c0ca6f0 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f217affd640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f217c0ca6f0 0x7f217c0c90b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f217000e760 tx=0x7f217000ec20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217000ee70 con 0x7f217c0ca6f0 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2170018070 con 0x7f217c0ca6f0 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.781+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f217c06f170 con 0x7f217c0ca6f0 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.782+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f217000cad0 con 0x7f217c0ca6f0 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.782+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f217c0cb340 con 0x7f217c0ca6f0 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.783+0000 7f2181ec0640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c0cb580 0x7f217c14c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.783+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f217c1f3a30 con 0x7f217c0cb580 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.783+0000 7f217b7fe640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c0cb580 0x7f217c14c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.783+0000 7f217b7fe640 1 -- 172.21.15.1:0/1093449483 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f217c050870 con 0x7f217c0cb580 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.783+0000 7f217b7fe640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c0cb580 0x7f217c14c4a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f216c009a90 tx=0x7f216c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.784+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f216c043070 con 0x7f217c0cb580 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.784+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f216c03c450 con 0x7f217c0cb580 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.784+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c0cb580 msgr2=0x7f217c14c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.784+0000 7f2163fff640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c0cb580 0x7f217c14c4a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f216c009a90 tx=0x7f216c004730 comp rx=0 tx=0).stop 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.784+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c0cb580 msgr2=0x7f217c14c4a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.784+0000 7f2163fff640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c0cb580 0x7f217c14c4a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.784+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f217000cc70 con 0x7f217c0ca6f0 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.785+0000 7f2163fff640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2168075d70 0x7f2168078280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.785+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f21700970d0 con 0x7f217c0ca6f0 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.785+0000 7f217b7fe640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2168075d70 0x7f2168078280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.786+0000 7f217b7fe640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2168075d70 0x7f2168078280 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f217c0780e0 tx=0x7f216c03e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.794+0000 7f2181ec0640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c14c9e0 0x7f217c15b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.794+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f217c0cae10 con 0x7f217c14c9e0 2024-05-09T05:15:37.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.795+0000 7f217bfff640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c14c9e0 0x7f217c15b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.795+0000 7f217bfff640 1 -- 172.21.15.1:0/1093449483 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f217c0cd2b0 con 0x7f217c14c9e0 2024-05-09T05:15:37.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.795+0000 7f217bfff640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c14c9e0 0x7f217c15b470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f217c15a3a0 tx=0x7f2174012670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.795+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2174012ea0 con 0x7f217c14c9e0 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 74, 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 200074, 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.795 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 14, 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.796 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.797 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:15:37.798 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.799 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.800 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.801 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:15:37.804 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.805 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.806 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:15:37.807 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:15:37.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4892 (secure 0 0 0) 0x7f217401e630 con 0x7f217c14c9e0 2024-05-09T05:15:37.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c14c9e0 msgr2=0x7f217c15b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2163fff640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c14c9e0 0x7f217c15b470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f217c15a3a0 tx=0x7f2174012670 comp rx=0 tx=0).stop 2024-05-09T05:15:37.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2163fff640 1 -- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c14c9e0 msgr2=0x7f217c15b470 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:37.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2163fff640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f217c14c9e0 0x7f217c15b470 secure :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f217c15a3a0 tx=0x7f2174012670 comp rx=0 tx=0).stop 2024-05-09T05:15:37.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2168075d70 msgr2=0x7f2168078280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2181ec0640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2168075d70 0x7f2168078280 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f217c0780e0 tx=0x7f216c03e000 comp rx=0 tx=0).stop 2024-05-09T05:15:37.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f217c0ca6f0 msgr2=0x7f217c0c90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2181ec0640 1 --2- 172.21.15.1:0/1093449483 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f217c0ca6f0 0x7f217c0c90b0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f217000e760 tx=0x7f217000ec20 comp rx=0 tx=0).stop 2024-05-09T05:15:37.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f217bfff640 1 -- 172.21.15.1:0/1093449483 reap_dead start 2024-05-09T05:15:37.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 shutdown_connections 2024-05-09T05:15:37.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 >> 172.21.15.1:0/1093449483 conn(0x7f217c0d2360 msgr2=0x7f217c155eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:37.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 shutdown_connections 2024-05-09T05:15:37.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.796+0000 7f2181ec0640 1 -- 172.21.15.1:0/1093449483 wait complete. 2024-05-09T05:15:37.810 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:15:37.810 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:15:37.811 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:15:37.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.939+0000 7f22bb97e640 1 Processor -- start 2024-05-09T05:15:37.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.940+0000 7f22bb97e640 1 -- start start 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.941+0000 7f22bb97e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b415d130 0x7f22b41569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.941+0000 7f22bb97e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f22b40c48c0 0x7f22b40c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.941+0000 7f22bb97e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f22b40c3ce0 con 0x7f22b406c6f0 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.941+0000 7f22bb97e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f22b415d540 con 0x7f22b40c48c0 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.941+0000 7f22bb97e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f22b40c3fc0 con 0x7f22b415d130 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b971a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b415d130 0x7f22b41569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b9f1b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f22b40c48c0 0x7f22b40c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b8f19640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f22b406c6f0 0x7f22b40716a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:50320/0 (socket says 172.21.15.1:50320) 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b971a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b415d130 0x7f22b41569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:55128/0 (socket says 172.21.15.1:55128) 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b9f1b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f22b40c48c0 0x7f22b40c4cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:34468/0 (socket says 172.21.15.1:34468) 2024-05-09T05:15:37.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b8f19640 1 -- 172.21.15.1:0/3261209938 learned_addr learned my addr 172.21.15.1:0/3261209938 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:15:37.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b971a640 1 -- 172.21.15.1:0/3261209938 >> v2:172.21.15.196:3300/0 conn(0x7f22b40c48c0 msgr2=0x7f22b40c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:37.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b971a640 1 --2- 172.21.15.1:0/3261209938 >> v2:172.21.15.196:3300/0 conn(0x7f22b40c48c0 0x7f22b40c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b971a640 1 -- 172.21.15.1:0/3261209938 >> v1:172.21.15.196:6789/0 conn(0x7f22b406c6f0 legacy=0x7f22b40716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b971a640 1 -- 172.21.15.1:0/3261209938 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b406e4a0 con 0x7f22b415d130 2024-05-09T05:15:37.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.942+0000 7f22b9f1b640 1 --2- 172.21.15.1:0/3261209938 >> v2:172.21.15.196:3300/0 conn(0x7f22b40c48c0 0x7f22b40c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:15:37.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.943+0000 7f22b971a640 1 --2- 172.21.15.1:0/3261209938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b415d130 0x7f22b41569c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22b00097a0 tx=0x7f22b002a180 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18e3737595d4dba5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.943+0000 7f22abfff640 1 -- 172.21.15.1:0/3261209938 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0037070 con 0x7f22b415d130 2024-05-09T05:15:37.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.943+0000 7f22abfff640 1 -- 172.21.15.1:0/3261209938 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22b0032440 con 0x7f22b415d130 2024-05-09T05:15:37.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.943+0000 7f22abfff640 1 -- 172.21.15.1:0/3261209938 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0032810 con 0x7f22b415d130 2024-05-09T05:15:37.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.944+0000 7f22bb97e640 1 -- 172.21.15.1:0/3261209938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b415d130 msgr2=0x7f22b41569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.944+0000 7f22bb97e640 1 --2- 172.21.15.1:0/3261209938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b415d130 0x7f22b41569c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f22b00097a0 tx=0x7f22b002a180 comp rx=0 tx=0).stop 2024-05-09T05:15:37.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.944+0000 7f22bb97e640 1 -- 172.21.15.1:0/3261209938 shutdown_connections 2024-05-09T05:15:37.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.944+0000 7f22bb97e640 1 --2- 172.21.15.1:0/3261209938 >> v2:172.21.15.196:3300/0 conn(0x7f22b40c48c0 0x7f22b40c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.944+0000 7f22bb97e640 1 --2- 172.21.15.1:0/3261209938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b415d130 0x7f22b41569c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.944+0000 7f22bb97e640 1 -- 172.21.15.1:0/3261209938 >> 172.21.15.1:0/3261209938 conn(0x7f22b40747b0 msgr2=0x7f22b40ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:37.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.944+0000 7f22bb97e640 1 -- 172.21.15.1:0/3261209938 shutdown_connections 2024-05-09T05:15:37.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.944+0000 7f22bb97e640 1 -- 172.21.15.1:0/3261209938 wait complete. 2024-05-09T05:15:37.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.946+0000 7f22bb97e640 1 Processor -- start 2024-05-09T05:15:37.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.947+0000 7f22bb97e640 1 -- start start 2024-05-09T05:15:37.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.948+0000 7f22bb97e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b406c6f0 0x7f22b41e63d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.948+0000 7f22bb97e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b40c48c0 0x7f22b414d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.948+0000 7f22bb97e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b415d130 0x7f22b41f5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.948+0000 7f22bb97e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f22b406e1a0 con 0x7f22b415d130 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.948+0000 7f22bb97e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f22b40c5210 con 0x7f22b40c48c0 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.948+0000 7f22bb97e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f22b406cb00 con 0x7f22b406c6f0 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.948+0000 7f22b9f1b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b415d130 0x7f22b41f5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b9f1b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b415d130 0x7f22b41f5f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42804/0 (socket says 172.21.15.1:42804) 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b9f1b640 1 -- 172.21.15.1:0/3990989237 learned_addr learned my addr 172.21.15.1:0/3990989237 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b8f19640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b40c48c0 0x7f22b414d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b971a640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b406c6f0 0x7f22b41e63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b9f1b640 1 -- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b406c6f0 msgr2=0x7f22b41e63d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b9f1b640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b406c6f0 0x7f22b41e63d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b9f1b640 1 -- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b40c48c0 msgr2=0x7f22b414d5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b9f1b640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f22b40c48c0 0x7f22b414d5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b9f1b640 1 -- 172.21.15.1:0/3990989237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b41e6910 con 0x7f22b415d130 2024-05-09T05:15:37.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22b9f1b640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b415d130 0x7f22b41f5f20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00efc0 tx=0x7f22ac00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac018030 con 0x7f22b415d130 2024-05-09T05:15:37.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f22ac00cdf0 con 0x7f22b415d130 2024-05-09T05:15:37.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b40ce670 con 0x7f22b415d130 2024-05-09T05:15:37.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.949+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ac003bb0 con 0x7f22b415d130 2024-05-09T05:15:37.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.950+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b4150cb0 con 0x7f22b415d130 2024-05-09T05:15:37.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.951+0000 7f22bb97e640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b41e6ae0 0x7f22b41f6460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.951+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f22b4050870 con 0x7f22b41e6ae0 2024-05-09T05:15:37.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.951+0000 7f22b971a640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b41e6ae0 0x7f22b41f6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.951+0000 7f22b971a640 1 -- 172.21.15.1:0/3990989237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f22b40cb100 con 0x7f22b41e6ae0 2024-05-09T05:15:37.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.952+0000 7f22b971a640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b41e6ae0 0x7f22b41f6460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22b40cf420 tx=0x7f22b00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.952 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.952+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0041070 con 0x7f22b41e6ae0 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.952+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f22ac01b070 con 0x7f22b415d130 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22a9ffb640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f22980761d0 0x7f22980786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f22ac097fc0 con 0x7f22b415d130 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f22b003c660 con 0x7f22b41e6ae0 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b41e6ae0 msgr2=0x7f22b41f6460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22a9ffb640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b41e6ae0 0x7f22b41f6460 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22b40cf420 tx=0x7f22b00046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b41e6ae0 msgr2=0x7f22b41f6460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22a9ffb640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b41e6ae0 0x7f22b41f6460 secure :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f22b40cf420 tx=0x7f22b00046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22b8f19640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f22980761d0 0x7f22980786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.953+0000 7f22b8f19640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f22980761d0 0x7f22980786e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f22b41f05e0 tx=0x7f22a4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.960+0000 7f22bb97e640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b4159e40 0x7f22b41f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:37.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.960+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f22b40766b0 con 0x7f22b4159e40 2024-05-09T05:15:37.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.960+0000 7f22b971a640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b4159e40 0x7f22b41f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.961+0000 7f22b971a640 1 -- 172.21.15.1:0/3990989237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f22b41f35b0 con 0x7f22b4159e40 2024-05-09T05:15:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.961+0000 7f22b971a640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b4159e40 0x7f22b41f69a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22b002a1c0 tx=0x7f22b0008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.961+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22b0037040 con 0x7f22b4159e40 2024-05-09T05:15:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.961+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f22b0003610 con 0x7f22b4159e40 2024-05-09T05:15:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.961+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b4159e40 msgr2=0x7f22b41f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.961+0000 7f22a9ffb640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b4159e40 0x7f22b41f69a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22b002a1c0 tx=0x7f22b0008030 comp rx=0 tx=0).stop 2024-05-09T05:15:37.960 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.961+0000 7f22a9ffb640 1 -- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b4159e40 msgr2=0x7f22b41f69a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.961+0000 7f22a9ffb640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f22b4159e40 0x7f22b41f69a0 secure :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22b002a1c0 tx=0x7f22b0008030 comp rx=0 tx=0).stop 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 74, 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19256, 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:15:37.961 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 14, 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.962 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.963 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:15:37.964 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:15:37.965 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.966 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.968 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.969 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.970 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:15:37.971 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:15:37.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:15:37.973 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f22980761d0 msgr2=0x7f22980786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22bb97e640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f22980761d0 0x7f22980786e0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f22b41f05e0 tx=0x7f22a4009040 comp rx=0 tx=0).stop 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b415d130 msgr2=0x7f22b41f5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22bb97e640 1 --2- 172.21.15.1:0/3990989237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f22b415d130 0x7f22b41f5f20 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f22ac00efc0 tx=0x7f22ac00c490 comp rx=0 tx=0).stop 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22b9f1b640 1 -- 172.21.15.1:0/3990989237 reap_dead start 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22b9f1b640 1 -- 172.21.15.1:0/3990989237 reap_dead start 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 shutdown_connections 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 >> 172.21.15.1:0/3990989237 conn(0x7f22b40747b0 msgr2=0x7f22b4151ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:37.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 shutdown_connections 2024-05-09T05:15:37.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:37.962+0000 7f22bb97e640 1 -- 172.21.15.1:0/3990989237 wait complete. 2024-05-09T05:15:37.976 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:15:37.976 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-05-09T05:15:37.976 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: ['a'] 2024-05-09T05:15:37.976 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-05-09T05:15:37.976 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:15:37.976 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:15:37.977 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:15:37.989 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:15:37.989 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-05-09T05:15:37.990 INFO:tasks.ceph.mon.a:Sent signal 19 2024-05-09T05:15:37.990 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-05-09T05:15:52.990 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-05-09T05:15:52.991 INFO:tasks.ceph.mon.a:Sent signal 18 2024-05-09T05:15:52.991 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:15:52.991 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:15:53.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.120+0000 7f808ebe3640 1 Processor -- start 2024-05-09T05:15:53.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.120+0000 7f808ebe3640 1 -- start start 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.121+0000 7f808ebe3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.121+0000 7f808ebe3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 0x7f80880cc590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.121+0000 7f808ebe3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f808815a3a0 con 0x7f80880c8180 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.121+0000 7f808ebe3640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f80880cbf60 con 0x7f8088150040 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.121+0000 7f808ebe3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f80880c5f00 con 0x7f80880cc180 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.121+0000 7f808c97f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f80880c8180 0x7f8088159c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:48860/0 (socket says 172.21.15.1:48860) 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.121+0000 7f808c97f640 1 -- 172.21.15.1:0/2814984075 learned_addr learned my addr 172.21.15.1:0/2814984075 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.121+0000 7f808d180640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 msgr2=0x7f80880cc590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.122+0000 7f808d180640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 0x7f80880cc590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.122+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.122+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:15:53.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.122+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:53.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.122+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:53.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.122+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.323+0000 7f808d180640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 msgr2=0x7f80880cc590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.323+0000 7f808d180640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 0x7f80880cc590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:15:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.323+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.323+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:15:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.323+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.323+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:53.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.323+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:15:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.724+0000 7f808d180640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 msgr2=0x7f80880cc590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.724+0000 7f808d180640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 0x7f80880cc590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:15:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.724+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.724+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:15:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.724+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.724+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:53.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:53.724+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:15:54.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:54.525+0000 7f808d180640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 msgr2=0x7f80880cc590 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:54.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:54.525+0000 7f808d180640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 0x7f80880cc590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:15:54.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:54.525+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:54.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:54.526+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:15:54.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:54.526+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:54.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:54.526+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:54.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:54.526+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:15:56.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.121+0000 7f80867fc640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 msgr2=0x7f80880cc590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:15:56.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f80867fc640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f80880cc180 0x7f80880cc590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:56.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f80867fc640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 msgr2=0x7f80880c5590 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:15:56.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f80867fc640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.196:3300/0 conn(0x7f8088150040 0x7f80880c5590 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f80867fc640 1 -- 172.21.15.1:0/2814984075 >> v1:172.21.15.1:6789/0 conn(0x7f80880c8180 legacy=0x7f8088159c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f80867fc640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f80867fc640 1 -- 172.21.15.1:0/2814984075 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f806c00c160 con 0x7f806c0058d0 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f80867fc640 1 -- 172.21.15.1:0/2814984075 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f806c00d340 con 0x7f806c0095d0 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:56.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.122+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:56.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.323+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:56.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.323+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:15:56.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.323+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:56.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.323+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:56.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.323+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:15:56.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.724+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:56.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.724+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:15:56.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.724+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:56.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.724+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:56.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:56.724+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:15:57.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:57.525+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:57.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:57.525+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:15:57.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:57.525+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:15:57.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:57.525+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:15:57.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:57.525+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:15:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2903387790 0 0) 0x7f806c00c160 con 0x7f806c0058d0 2024-05-09T05:15:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8078003520 con 0x7f806c0058d0 2024-05-09T05:15:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2041036414 0 0) 0x7f8078003520 con 0x7f806c0058d0 2024-05-09T05:15:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 msgr2=0x7f806c00bae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 --2- 172.21.15.1:0/2814984075 >> v2:172.21.15.1:3300/0 conn(0x7f806c0095d0 0x7f806c00bae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f806c00d550 con 0x7f806c0058d0 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f807c001730 con 0x7f806c0058d0 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 >> v1:172.21.15.196:6789/0 conn(0x7f806c0058d0 legacy=0x7f806c007da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f808d180640 1 -- 172.21.15.1:0/2814984075 reap_dead start 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80780039b0 0x7f8078003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.004+0000 7f80877fe640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8078004800 0x7f8078004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f80877fe640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 0x7f807800b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f806c00d340 con 0x7f80780039b0 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f807800c2c0 con 0x7f8078004800 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f807800c4c0 con 0x7f8078005330 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f808d180640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 msgr2=0x7f807800b9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f808d180640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 0x7f807800b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8078004800 0x7f8078004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 msgr2=0x7f807800b9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:15:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 0x7f807800b9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80780039b0 msgr2=0x7f8078003e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80780039b0 0x7f8078003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.005+0000 7f8087fff640 1 -- 172.21.15.1:0/2814984075 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8078003520 con 0x7f8078004800 2024-05-09T05:15:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.007+0000 7f8087fff640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8078004800 0x7f8078004c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8088074830 tx=0x7f8078005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d657f3f977489ef9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.007+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074032070 con 0x7f8078004800 2024-05-09T05:15:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.007+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f807402ccd0 con 0x7f8078004800 2024-05-09T05:15:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.007+0000 7f80877fe640 1 -- 172.21.15.1:0/2814984075 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074019960 con 0x7f8078004800 2024-05-09T05:15:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 -- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8078004800 msgr2=0x7f8078004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8078004800 0x7f8078004c10 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f8088074830 tx=0x7f8078005830 comp rx=0 tx=0).stop 2024-05-09T05:15:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 -- 172.21.15.1:0/2814984075 shutdown_connections 2024-05-09T05:15:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8078004800 0x7f8078004c10 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 0x7f807800b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 --2- 172.21.15.1:0/2814984075 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f80780039b0 0x7f8078003e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 -- 172.21.15.1:0/2814984075 >> 172.21.15.1:0/2814984075 conn(0x7f80880ce110 msgr2=0x7f80880ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 -- 172.21.15.1:0/2814984075 shutdown_connections 2024-05-09T05:15:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.008+0000 7f808ebe3640 1 -- 172.21.15.1:0/2814984075 wait complete. 2024-05-09T05:15:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.012+0000 7f808ebe3640 1 Processor -- start 2024-05-09T05:15:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.012+0000 7f808ebe3640 1 -- start start 2024-05-09T05:15:58.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.013+0000 7f808ebe3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80780039b0 0x7f808815abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.013+0000 7f808ebe3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8078004800 0x7f808815d070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.013+0000 7f808ebe3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 0x7f808814a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.013+0000 7f808ebe3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f80880c66f0 con 0x7f8078004800 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.013+0000 7f808ebe3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f80880c6410 con 0x7f80780039b0 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.013+0000 7f808ebe3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f80880c69d0 con 0x7f8078005330 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.013+0000 7f808d180640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 msgr2=0x7f808814a8e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.013+0000 7f808d180640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 0x7f808814a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8078004800 0x7f808815d070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8078004800 0x7f808815d070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53030/0 (socket says 172.21.15.1:53030) 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 -- 172.21.15.1:0/1449605849 learned_addr learned my addr 172.21.15.1:0/1449605849 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f808c97f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80780039b0 0x7f808815abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 -- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 msgr2=0x7f808814a8e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 0x7f808814a8e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 -- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80780039b0 msgr2=0x7f808815abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80780039b0 0x7f808815abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 -- 172.21.15.1:0/1449605849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f808815b110 con 0x7f8078004800 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f808c97f640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80780039b0 0x7f808815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.014+0000 7f8087fff640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8078004800 0x7f808815d070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8074019dd0 tx=0x7f80740046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.015+0000 7f80857fa640 1 -- 172.21.15.1:0/1449605849 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074042070 con 0x7f8078004800 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.015+0000 7f80857fa640 1 -- 172.21.15.1:0/1449605849 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f807400f040 con 0x7f8078004800 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.015+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8088155610 con 0x7f8078004800 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.015+0000 7f80857fa640 1 -- 172.21.15.1:0/1449605849 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074019960 con 0x7f8078004800 2024-05-09T05:15:58.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.015+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8088155b50 con 0x7f8078004800 2024-05-09T05:15:58.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.016+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8088159770 con 0x7f8078004800 2024-05-09T05:15:58.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.017+0000 7f80857fa640 1 -- 172.21.15.1:0/1449605849 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8074045070 con 0x7f8078004800 2024-05-09T05:15:58.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.018+0000 7f80857fa640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f806c076cf0 0x7f806c079200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.018+0000 7f80857fa640 1 -- 172.21.15.1:0/1449605849 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f80740b9360 con 0x7f8078004800 2024-05-09T05:15:58.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.018+0000 7f808c97f640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f806c076cf0 0x7f806c079200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.019+0000 7f808c97f640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f806c076cf0 0x7f806c079200 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f807c000f60 tx=0x7f807c007c90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.023+0000 7f80857fa640 1 -- 172.21.15.1:0/1449605849 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8088159770 con 0x7f8078004800 2024-05-09T05:15:58.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.289+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8088050870 con 0x7f8078004800 2024-05-09T05:15:58.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.289+0000 7f80857fa640 1 -- 172.21.15.1:0/1449605849 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f80740825e0 con 0x7f8078004800 2024-05-09T05:15:58.288 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":76,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:15:58.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f806c076cf0 msgr2=0x7f806c079200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f806c076cf0 0x7f806c079200 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f807c000f60 tx=0x7f807c007c90 comp rx=0 tx=0).stop 2024-05-09T05:15:58.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8078004800 msgr2=0x7f808815d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8078004800 0x7f808815d070 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8074019dd0 tx=0x7f80740046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 shutdown_connections 2024-05-09T05:15:58.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f806c076cf0 0x7f806c079200 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8078005330 0x7f808814a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8078004800 0x7f808815d070 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 --2- 172.21.15.1:0/1449605849 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f80780039b0 0x7f808815abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 >> 172.21.15.1:0/1449605849 conn(0x7f80880ce110 msgr2=0x7f80881535d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:58.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 shutdown_connections 2024-05-09T05:15:58.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.291+0000 7f808ebe3640 1 -- 172.21.15.1:0/1449605849 wait complete. 2024-05-09T05:15:58.301 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:15:58.301 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:15:58.301 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:15:58.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.418+0000 7fcec9ab8640 1 Processor -- start 2024-05-09T05:15:58.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.418+0000 7fcec9ab8640 1 -- start start 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.419+0000 7fcec9ab8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcec40cf360 0x7fcec40b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.419+0000 7fcec9ab8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcec40b4570 0x7fcec4156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.419+0000 7fcec9ab8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcec40ce5c0 con 0x7fcec415c9a0 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.419+0000 7fcec9ab8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcec415d4c0 con 0x7fcec40cf360 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.419+0000 7fcec9ab8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fcec415d7a0 con 0x7fcec40b4570 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.420+0000 7fcec37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcec40cf360 0x7fcec40b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.420+0000 7fcec37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcec40cf360 0x7fcec40b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53038/0 (socket says 172.21.15.1:53038) 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.420+0000 7fcec37fe640 1 -- 172.21.15.1:0/1124453172 learned_addr learned my addr 172.21.15.1:0/1124453172 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.420+0000 7fcec2ffd640 1 --2- 172.21.15.1:0/1124453172 >> v2:172.21.15.196:3300/0 conn(0x7fcec40b4570 0x7fcec4156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.420+0000 7fcec37fe640 1 -- 172.21.15.1:0/1124453172 >> v2:172.21.15.196:3300/0 conn(0x7fcec40b4570 msgr2=0x7fcec4156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.420+0000 7fcec37fe640 1 --2- 172.21.15.1:0/1124453172 >> v2:172.21.15.196:3300/0 conn(0x7fcec40b4570 0x7fcec4156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.420+0000 7fcec37fe640 1 -- 172.21.15.1:0/1124453172 >> v1:172.21.15.196:6789/0 conn(0x7fcec415c9a0 legacy=0x7fcec40cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.420+0000 7fcec37fe640 1 -- 172.21.15.1:0/1124453172 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec40ce880 con 0x7fcec40cf360 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.421+0000 7fcec37fe640 1 --2- 172.21.15.1:0/1124453172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40cf360 0x7fcec40b4b50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcec40cf8c0 tx=0x7fceb402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df6d80631e5d7cef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.421+0000 7fcec27fc640 1 -- 172.21.15.1:0/1124453172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4037070 con 0x7fcec40cf360 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.421+0000 7fcec27fc640 1 -- 172.21.15.1:0/1124453172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb402ae50 con 0x7fcec40cf360 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.421+0000 7fcec27fc640 1 -- 172.21.15.1:0/1124453172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4032860 con 0x7fcec40cf360 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.422+0000 7fcec9ab8640 1 -- 172.21.15.1:0/1124453172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40cf360 msgr2=0x7fcec40b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.422+0000 7fcec9ab8640 1 --2- 172.21.15.1:0/1124453172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40cf360 0x7fcec40b4b50 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fcec40cf8c0 tx=0x7fceb402a300 comp rx=0 tx=0).stop 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.422+0000 7fcec9ab8640 1 -- 172.21.15.1:0/1124453172 shutdown_connections 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.422+0000 7fcec9ab8640 1 --2- 172.21.15.1:0/1124453172 >> v2:172.21.15.196:3300/0 conn(0x7fcec40b4570 0x7fcec4156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.422+0000 7fcec9ab8640 1 --2- 172.21.15.1:0/1124453172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40cf360 0x7fcec40b4b50 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.422+0000 7fcec9ab8640 1 -- 172.21.15.1:0/1124453172 >> 172.21.15.1:0/1124453172 conn(0x7fcec406de10 msgr2=0x7fcec406e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.422+0000 7fcec9ab8640 1 -- 172.21.15.1:0/1124453172 shutdown_connections 2024-05-09T05:15:58.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.422+0000 7fcec9ab8640 1 -- 172.21.15.1:0/1124453172 wait complete. 2024-05-09T05:15:58.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.423+0000 7fcec9ab8640 1 Processor -- start 2024-05-09T05:15:58.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.424+0000 7fcec9ab8640 1 -- start start 2024-05-09T05:15:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec9ab8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40b4570 0x7fcec41f2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec9ab8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcec415c9a0 0x7fcec4159080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec9ab8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec41e7c80 0x7fcec41e8860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec9ab8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40b5090 con 0x7fcec40b4570 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec9ab8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcec40cee40 con 0x7fcec41e7c80 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec9ab8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcec4157090 con 0x7fcec415c9a0 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40b4570 0x7fcec41f2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec37fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40b4570 0x7fcec41f2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53042/0 (socket says 172.21.15.1:53042) 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec37fe640 1 -- 172.21.15.1:0/3889257111 learned_addr learned my addr 172.21.15.1:0/3889257111 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec2ffd640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcec415c9a0 msgr2=0x7fcec4159080 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec2ffd640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcec415c9a0 0x7fcec4159080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec3fff640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec41e7c80 0x7fcec41e8860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec37fe640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcec415c9a0 msgr2=0x7fcec4159080 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec37fe640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcec415c9a0 0x7fcec4159080 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec37fe640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec41e7c80 msgr2=0x7fcec41e8860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec37fe640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcec41e7c80 0x7fcec41e8860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.425+0000 7fcec37fe640 1 -- 172.21.15.1:0/3889257111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec41595c0 con 0x7fcec40b4570 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.426+0000 7fcec37fe640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40b4570 0x7fcec41f2470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fceb402a340 tx=0x7fceb40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.426+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4041070 con 0x7fcec40b4570 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.426+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fceb403c680 con 0x7fcec40b4570 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.426+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec41e6660 con 0x7fcec40b4570 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.426+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb4037040 con 0x7fcec40b4570 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.426+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec41e6c20 con 0x7fcec40b4570 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.427+0000 7fcec9ab8640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec41e6e60 0x7fcec41552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.427+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcec41e6410 con 0x7fcec41e6e60 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.427+0000 7fcec2ffd640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec41e6e60 0x7fcec41552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.427+0000 7fcec2ffd640 1 -- 172.21.15.1:0/3889257111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcec4050870 con 0x7fcec41e6e60 2024-05-09T05:15:58.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.427+0000 7fcec2ffd640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec41e6e60 0x7fcec41552d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcec415a750 tx=0x7fceb800ec00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.427+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8017070 con 0x7fcec41e6e60 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.427+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fceb4044070 con 0x7fcec40b4570 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.428+0000 7fceabfff640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fceb00761d0 0x7fceb00786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.428+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fceb40b7cb0 con 0x7fcec40b4570 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.428+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fceb8004020 con 0x7fcec41e6e60 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.428+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec41e6e60 msgr2=0x7fcec41552d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.428+0000 7fceabfff640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec41e6e60 0x7fcec41552d0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcec415a750 tx=0x7fceb800ec00 comp rx=0 tx=0).stop 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.428+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec41e6e60 msgr2=0x7fcec41552d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.428+0000 7fceabfff640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec41e6e60 0x7fcec41552d0 secure :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcec415a750 tx=0x7fceb800ec00 comp rx=0 tx=0).stop 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.428+0000 7fcec3fff640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fceb00761d0 0x7fceb00786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.429+0000 7fcec3fff640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fceb00761d0 0x7fceb00786e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcebc007fd0 tx=0x7fcebc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.435+0000 7fcec9ab8640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec4155810 0x7fcec41f59e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.435+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcec4155c20 con 0x7fcec4155810 2024-05-09T05:15:58.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.435+0000 7fcec2ffd640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec4155810 0x7fcec41f59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.435+0000 7fcec2ffd640 1 -- 172.21.15.1:0/3889257111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcec40cdf30 con 0x7fcec4155810 2024-05-09T05:15:58.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.435+0000 7fcec2ffd640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec4155810 0x7fcec41f59e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009fd0 tx=0x7fceb8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.435+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb800ccf0 con 0x7fcec4155810 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5052 (secure 0 0 0) 0x7fceb800ccf0 con 0x7fcec4155810 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec4155810 msgr2=0x7fcec41f59e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fceabfff640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec4155810 0x7fcec41f59e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fceb8009fd0 tx=0x7fceb8004730 comp rx=0 tx=0).stop 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fceabfff640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec4155810 msgr2=0x7fcec41f59e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fceabfff640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec4155810 0x7fcec41f59e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:58.435 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 76, 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 160630, 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:58.436 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:58.437 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:15:58.438 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:58.439 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.440 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.441 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.442 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:58.443 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:58.444 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:15:58.445 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.446 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.447 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:15:58.448 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:15:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fceb00761d0 msgr2=0x7fceb00786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fceb00761d0 0x7fceb00786e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fcebc007fd0 tx=0x7fcebc009040 comp rx=0 tx=0).stop 2024-05-09T05:15:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40b4570 msgr2=0x7fcec41f2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40b4570 0x7fcec41f2470 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fceb402a340 tx=0x7fceb40046c0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec3fff640 1 -- 172.21.15.1:0/3889257111 reap_dead start 2024-05-09T05:15:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 shutdown_connections 2024-05-09T05:15:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 --2- 172.21.15.1:0/3889257111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcec40b4570 0x7fcec41f2470 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 >> 172.21.15.1:0/3889257111 conn(0x7fcec406de10 msgr2=0x7fcec40b52a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 shutdown_connections 2024-05-09T05:15:58.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.436+0000 7fcec9ab8640 1 -- 172.21.15.1:0/3889257111 wait complete. 2024-05-09T05:15:58.450 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:15:58.450 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:15:58.451 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:15:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.567+0000 7f5cccec0640 1 Processor -- start 2024-05-09T05:15:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.567+0000 7f5cccec0640 1 -- start start 2024-05-09T05:15:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.568+0000 7f5cccec0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cc80c69a0 0x7f5cc8160a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.568+0000 7f5cccec0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5cc806e1d0 0x7f5cc8156ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cccec0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806d800 0x7f5cc815c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cccec0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc815cd50 con 0x7f5cc80c69a0 2024-05-09T05:15:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cccec0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f5cc80cb610 con 0x7f5cc806e1d0 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cccec0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc80cb050 con 0x7f5cc806d800 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cc80c69a0 0x7f5cc8160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc6d9d640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806d800 msgr2=0x7f5cc815c810 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc6d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806d800 0x7f5cc815c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5cc80c69a0 0x7f5cc8160a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53076/0 (socket says 172.21.15.1:53076) 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc659c640 1 -- 172.21.15.1:0/4072070269 learned_addr learned my addr 172.21.15.1:0/4072070269 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc5d9b640 1 --2- 172.21.15.1:0/4072070269 >> v2:172.21.15.196:3300/0 conn(0x7f5cc806e1d0 0x7f5cc8156ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc659c640 1 -- 172.21.15.1:0/4072070269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806d800 msgr2=0x7f5cc815c810 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc659c640 1 --2- 172.21.15.1:0/4072070269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806d800 0x7f5cc815c810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc659c640 1 -- 172.21.15.1:0/4072070269 >> v2:172.21.15.196:3300/0 conn(0x7f5cc806e1d0 msgr2=0x7f5cc8156ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc659c640 1 --2- 172.21.15.1:0/4072070269 >> v2:172.21.15.196:3300/0 conn(0x7f5cc806e1d0 0x7f5cc8156ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.569+0000 7f5cc659c640 1 -- 172.21.15.1:0/4072070269 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc80cb330 con 0x7f5cc80c69a0 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.570+0000 7f5cc659c640 1 --2- 172.21.15.1:0/4072070269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc80c69a0 0x7f5cc8160a10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5cc80cd530 tx=0x7f5cb802a6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3998dc76df015dc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.570+0000 7f5cc559a640 1 -- 172.21.15.1:0/4072070269 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8036070 con 0x7f5cc80c69a0 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.570+0000 7f5cc559a640 1 -- 172.21.15.1:0/4072070269 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cb8031600 con 0x7f5cc80c69a0 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.570+0000 7f5cc559a640 1 -- 172.21.15.1:0/4072070269 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb80319d0 con 0x7f5cc80c69a0 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 -- 172.21.15.1:0/4072070269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc80c69a0 msgr2=0x7f5cc8160a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 --2- 172.21.15.1:0/4072070269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc80c69a0 0x7f5cc8160a10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5cc80cd530 tx=0x7f5cb802a6c0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 -- 172.21.15.1:0/4072070269 shutdown_connections 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 --2- 172.21.15.1:0/4072070269 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806d800 0x7f5cc815c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 --2- 172.21.15.1:0/4072070269 >> v2:172.21.15.196:3300/0 conn(0x7f5cc806e1d0 0x7f5cc8156ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 --2- 172.21.15.1:0/4072070269 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc80c69a0 0x7f5cc8160a10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 -- 172.21.15.1:0/4072070269 >> 172.21.15.1:0/4072070269 conn(0x7f5cc8068830 msgr2=0x7f5cc8068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 -- 172.21.15.1:0/4072070269 shutdown_connections 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.571+0000 7f5cccec0640 1 -- 172.21.15.1:0/4072070269 wait complete. 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.573+0000 7f5cccec0640 1 Processor -- start 2024-05-09T05:15:58.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.573+0000 7f5cccec0640 1 -- start start 2024-05-09T05:15:58.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cccec0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc806d800 0x7f5cc814b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cccec0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806e1d0 0x7f5cc81f63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cccec0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc81f2e30 0x7f5cc81f68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cccec0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81604f0 con 0x7f5cc81f2e30 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cccec0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc8160210 con 0x7f5cc806d800 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cccec0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5cc806dc10 con 0x7f5cc806e1d0 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc81f2e30 0x7f5cc81f68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc5d9b640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806e1d0 msgr2=0x7f5cc81f63a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc81f2e30 0x7f5cc81f68e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:53092/0 (socket says 172.21.15.1:53092) 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc6d9d640 1 -- 172.21.15.1:0/589443025 learned_addr learned my addr 172.21.15.1:0/589443025 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc5d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806e1d0 0x7f5cc81f63a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc659c640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc806d800 0x7f5cc814b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc6d9d640 1 -- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806e1d0 msgr2=0x7f5cc81f63a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc6d9d640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5cc806e1d0 0x7f5cc81f63a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc6d9d640 1 -- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc806d800 msgr2=0x7f5cc814b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc6d9d640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc806d800 0x7f5cc814b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.574+0000 7f5cc6d9d640 1 -- 172.21.15.1:0/589443025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc814fe20 con 0x7f5cc81f2e30 2024-05-09T05:15:58.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.575+0000 7f5cc6d9d640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc81f2e30 0x7f5cc81f68e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000b410 tx=0x7f5cc000b8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.575+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0017070 con 0x7f5cc81f2e30 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.575+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5cc0003d70 con 0x7f5cc81f2e30 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.575+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0006310 con 0x7f5cc81f2e30 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.575+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc815b650 con 0x7f5cc81f2e30 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.575+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc815b1a0 con 0x7f5cc81f2e30 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.576+0000 7f5cccec0640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc81f6e20 0x7f5cc815bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.576+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81f7230 con 0x7f5cc81f6e20 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.576+0000 7f5cc659c640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc81f6e20 0x7f5cc815bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.576+0000 7f5cc659c640 1 -- 172.21.15.1:0/589443025 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5cc815b480 con 0x7f5cc81f6e20 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.576+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5cc00064b0 con 0x7f5cc81f2e30 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cc659c640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc81f6e20 0x7f5cc815bc30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8074770 tx=0x7f5cb8004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cbeffd640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca8076280 0x7f5ca8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5cc00966a0 con 0x7f5cc81f2e30 2024-05-09T05:15:58.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cc5d9b640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca8076280 0x7f5ca8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8042070 con 0x7f5cc81f6e20 2024-05-09T05:15:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5cb8037040 con 0x7f5cc81f6e20 2024-05-09T05:15:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc81f6e20 msgr2=0x7f5cc815bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cbeffd640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc81f6e20 0x7f5cc815bc30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f5cc8074770 tx=0x7f5cb8004a40 comp rx=0 tx=0).stop 2024-05-09T05:15:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc81f6e20 msgr2=0x7f5cc815bc30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.577+0000 7f5cbeffd640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc81f6e20 0x7f5cc815bc30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.578+0000 7f5cc5d9b640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca8076280 0x7f5ca8078790 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5cc814bb70 tx=0x7f5cb0009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.584+0000 7f5cccec0640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc8149e30 0x7f5cc814a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:15:58.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.584+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81f35a0 con 0x7f5cc8149e30 2024-05-09T05:15:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.584+0000 7f5cc659c640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc8149e30 0x7f5cc814a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:15:58.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.585+0000 7f5cc659c640 1 -- 172.21.15.1:0/589443025 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5cc8152030 con 0x7f5cc8149e30 2024-05-09T05:15:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.585+0000 7f5cc659c640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc8149e30 0x7f5cc814a240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5cc806d200 tx=0x7f5cb802f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:15:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.585+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cb8036040 con 0x7f5cc8149e30 2024-05-09T05:15:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.585+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4880 (secure 0 0 0) 0x7f5cb8003e80 con 0x7f5cc8149e30 2024-05-09T05:15:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.585+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc8149e30 msgr2=0x7f5cc814a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.585+0000 7f5cbeffd640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc8149e30 0x7f5cc814a240 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5cc806d200 tx=0x7f5cb802f000 comp rx=0 tx=0).stop 2024-05-09T05:15:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.585+0000 7f5cbeffd640 1 -- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc8149e30 msgr2=0x7f5cc814a240 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:15:58.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.585+0000 7f5cbeffd640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5cc8149e30 0x7f5cc814a240 secure :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f5cc806d200 tx=0x7f5cb802f000 comp rx=0 tx=0).stop 2024-05-09T05:15:58.585 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:15:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:15:58.585 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 76, 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 220864, 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:15:58.586 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:15:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:58.587 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:58.588 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:15:58.589 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:15:58.590 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:15:58.591 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.592 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.593 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:58.594 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:15:58.595 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:15:58.596 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.597 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:15:58.598 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca8076280 msgr2=0x7f5ca8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cccec0640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5ca8076280 0x7f5ca8078790 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f5cc814bb70 tx=0x7f5cb0009190 comp rx=0 tx=0).stop 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc81f2e30 msgr2=0x7f5cc81f68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cccec0640 1 --2- 172.21.15.1:0/589443025 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5cc81f2e30 0x7f5cc81f68e0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5cc000b410 tx=0x7f5cc000b8d0 comp rx=0 tx=0).stop 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cc6d9d640 1 -- 172.21.15.1:0/589443025 reap_dead start 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cc6d9d640 1 -- 172.21.15.1:0/589443025 reap_dead start 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 shutdown_connections 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 >> 172.21.15.1:0/589443025 conn(0x7f5cc8068830 msgr2=0x7f5cc8068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 shutdown_connections 2024-05-09T05:15:58.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:15:58.586+0000 7f5cccec0640 1 -- 172.21.15.1:0/589443025 wait complete. 2024-05-09T05:15:58.601 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:15:58.601 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:16:04.101 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:16:04.104 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:16:18.601 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:16:18.601 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-05-09T05:16:18.602 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T05:16:18.602 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T05:16:18.605 INFO:tasks.ceph.mon.c:Started 2024-05-09T05:16:18.605 INFO:tasks.mon_thrash:Sending STOP to mon a 2024-05-09T05:16:18.605 INFO:tasks.ceph.mon.a:Sent signal 19 2024-05-09T05:16:18.606 INFO:tasks.mon_thrash.mon_thrasher:waiting for 15.0 secs to unfreeze mons 2024-05-09T05:16:33.606 INFO:tasks.mon_thrash:Sending CONT to mon a 2024-05-09T05:16:33.607 INFO:tasks.ceph.mon.a:Sent signal 18 2024-05-09T05:16:33.607 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:16:33.607 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:16:33.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.728+0000 7f23d558d640 1 Processor -- start 2024-05-09T05:16:33.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.728+0000 7f23d558d640 1 -- start start 2024-05-09T05:16:33.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23d558d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 0x7f23d00d1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:33.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23d558d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d0156b30 0x7f23d015ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:33.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23d558d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23d015ff70 con 0x7f23d006f440 2024-05-09T05:16:33.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23d558d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f23d00cd180 con 0x7f23d00cebc0 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23d558d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23d0160250 con 0x7f23d0156b30 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23ce7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 0x7f23d00d1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23ce7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 0x7f23d00d1530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46000/0 (socket says 172.21.15.1:46000) 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23ce7fc640 1 -- 172.21.15.1:0/1354875214 learned_addr learned my addr 172.21.15.1:0/1354875214 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23cf7fe640 1 --2- 172.21.15.1:0/1354875214 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d0156b30 0x7f23d015ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23ce7fc640 1 -- 172.21.15.1:0/1354875214 >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 msgr2=0x7f23d00d1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23ce7fc640 1 -- 172.21.15.1:0/1354875214 >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 msgr2=0x7f23d00d1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23ce7fc640 1 --2- 172.21.15.1:0/1354875214 >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 0x7f23d00d1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23ce7fc640 1 --2- 172.21.15.1:0/1354875214 >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 0x7f23d00d1530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23cf7fe640 1 -- 172.21.15.1:0/1354875214 >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 msgr2=0x7f23d00d1530 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23cf7fe640 1 --2- 172.21.15.1:0/1354875214 >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 0x7f23d00d1530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23cf7fe640 1 -- 172.21.15.1:0/1354875214 >> v1:172.21.15.196:6789/0 conn(0x7f23d006f440 legacy=0x7f23d00ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:33.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.730+0000 7f23cf7fe640 1 -- 172.21.15.1:0/1354875214 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d0160530 con 0x7f23d0156b30 2024-05-09T05:16:33.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.731+0000 7f23cf7fe640 1 --2- 172.21.15.1:0/1354875214 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d0156b30 0x7f23d015ca30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009d90 tx=0x7f23c802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f99edbbe7275501 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.731+0000 7f23cdffb640 1 -- 172.21.15.1:0/1354875214 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c8007740 con 0x7f23d0156b30 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.731+0000 7f23cdffb640 1 -- 172.21.15.1:0/1354875214 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23c80078e0 con 0x7f23d0156b30 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.731+0000 7f23cdffb640 1 -- 172.21.15.1:0/1354875214 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c8007cb0 con 0x7f23d0156b30 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.732+0000 7f23d558d640 1 -- 172.21.15.1:0/1354875214 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d0156b30 msgr2=0x7f23d015ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.732+0000 7f23d558d640 1 --2- 172.21.15.1:0/1354875214 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d0156b30 0x7f23d015ca30 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f23c8009d90 tx=0x7f23c802a660 comp rx=0 tx=0).stop 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.732+0000 7f23d558d640 1 -- 172.21.15.1:0/1354875214 shutdown_connections 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.732+0000 7f23d558d640 1 --2- 172.21.15.1:0/1354875214 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d0156b30 0x7f23d015ca30 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.732+0000 7f23d558d640 1 --2- 172.21.15.1:0/1354875214 >> v2:172.21.15.1:3300/0 conn(0x7f23d00cebc0 0x7f23d00d1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.732+0000 7f23d558d640 1 -- 172.21.15.1:0/1354875214 >> 172.21.15.1:0/1354875214 conn(0x7f23d00b3670 msgr2=0x7f23d00b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.732+0000 7f23d558d640 1 -- 172.21.15.1:0/1354875214 shutdown_connections 2024-05-09T05:16:33.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.732+0000 7f23d558d640 1 -- 172.21.15.1:0/1354875214 wait complete. 2024-05-09T05:16:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.734+0000 7f23d558d640 1 Processor -- start 2024-05-09T05:16:33.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.734+0000 7f23d558d640 1 -- start start 2024-05-09T05:16:33.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23d558d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d006f440 0x7f23d015dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:33.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23d558d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 0x7f23d015ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:33.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23d558d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 0x7f23d015f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:33.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23d558d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f23d00cf940 con 0x7f23d014f400 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23d558d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23d0156f40 con 0x7f23d0156b30 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23d558d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23d00cfac0 con 0x7f23d006f440 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23cf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 0x7f23d015f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23cf7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 0x7f23d015f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46008/0 (socket says 172.21.15.1:46008) 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.735+0000 7f23cf7fe640 1 -- 172.21.15.1:0/2260186540 learned_addr learned my addr 172.21.15.1:0/2260186540 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ceffd640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d006f440 0x7f23d015dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23cf7fe640 1 -- 172.21.15.1:0/2260186540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 msgr2=0x7f23d015f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23cf7fe640 1 -- 172.21.15.1:0/2260186540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 msgr2=0x7f23d015f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23cf7fe640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 0x7f23d015f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ce7fc640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 0x7f23d015ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23cf7fe640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 0x7f23d015f120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ce7fc640 1 -- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 msgr2=0x7f23d015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ce7fc640 1 -- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 msgr2=0x7f23d015ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:16:33.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ce7fc640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 0x7f23d015ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ce7fc640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 0x7f23d015ebe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ceffd640 1 -- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 msgr2=0x7f23d015ebe0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ceffd640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 0x7f23d015ebe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ceffd640 1 -- 172.21.15.1:0/2260186540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 msgr2=0x7f23d015f120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ceffd640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 0x7f23d015f120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ceffd640 1 -- 172.21.15.1:0/2260186540 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23d01f34f0 con 0x7f23d006f440 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23ceffd640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d006f440 0x7f23d015dd10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23c000c3f0 tx=0x7f23c000c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23b77fe640 1 -- 172.21.15.1:0/2260186540 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c0017070 con 0x7f23d006f440 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23b77fe640 1 -- 172.21.15.1:0/2260186540 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23c00052d0 con 0x7f23d006f440 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23d015e250 con 0x7f23d006f440 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23b77fe640 1 -- 172.21.15.1:0/2260186540 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23c0003640 con 0x7f23d006f440 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.736+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f23d01f2c50 con 0x7f23d006f440 2024-05-09T05:16:33.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.737+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23d0050870 con 0x7f23d006f440 2024-05-09T05:16:33.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.738+0000 7f23b77fe640 1 -- 172.21.15.1:0/2260186540 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f23c001a070 con 0x7f23d006f440 2024-05-09T05:16:33.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.739+0000 7f23b77fe640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23bc076300 0x7f23bc078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.739+0000 7f23b77fe640 1 -- 172.21.15.1:0/2260186540 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f23c0096ac0 con 0x7f23d006f440 2024-05-09T05:16:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.739+0000 7f23ce7fc640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23bc076300 0x7f23bc078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:33.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.739+0000 7f23ce7fc640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23bc076300 0x7f23bc078810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f23d00d1470 tx=0x7f23c4008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:33.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:33.744+0000 7f23b77fe640 1 -- 172.21.15.1:0/2260186540 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f23c003a020 con 0x7f23d006f440 2024-05-09T05:16:34.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.016+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f23d00b7ac0 con 0x7f23d006f440 2024-05-09T05:16:34.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.017+0000 7f23b77fe640 1 -- 172.21.15.1:0/2260186540 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f23c0068400 con 0x7f23d006f440 2024-05-09T05:16:34.015 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":78,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":10,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23bc076300 msgr2=0x7f23bc078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23bc076300 0x7f23bc078810 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f23d00d1470 tx=0x7f23c4008040 comp rx=0 tx=0).stop 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d006f440 msgr2=0x7f23d015dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d006f440 0x7f23d015dd10 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f23c000c3f0 tx=0x7f23c000c7f0 comp rx=0 tx=0).stop 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 shutdown_connections 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23bc076300 0x7f23bc078810 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23d0156b30 0x7f23d015ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23d014f400 0x7f23d015f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:34.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 --2- 172.21.15.1:0/2260186540 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23d006f440 0x7f23d015dd10 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:34.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 >> 172.21.15.1:0/2260186540 conn(0x7f23d00b3670 msgr2=0x7f23d006e5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:34.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 shutdown_connections 2024-05-09T05:16:34.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:34.019+0000 7f23d558d640 1 -- 172.21.15.1:0/2260186540 wait complete. 2024-05-09T05:16:34.247 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:16:34.250 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:16:37.030 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:16:37.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.162+0000 7f79c9f4d640 1 Processor -- start 2024-05-09T05:16:37.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.162+0000 7f79c9f4d640 1 -- start start 2024-05-09T05:16:37.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.163+0000 7f79c9f4d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:37.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.163+0000 7f79c9f4d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79c40cc450 con 0x7f79c40cbd00 2024-05-09T05:16:37.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.163+0000 7f79c9f4d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f79c415b230 con 0x7f79c40c4940 2024-05-09T05:16:37.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.163+0000 7f79c9f4d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f79c40caf30 con 0x7f79c4163e70 2024-05-09T05:16:37.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.163+0000 7f79c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:37.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.163+0000 7f79c3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46018/0 (socket says 172.21.15.1:46018) 2024-05-09T05:16:37.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.163+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 learned_addr learned my addr 172.21.15.1:0/3596907345 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:37.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.164+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:16:37.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.164+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:16:37.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.164+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:16:37.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.164+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:16:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.365+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:37.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.365+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:16:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.365+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:16:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.365+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:16:37.364 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.365+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:16:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.766+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.766+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:16:37.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.766+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:16:37.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.766+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:16:37.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:37.766+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:16:38.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.567+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:38.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.567+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:16:38.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.567+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:16:38.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.567+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:16:38.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.567+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:16:38.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.618+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1812377563 0 0) 0x7f79c415b230 con 0x7f79c40c4940 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.618+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f79ac003520 con 0x7f79c40c4940 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 532858541 0 0) 0x7f79ac003520 con 0x7f79c40c4940 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 msgr2=0x7f79c4166290 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 --2- 172.21.15.1:0/3596907345 >> v2:172.21.15.1:3300/0 conn(0x7f79c4163e70 0x7f79c4166290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 >> v1:172.21.15.1:6789/0 conn(0x7f79c40cbd00 legacy=0x7f79c4072f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c40cb180 con 0x7f79c40c4940 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f79b40016f0 con 0x7f79c40c4940 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 >> v1:172.21.15.196:6789/0 conn(0x7f79c40c4940 legacy=0x7f79c40ccc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79ac003aa0 0x7f79ac003f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ac004800 0x7f79ac004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac005350 0x7f79ac00ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:38.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f79c40caf30 con 0x7f79ac004800 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f79ac00c2e0 con 0x7f79ac005350 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f79ac00c4e0 con 0x7f79ac003aa0 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c37fe640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79ac003aa0 0x7f79ac003f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.619+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac005350 0x7f79ac00ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.620+0000 7f79c37fe640 1 -- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac005350 msgr2=0x7f79ac00ba00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.620+0000 7f79c37fe640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac005350 0x7f79ac00ba00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.620+0000 7f79c37fe640 1 -- 172.21.15.1:0/3596907345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ac004800 msgr2=0x7f79ac004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.620+0000 7f79c37fe640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79ac004800 0x7f79ac004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.620+0000 7f79c37fe640 1 -- 172.21.15.1:0/3596907345 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79ac003520 con 0x7f79ac003aa0 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.620+0000 7f79c3fff640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac005350 0x7f79ac00ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:16:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.620+0000 7f79c3fff640 1 -- 172.21.15.1:0/3596907345 reap_dead start 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.621+0000 7f79c37fe640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79ac003aa0 0x7f79ac003f10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f79b4007f40 tx=0x7f79ac005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a490bfd9a406775d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.621+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4036070 con 0x7f79ac003aa0 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.621+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b4031620 con 0x7f79ac003aa0 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.621+0000 7f79c27fc640 1 -- 172.21.15.1:0/3596907345 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4007b90 con 0x7f79ac003aa0 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.622+0000 7f79c9f4d640 1 -- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79ac003aa0 msgr2=0x7f79ac003f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.622+0000 7f79c9f4d640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79ac003aa0 0x7f79ac003f10 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f79b4007f40 tx=0x7f79ac005830 comp rx=0 tx=0).stop 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.622+0000 7f79c9f4d640 1 -- 172.21.15.1:0/3596907345 shutdown_connections 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.622+0000 7f79c9f4d640 1 --2- 172.21.15.1:0/3596907345 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79ac003aa0 0x7f79ac003f10 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.622+0000 7f79c9f4d640 1 -- 172.21.15.1:0/3596907345 >> 172.21.15.1:0/3596907345 conn(0x7f79c4068830 msgr2=0x7f79c4068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.622+0000 7f79c9f4d640 1 -- 172.21.15.1:0/3596907345 shutdown_connections 2024-05-09T05:16:38.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.622+0000 7f79c9f4d640 1 -- 172.21.15.1:0/3596907345 wait complete. 2024-05-09T05:16:38.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.624+0000 7f79c9f4d640 1 Processor -- start 2024-05-09T05:16:38.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.625+0000 7f79c9f4d640 1 -- start start 2024-05-09T05:16:38.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c9f4d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac003aa0 0x7f79c41f3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:38.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c9f4d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79c40b24f0 0x7f79c406f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:38.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c9f4d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c4159ba0 0x7f79c4146ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:38.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c9f4d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f79c4154b60 con 0x7f79c4159ba0 2024-05-09T05:16:38.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c9f4d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f79c415af20 con 0x7f79ac003aa0 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c9f4d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f79c40cb460 con 0x7f79c40b24f0 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c37fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac003aa0 0x7f79c41f3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c2ffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79c40b24f0 0x7f79c406f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c4159ba0 0x7f79c4146ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c37fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac003aa0 0x7f79c41f3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:53506/0 (socket says 172.21.15.1:53506) 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.626+0000 7f79c37fe640 1 -- 172.21.15.1:0/552209691 learned_addr learned my addr 172.21.15.1:0/552209691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.627+0000 7f79c37fe640 1 -- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79c40b24f0 msgr2=0x7f79c406f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.627+0000 7f79c37fe640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79c40b24f0 0x7f79c406f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.627+0000 7f79c37fe640 1 -- 172.21.15.1:0/552209691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c4159ba0 msgr2=0x7f79c4146ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.627+0000 7f79c37fe640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c4159ba0 0x7f79c4146ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.627+0000 7f79c37fe640 1 -- 172.21.15.1:0/552209691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79c41f3980 con 0x7f79ac003aa0 2024-05-09T05:16:38.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.627+0000 7f79c2ffd640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79c40b24f0 0x7f79c406f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:16:38.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.627+0000 7f79c37fe640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac003aa0 0x7f79c41f3440 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f79b402ab50 tx=0x7f79c40cc290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.630+0000 7f79abfff640 1 -- 172.21.15.1:0/552209691 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4040020 con 0x7f79ac003aa0 2024-05-09T05:16:38.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.630+0000 7f79abfff640 1 -- 172.21.15.1:0/552209691 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f79b40319d0 con 0x7f79ac003aa0 2024-05-09T05:16:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.630+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79c4163590 con 0x7f79ac003aa0 2024-05-09T05:16:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.630+0000 7f79abfff640 1 -- 172.21.15.1:0/552209691 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f79b4036040 con 0x7f79ac003aa0 2024-05-09T05:16:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.630+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79c4163b30 con 0x7f79ac003aa0 2024-05-09T05:16:38.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.631+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79c41f3b50 con 0x7f79ac003aa0 2024-05-09T05:16:38.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.632+0000 7f79abfff640 1 -- 172.21.15.1:0/552209691 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f79b4043020 con 0x7f79ac003aa0 2024-05-09T05:16:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.632+0000 7f79abfff640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79b00763d0 0x7f79b00788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.633+0000 7f79abfff640 1 -- 172.21.15.1:0/552209691 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f79b40b77e0 con 0x7f79ac003aa0 2024-05-09T05:16:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.633+0000 7f79c2ffd640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79b00763d0 0x7f79b00788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:38.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.633+0000 7f79c2ffd640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79b00763d0 0x7f79b00788e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f79c415ac00 tx=0x7f79b8007db0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:38.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.638+0000 7f79abfff640 1 -- 172.21.15.1:0/552209691 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f79c41f3b50 con 0x7f79ac003aa0 2024-05-09T05:16:38.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.906+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f79c4050870 con 0x7f79ac003aa0 2024-05-09T05:16:38.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.907+0000 7f79abfff640 1 -- 172.21.15.1:0/552209691 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f79b404a030 con 0x7f79ac003aa0 2024-05-09T05:16:38.905 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":80,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:16:38.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79b00763d0 msgr2=0x7f79b00788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79b00763d0 0x7f79b00788e0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f79c415ac00 tx=0x7f79b8007db0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac003aa0 msgr2=0x7f79c41f3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac003aa0 0x7f79c41f3440 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f79b402ab50 tx=0x7f79c40cc290 comp rx=0 tx=0).stop 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 shutdown_connections 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f79b00763d0 0x7f79b00788e0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f79c4159ba0 0x7f79c4146ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f79c40b24f0 0x7f79c406f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 --2- 172.21.15.1:0/552209691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f79ac003aa0 0x7f79c41f3440 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 >> 172.21.15.1:0/552209691 conn(0x7f79c4068830 msgr2=0x7f79c4068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 shutdown_connections 2024-05-09T05:16:38.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:38.909+0000 7f79c9f4d640 1 -- 172.21.15.1:0/552209691 wait complete. 2024-05-09T05:16:41.920 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:16:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.048+0000 7f6954e8b640 1 Processor -- start 2024-05-09T05:16:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.048+0000 7f6954e8b640 1 -- start start 2024-05-09T05:16:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f6954e8b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f69500d16b0 0x7f69500d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f6954e8b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f695006f1b0 0x7f69500ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f6954e8b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f695006f6d0 con 0x7f695015ca90 2024-05-09T05:16:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f6954e8b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f69500cd0e0 con 0x7f69500d16b0 2024-05-09T05:16:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f6954e8b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f69500cc4d0 con 0x7f695006f1b0 2024-05-09T05:16:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f694e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f69500d16b0 0x7f69500d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f694e59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f69500d16b0 0x7f69500d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44086/0 (socket says 172.21.15.1:44086) 2024-05-09T05:16:42.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f694e59c640 1 -- 172.21.15.1:0/1206665788 learned_addr learned my addr 172.21.15.1:0/1206665788 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:42.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f694dd9b640 1 --2- 172.21.15.1:0/1206665788 >> v2:172.21.15.196:3300/0 conn(0x7f695006f1b0 0x7f69500ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f694e59c640 1 -- 172.21.15.1:0/1206665788 >> v2:172.21.15.196:3300/0 conn(0x7f695006f1b0 msgr2=0x7f69500ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:42.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f694e59c640 1 --2- 172.21.15.1:0/1206665788 >> v2:172.21.15.196:3300/0 conn(0x7f695006f1b0 0x7f69500ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f694e59c640 1 -- 172.21.15.1:0/1206665788 >> v1:172.21.15.196:6789/0 conn(0x7f695015ca90 legacy=0x7f695015cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.050 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.050+0000 7f694e59c640 1 -- 172.21.15.1:0/1206665788 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69500cc7b0 con 0x7f69500d16b0 2024-05-09T05:16:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.051+0000 7f694e59c640 1 --2- 172.21.15.1:0/1206665788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69500d16b0 0x7f69500d03d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f695006d0d0 tx=0x7f694002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a864572405bc296 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.052+0000 7f694d59a640 1 -- 172.21.15.1:0/1206665788 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940037070 con 0x7f69500d16b0 2024-05-09T05:16:42.051 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.052+0000 7f694d59a640 1 -- 172.21.15.1:0/1206665788 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f694002ae50 con 0x7f69500d16b0 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.052+0000 7f694d59a640 1 -- 172.21.15.1:0/1206665788 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6940032860 con 0x7f69500d16b0 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.052+0000 7f6954e8b640 1 -- 172.21.15.1:0/1206665788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69500d16b0 msgr2=0x7f69500d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.052+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1206665788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69500d16b0 0x7f69500d03d0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f695006d0d0 tx=0x7f694002a300 comp rx=0 tx=0).stop 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.052+0000 7f6954e8b640 1 -- 172.21.15.1:0/1206665788 shutdown_connections 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.053+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1206665788 >> v2:172.21.15.196:3300/0 conn(0x7f695006f1b0 0x7f69500ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.053+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1206665788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69500d16b0 0x7f69500d03d0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.053+0000 7f6954e8b640 1 -- 172.21.15.1:0/1206665788 >> 172.21.15.1:0/1206665788 conn(0x7f69500b3660 msgr2=0x7f69500b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.053+0000 7f6954e8b640 1 -- 172.21.15.1:0/1206665788 shutdown_connections 2024-05-09T05:16:42.052 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.053+0000 7f6954e8b640 1 -- 172.21.15.1:0/1206665788 wait complete. 2024-05-09T05:16:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.054+0000 7f6954e8b640 1 Processor -- start 2024-05-09T05:16:42.053 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.055+0000 7f6954e8b640 1 -- start start 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f6954e8b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f695006f1b0 0x7f69501f25a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f6954e8b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69500d16b0 0x7f69501f6430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f6954e8b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695015ca90 0x7f69500b6e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f6954e8b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f69500d0910 con 0x7f695015ca90 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f6954e8b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f69500ccd70 con 0x7f69500d16b0 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f6954e8b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f69500cec80 con 0x7f695006f1b0 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695015ca90 0x7f69500b6e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694ed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695015ca90 0x7f69500b6e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44092/0 (socket says 172.21.15.1:44092) 2024-05-09T05:16:42.056 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694ed9d640 1 -- 172.21.15.1:0/1338242222 learned_addr learned my addr 172.21.15.1:0/1338242222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694dd9b640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69500d16b0 0x7f69501f6430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694e59c640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f695006f1b0 0x7f69501f25a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694ed9d640 1 -- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f695006f1b0 msgr2=0x7f69501f25a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694ed9d640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f695006f1b0 0x7f69501f25a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694ed9d640 1 -- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69500d16b0 msgr2=0x7f69501f6430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694ed9d640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69500d16b0 0x7f69501f6430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.056+0000 7f694ed9d640 1 -- 172.21.15.1:0/1338242222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69501f6970 con 0x7f695015ca90 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.057+0000 7f694ed9d640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695015ca90 0x7f69500b6e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f694800c700 tx=0x7f694800cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.057+0000 7f693effd640 1 -- 172.21.15.1:0/1338242222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6948003b70 con 0x7f695015ca90 2024-05-09T05:16:42.057 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.057+0000 7f693effd640 1 -- 172.21.15.1:0/1338242222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6948003d10 con 0x7f695015ca90 2024-05-09T05:16:42.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.057+0000 7f693effd640 1 -- 172.21.15.1:0/1338242222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f694800aa50 con 0x7f695015ca90 2024-05-09T05:16:42.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.057+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69501f2ec0 con 0x7f695015ca90 2024-05-09T05:16:42.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.057+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69501f5c30 con 0x7f695015ca90 2024-05-09T05:16:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.058+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6950050870 con 0x7f695015ca90 2024-05-09T05:16:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.059+0000 7f693effd640 1 -- 172.21.15.1:0/1338242222 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f694800abf0 con 0x7f695015ca90 2024-05-09T05:16:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.059+0000 7f693effd640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6930076350 0x7f6930078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.060+0000 7f693effd640 1 -- 172.21.15.1:0/1338242222 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6948097350 con 0x7f695015ca90 2024-05-09T05:16:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.060+0000 7f694e59c640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6930076350 0x7f6930078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.060+0000 7f694e59c640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6930076350 0x7f6930078860 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6940034000 tx=0x7f694002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.064+0000 7f693effd640 1 -- 172.21.15.1:0/1338242222 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6948060630 con 0x7f695015ca90 2024-05-09T05:16:42.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.395+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6950152770 con 0x7f695015ca90 2024-05-09T05:16:42.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.395+0000 7f693effd640 1 -- 172.21.15.1:0/1338242222 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f6948064930 con 0x7f695015ca90 2024-05-09T05:16:42.394 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:16:42.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6930076350 msgr2=0x7f6930078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6930076350 0x7f6930078860 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f6940034000 tx=0x7f694002e000 comp rx=0 tx=0).stop 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695015ca90 msgr2=0x7f69500b6e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695015ca90 0x7f69500b6e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f694800c700 tx=0x7f694800cbc0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 shutdown_connections 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6930076350 0x7f6930078860 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69500d16b0 0x7f69501f6430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f695015ca90 0x7f69500b6e60 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 --2- 172.21.15.1:0/1338242222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f695006f1b0 0x7f69501f25a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 >> 172.21.15.1:0/1338242222 conn(0x7f69500b3660 msgr2=0x7f6950075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 shutdown_connections 2024-05-09T05:16:42.397 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.398+0000 7f6954e8b640 1 -- 172.21.15.1:0/1338242222 wait complete. 2024-05-09T05:16:42.408 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:16:42.408 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:16:42.409 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:16:42.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.533+0000 7f8904546640 1 Processor -- start 2024-05-09T05:16:42.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.533+0000 7f8904546640 1 -- start start 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8904546640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc15d130 0x7f88fc1569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8904546640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88fc06c6f0 0x7f88fc0716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8904546640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f88fc0c3ce0 con 0x7f88fc071be0 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8904546640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f88fc15d540 con 0x7f88fc06c6f0 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8904546640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f88fc0c3fc0 con 0x7f88fc15d130 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8901ae1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88fc06c6f0 0x7f88fc0716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8901ae1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f88fc06c6f0 0x7f88fc0716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44106/0 (socket says 172.21.15.1:44106) 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8901ae1640 1 -- 172.21.15.1:0/1157383100 learned_addr learned my addr 172.21.15.1:0/1157383100 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f89022e2640 1 --2- 172.21.15.1:0/1157383100 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc15d130 0x7f88fc1569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8901ae1640 1 -- 172.21.15.1:0/1157383100 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc15d130 msgr2=0x7f88fc1569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:42.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8901ae1640 1 --2- 172.21.15.1:0/1157383100 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc15d130 0x7f88fc1569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8901ae1640 1 -- 172.21.15.1:0/1157383100 >> v1:172.21.15.1:6789/0 conn(0x7f88fc071be0 legacy=0x7f88fc0c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f8901ae1640 1 -- 172.21.15.1:0/1157383100 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc06e4a0 con 0x7f88fc06c6f0 2024-05-09T05:16:42.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.535+0000 7f89022e2640 1 --2- 172.21.15.1:0/1157383100 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc15d130 0x7f88fc1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.536+0000 7f8901ae1640 1 --2- 172.21.15.1:0/1157383100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc06c6f0 0x7f88fc0716a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009870 tx=0x7f88ec02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df250c3875fb2f8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.537+0000 7f89012e0640 1 -- 172.21.15.1:0/1157383100 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec037070 con 0x7f88fc06c6f0 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.537+0000 7f89012e0640 1 -- 172.21.15.1:0/1157383100 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88ec032440 con 0x7f88fc06c6f0 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.537+0000 7f89012e0640 1 -- 172.21.15.1:0/1157383100 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec032810 con 0x7f88fc06c6f0 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.537+0000 7f8904546640 1 -- 172.21.15.1:0/1157383100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc06c6f0 msgr2=0x7f88fc0716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.537+0000 7f8904546640 1 --2- 172.21.15.1:0/1157383100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc06c6f0 0x7f88fc0716a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f88ec009870 tx=0x7f88ec02a270 comp rx=0 tx=0).stop 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.537+0000 7f8904546640 1 -- 172.21.15.1:0/1157383100 shutdown_connections 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.538+0000 7f8904546640 1 --2- 172.21.15.1:0/1157383100 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc06c6f0 0x7f88fc0716a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.538+0000 7f8904546640 1 --2- 172.21.15.1:0/1157383100 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc15d130 0x7f88fc1569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.538+0000 7f8904546640 1 -- 172.21.15.1:0/1157383100 >> 172.21.15.1:0/1157383100 conn(0x7f88fc0747b0 msgr2=0x7f88fc0ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:42.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.538+0000 7f8904546640 1 -- 172.21.15.1:0/1157383100 shutdown_connections 2024-05-09T05:16:42.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.538+0000 7f8904546640 1 -- 172.21.15.1:0/1157383100 wait complete. 2024-05-09T05:16:42.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.540+0000 7f8904546640 1 Processor -- start 2024-05-09T05:16:42.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.540+0000 7f8904546640 1 -- start start 2024-05-09T05:16:42.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.541+0000 7f8904546640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc06c6f0 0x7f88fc0cdc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.541+0000 7f8904546640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88fc071be0 0x7f88fc159010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.541+0000 7f8904546640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc15d130 0x7f88fc159550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.541+0000 7f8904546640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc06cb00 con 0x7f88fc15d130 2024-05-09T05:16:42.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.541+0000 7f8904546640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc06e1a0 con 0x7f88fc071be0 2024-05-09T05:16:42.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.541+0000 7f8904546640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f88fc06cc80 con 0x7f88fc06c6f0 2024-05-09T05:16:42.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc15d130 0x7f88fc159550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc15d130 0x7f88fc159550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44120/0 (socket says 172.21.15.1:44120) 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 -- 172.21.15.1:0/1336879746 learned_addr learned my addr 172.21.15.1:0/1336879746 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8901ae1640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88fc071be0 0x7f88fc159010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f89022e2640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc06c6f0 0x7f88fc0cdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 -- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc06c6f0 msgr2=0x7f88fc0cdc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f88fc06c6f0 0x7f88fc0cdc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 -- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88fc071be0 msgr2=0x7f88fc159010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f88fc071be0 0x7f88fc159010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 -- 172.21.15.1:0/1336879746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc1f2c10 con 0x7f88fc15d130 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8902ae3640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc15d130 0x7f88fc159550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f88f400c730 tx=0x7f88f400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f4003b70 con 0x7f88fc15d130 2024-05-09T05:16:42.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f88f4003d10 con 0x7f88fc15d130 2024-05-09T05:16:42.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f400aa70 con 0x7f88fc15d130 2024-05-09T05:16:42.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88fc1f2ef0 con 0x7f88fc15d130 2024-05-09T05:16:42.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.542+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88fc1f34b0 con 0x7f88fc15d130 2024-05-09T05:16:42.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.543+0000 7f8904546640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0b49b0 0x7f88fc0ce190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.543+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc050870 con 0x7f88fc0b49b0 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.544+0000 7f89022e2640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0b49b0 0x7f88fc0ce190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.544+0000 7f89022e2640 1 -- 172.21.15.1:0/1336879746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f88fc15a120 con 0x7f88fc0b49b0 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.544+0000 7f89022e2640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0b49b0 0x7f88fc0ce190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f88f800b730 tx=0x7f88f800bbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.544+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f8018070 con 0x7f88fc0b49b0 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.545+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f88f400ac10 con 0x7f88fc15d130 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.545+0000 7f88f2ffd640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f88e0076200 0x7f88e0078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.545+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f88f40973f0 con 0x7f88fc15d130 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.545+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f88f8010820 con 0x7f88fc0b49b0 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.545+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0b49b0 msgr2=0x7f88fc0ce190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.545+0000 7f88f2ffd640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0b49b0 0x7f88fc0ce190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f88f800b730 tx=0x7f88f800bbf0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.545+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0b49b0 msgr2=0x7f88fc0ce190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:42.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.545+0000 7f88f2ffd640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0b49b0 0x7f88fc0ce190 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f88f800b730 tx=0x7f88f800bbf0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.546+0000 7f8901ae1640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f88e0076200 0x7f88e0078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.546+0000 7f8901ae1640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f88e0076200 0x7f88e0078710 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f88ec034000 tx=0x7f88ec02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.553+0000 7f8904546640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0c6990 0x7f88fc0c6da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.553+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc0cb100 con 0x7f88fc0c6990 2024-05-09T05:16:42.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.553+0000 7f89022e2640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0c6990 0x7f88fc0c6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.554+0000 7f89022e2640 1 -- 172.21.15.1:0/1336879746 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f88fc159a90 con 0x7f88fc0c6990 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.554+0000 7f89022e2640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0c6990 0x7f88fc0c6da0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f88f800b730 tx=0x7f88f8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.554+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88f801de60 con 0x7f88fc0c6990 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.554+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7f88f8014840 con 0x7f88fc0c6990 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.554+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0c6990 msgr2=0x7f88fc0c6da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.554+0000 7f88f2ffd640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0c6990 0x7f88fc0c6da0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f88f800b730 tx=0x7f88f8004730 comp rx=0 tx=0).stop 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.554+0000 7f88f2ffd640 1 -- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0c6990 msgr2=0x7f88fc0c6da0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.554+0000 7f88f2ffd640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc0c6990 0x7f88fc0c6da0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.553 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 82, 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 204749, 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:42.554 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:16:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:16:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.555 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:16:42.556 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.557 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:16:42.558 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.559 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.560 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.561 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.562 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.563 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:16:42.564 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:42.565 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.566 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.567 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.568 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:16:42.568 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:16:42.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f88e0076200 msgr2=0x7f88e0078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8904546640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f88e0076200 0x7f88e0078710 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f88ec034000 tx=0x7f88ec02e000 comp rx=0 tx=0).stop 2024-05-09T05:16:42.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc15d130 msgr2=0x7f88fc159550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8904546640 1 --2- 172.21.15.1:0/1336879746 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f88fc15d130 0x7f88fc159550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f88f400c730 tx=0x7f88f400cbf0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8902ae3640 1 -- 172.21.15.1:0/1336879746 reap_dead start 2024-05-09T05:16:42.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8902ae3640 1 -- 172.21.15.1:0/1336879746 reap_dead start 2024-05-09T05:16:42.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 shutdown_connections 2024-05-09T05:16:42.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 >> 172.21.15.1:0/1336879746 conn(0x7f88fc0747b0 msgr2=0x7f88fc154220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:42.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 shutdown_connections 2024-05-09T05:16:42.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.555+0000 7f8904546640 1 -- 172.21.15.1:0/1336879746 wait complete. 2024-05-09T05:16:42.570 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:16:42.570 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:16:42.570 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:16:42.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.710+0000 7fe1f929f640 1 Processor -- start 2024-05-09T05:16:42.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.710+0000 7fe1f929f640 1 -- start start 2024-05-09T05:16:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f929f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1f40cab00 0x7fe1f40cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f929f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe1f40ca4a0 0x7fe1f40cb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f929f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1f415a800 con 0x7fe1f40b2850 2024-05-09T05:16:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f929f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe1f4070c60 con 0x7fe1f40cab00 2024-05-09T05:16:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f929f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe1f415aae0 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f27fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1f40cab00 0x7fe1f40cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f27fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe1f40cab00 0x7fe1f40cb220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44140/0 (socket says 172.21.15.1:44140) 2024-05-09T05:16:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f27fc640 1 -- 172.21.15.1:0/3512663710 learned_addr learned my addr 172.21.15.1:0/3512663710 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f37fe640 1 --2- 172.21.15.1:0/3512663710 >> v2:172.21.15.196:3300/0 conn(0x7fe1f40ca4a0 0x7fe1f40cb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f27fc640 1 -- 172.21.15.1:0/3512663710 >> v2:172.21.15.196:3300/0 conn(0x7fe1f40ca4a0 msgr2=0x7fe1f40cb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f27fc640 1 --2- 172.21.15.1:0/3512663710 >> v2:172.21.15.196:3300/0 conn(0x7fe1f40ca4a0 0x7fe1f40cb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f27fc640 1 -- 172.21.15.1:0/3512663710 >> v1:172.21.15.196:6789/0 conn(0x7fe1f40b2850 legacy=0x7fe1f4154510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.712+0000 7fe1f27fc640 1 -- 172.21.15.1:0/3512663710 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f415adc0 con 0x7fe1f40cab00 2024-05-09T05:16:42.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.713+0000 7fe1f27fc640 1 --2- 172.21.15.1:0/3512663710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40cab00 0x7fe1f40cb220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40b31b0 tx=0x7fe1e802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=994acd009688ec6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.713+0000 7fe1f1ffb640 1 -- 172.21.15.1:0/3512663710 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8037070 con 0x7fe1f40cab00 2024-05-09T05:16:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.713+0000 7fe1f1ffb640 1 -- 172.21.15.1:0/3512663710 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1e8032440 con 0x7fe1f40cab00 2024-05-09T05:16:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.713+0000 7fe1f1ffb640 1 -- 172.21.15.1:0/3512663710 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8032810 con 0x7fe1f40cab00 2024-05-09T05:16:42.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.714+0000 7fe1f929f640 1 -- 172.21.15.1:0/3512663710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40cab00 msgr2=0x7fe1f40cb220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.714+0000 7fe1f929f640 1 --2- 172.21.15.1:0/3512663710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40cab00 0x7fe1f40cb220 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fe1f40b31b0 tx=0x7fe1e802a1e0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.714+0000 7fe1f929f640 1 -- 172.21.15.1:0/3512663710 shutdown_connections 2024-05-09T05:16:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.714+0000 7fe1f929f640 1 --2- 172.21.15.1:0/3512663710 >> v2:172.21.15.196:3300/0 conn(0x7fe1f40ca4a0 0x7fe1f40cb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.714+0000 7fe1f929f640 1 --2- 172.21.15.1:0/3512663710 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40cab00 0x7fe1f40cb220 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.714+0000 7fe1f929f640 1 -- 172.21.15.1:0/3512663710 >> 172.21.15.1:0/3512663710 conn(0x7fe1f4072c30 msgr2=0x7fe1f4073040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.714+0000 7fe1f929f640 1 -- 172.21.15.1:0/3512663710 shutdown_connections 2024-05-09T05:16:42.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.714+0000 7fe1f929f640 1 -- 172.21.15.1:0/3512663710 wait complete. 2024-05-09T05:16:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.716+0000 7fe1f929f640 1 Processor -- start 2024-05-09T05:16:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.716+0000 7fe1f929f640 1 -- start start 2024-05-09T05:16:42.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f929f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40b2850 0x7fe1f4157830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f929f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40ca4a0 0x7fe1f41f4090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f929f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1f40cab00 0x7fe1f414f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f929f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f40710f0 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f929f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f415b380 con 0x7fe1f40b2850 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f929f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe1f4071270 con 0x7fe1f40cab00 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40ca4a0 0x7fe1f41f4090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40ca4a0 0x7fe1f41f4090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44154/0 (socket says 172.21.15.1:44154) 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 -- 172.21.15.1:0/671106223 learned_addr learned my addr 172.21.15.1:0/671106223 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f37fe640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1f40cab00 0x7fe1f414f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f2ffd640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40b2850 0x7fe1f4157830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 -- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1f40cab00 msgr2=0x7fe1f414f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1f40cab00 0x7fe1f414f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 -- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40b2850 msgr2=0x7fe1f4157830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40b2850 0x7fe1f4157830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 -- 172.21.15.1:0/671106223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1f4157d70 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.718+0000 7fe1f27fc640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40ca4a0 0x7fe1f41f4090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8000c00 tx=0x7fe1e80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.719+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8037040 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.719+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe1e8042070 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.719+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1f41557f0 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.719+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8030880 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.719+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1f41f4cc0 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.720+0000 7fe1f929f640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40c6ea0 0x7fe1f41570c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.720+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f41555a0 con 0x7fe1f40c6ea0 2024-05-09T05:16:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.720+0000 7fe1f2ffd640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40c6ea0 0x7fe1f41570c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.720+0000 7fe1f2ffd640 1 -- 172.21.15.1:0/671106223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe1f40cb970 con 0x7fe1f40c6ea0 2024-05-09T05:16:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.721+0000 7fe1f2ffd640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40c6ea0 0x7fe1f41570c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4074280 tx=0x7fe1e400f9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.721+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe1e8030a20 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.721+0000 7fe1db7fe640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1e00762d0 0x7fe1e00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe1e80b7ed0 con 0x7fe1f40ca4a0 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1f37fe640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1e00762d0 0x7fe1e00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e40052e0 con 0x7fe1f40c6ea0 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe1e4018070 con 0x7fe1f40c6ea0 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40c6ea0 msgr2=0x7fe1f41570c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1db7fe640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40c6ea0 0x7fe1f41570c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4074280 tx=0x7fe1e400f9b0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40c6ea0 msgr2=0x7fe1f41570c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1db7fe640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f40c6ea0 0x7fe1f41570c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.722+0000 7fe1f37fe640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1e00762d0 0x7fe1e00787e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe1f41f3e80 tx=0x7fe1ec00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.729+0000 7fe1f929f640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f414f620 0x7fe1f4154ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.729+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe1f414fa30 con 0x7fe1f414f620 2024-05-09T05:16:42.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.729+0000 7fe1f2ffd640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f414f620 0x7fe1f4154ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.730+0000 7fe1f2ffd640 1 -- 172.21.15.1:0/671106223 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe1f40cb000 con 0x7fe1f414f620 2024-05-09T05:16:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.730+0000 7fe1f2ffd640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f414f620 0x7fe1f4154ec0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4074280 tx=0x7fe1e4001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.730+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1e4003b40 con 0x7fe1f414f620 2024-05-09T05:16:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.730+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4891 (secure 0 0 0) 0x7fe1e4003b40 con 0x7fe1f414f620 2024-05-09T05:16:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.730+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f414f620 msgr2=0x7fe1f4154ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.730+0000 7fe1db7fe640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f414f620 0x7fe1f4154ec0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4074280 tx=0x7fe1e4001080 comp rx=0 tx=0).stop 2024-05-09T05:16:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.730+0000 7fe1db7fe640 1 -- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f414f620 msgr2=0x7fe1f4154ec0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:42.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.730+0000 7fe1db7fe640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1f414f620 0x7fe1f4154ec0 secure :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fe1f4074280 tx=0x7fe1e4001080 comp rx=0 tx=0).stop 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 82, 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 265009, 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:16:42.730 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.731 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.732 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:16:42.733 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:42.734 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.735 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.736 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.737 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.738 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.740 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.741 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:16:42.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.743 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:16:42.744 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:16:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1e00762d0 msgr2=0x7fe1e00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1e00762d0 0x7fe1e00787e0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fe1f41f3e80 tx=0x7fe1ec00b040 comp rx=0 tx=0).stop 2024-05-09T05:16:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40ca4a0 msgr2=0x7fe1f41f4090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f37fe640 1 -- 172.21.15.1:0/671106223 reap_dead start 2024-05-09T05:16:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40ca4a0 0x7fe1f41f4090 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe1e8000c00 tx=0x7fe1e80046c0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 shutdown_connections 2024-05-09T05:16:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 --2- 172.21.15.1:0/671106223 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1f40ca4a0 0x7fe1f41f4090 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 >> 172.21.15.1:0/671106223 conn(0x7fe1f4072c30 msgr2=0x7fe1f4159e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 shutdown_connections 2024-05-09T05:16:42.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.731+0000 7fe1f929f640 1 -- 172.21.15.1:0/671106223 wait complete. 2024-05-09T05:16:42.747 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:16:42.747 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:16:42.747 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:16:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.873+0000 7f5c72708640 1 Processor -- start 2024-05-09T05:16:42.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.873+0000 7f5c72708640 1 -- start start 2024-05-09T05:16:42.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.874+0000 7f5c72708640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5c6c0c99a0 0x7f5c6c072020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.874+0000 7f5c72708640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c6c070b20 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.874+0000 7f5c72708640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5c6c0c9500 con 0x7f5c6c0c3b40 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.874+0000 7f5c72708640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f5c6c06ff80 con 0x7f5c6c0c99a0 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.874+0000 7f5c70ca5640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f5c6c0c3f50 0x7f5c6c0caa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:36982/0 (socket says 172.21.15.1:36982) 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.874+0000 7f5c70ca5640 1 -- 172.21.15.1:0/101831070 learned_addr learned my addr 172.21.15.1:0/101831070 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.875+0000 7f5c6bfff640 1 --2- 172.21.15.1:0/101831070 >> v2:172.21.15.196:3300/0 conn(0x7f5c6c0c99a0 0x7f5c6c072020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.875+0000 7f5c6affd640 1 -- 172.21.15.1:0/101831070 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 397797012 0 0) 0x7f5c6c070b20 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.875+0000 7f5c6affd640 1 -- 172.21.15.1:0/101831070 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5c54003520 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.875+0000 7f5c6bfff640 1 -- 172.21.15.1:0/101831070 >> v1:172.21.15.196:6789/0 conn(0x7f5c6c0c3b40 legacy=0x7f5c6c070410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.875+0000 7f5c6bfff640 1 -- 172.21.15.1:0/101831070 >> v1:172.21.15.1:6789/0 conn(0x7f5c6c0c3f50 legacy=0x7f5c6c0caa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.875+0000 7f5c6affd640 1 -- 172.21.15.1:0/101831070 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 865229808 0 0) 0x7f5c54003520 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.875+0000 7f5c6bfff640 1 -- 172.21.15.1:0/101831070 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c06fc30 con 0x7f5c6c0c99a0 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.876+0000 7f5c6bfff640 1 --2- 172.21.15.1:0/101831070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c6c0c99a0 0x7f5c6c072020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0097a0 tx=0x7f5c5c02a130 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aaa9df20424a9b7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.876+0000 7f5c6affd640 1 -- 172.21.15.1:0/101831070 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c037070 con 0x7f5c6c0c99a0 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.876+0000 7f5c6affd640 1 -- 172.21.15.1:0/101831070 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c5c02abc0 con 0x7f5c6c0c99a0 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.876+0000 7f5c6affd640 1 -- 172.21.15.1:0/101831070 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c64001a70 con 0x7f5c6c0c99a0 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.877+0000 7f5c72708640 1 -- 172.21.15.1:0/101831070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c6c0c99a0 msgr2=0x7f5c6c072020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.877+0000 7f5c72708640 1 --2- 172.21.15.1:0/101831070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c6c0c99a0 0x7f5c6c072020 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c0097a0 tx=0x7f5c5c02a130 comp rx=0 tx=0).stop 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.877+0000 7f5c72708640 1 -- 172.21.15.1:0/101831070 shutdown_connections 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.877+0000 7f5c72708640 1 --2- 172.21.15.1:0/101831070 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c6c0c99a0 0x7f5c6c072020 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.876 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.877+0000 7f5c72708640 1 -- 172.21.15.1:0/101831070 >> 172.21.15.1:0/101831070 conn(0x7f5c6c0cb590 msgr2=0x7f5c6c0cb9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:42.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.877+0000 7f5c72708640 1 -- 172.21.15.1:0/101831070 shutdown_connections 2024-05-09T05:16:42.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.877+0000 7f5c72708640 1 -- 172.21.15.1:0/101831070 wait complete. 2024-05-09T05:16:42.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.879+0000 7f5c72708640 1 Processor -- start 2024-05-09T05:16:42.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.879+0000 7f5c72708640 1 -- start start 2024-05-09T05:16:42.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c72708640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c0c3b40 0x7f5c6c151140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c72708640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c0c3f50 0x7f5c6c1e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c72708640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c6c0c99a0 0x7f5c6c155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c72708640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c0c4c20 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.880 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c72708640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c0c4940 con 0x7f5c6c0c99a0 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c72708640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6c0c4f00 con 0x7f5c6c0c3b40 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c0c3f50 0x7f5c6c1e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c0c3f50 0x7f5c6c1e3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44170/0 (socket says 172.21.15.1:44170) 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 -- 172.21.15.1:0/3605170589 learned_addr learned my addr 172.21.15.1:0/3605170589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c70ca5640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c6c0c99a0 0x7f5c6c155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6bfff640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c0c3b40 0x7f5c6c151140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 -- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c0c3b40 msgr2=0x7f5c6c151140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c0c3b40 0x7f5c6c151140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 -- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c6c0c99a0 msgr2=0x7f5c6c155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5c6c0c99a0 0x7f5c6c155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 -- 172.21.15.1:0/3605170589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c1f3b20 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.881+0000 7f5c6b7fe640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c0c3f50 0x7f5c6c1e3c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000c3c0 tx=0x7f5c6000c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.882+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60017070 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.882+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5c600052f0 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.882+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c60003850 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.882+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6c1f3dc0 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.882+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6c1f4380 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.883+0000 7f5c72708640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c15a1f0 0x7f5c6c156180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.883+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6c154350 con 0x7f5c6c15a1f0 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.883+0000 7f5c6bfff640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c15a1f0 0x7f5c6c156180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.883+0000 7f5c6bfff640 1 -- 172.21.15.1:0/3605170589 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5c6c156e30 con 0x7f5c6c15a1f0 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.883+0000 7f5c6bfff640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c15a1f0 0x7f5c6c156180 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0c9f30 tx=0x7f5c5c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.884+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5c043070 con 0x7f5c6c15a1f0 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.884+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5c5c0044f0 con 0x7f5c6c15a1f0 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.884+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c15a1f0 msgr2=0x7f5c6c156180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.884+0000 7f5c68ff9640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c15a1f0 0x7f5c6c156180 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0c9f30 tx=0x7f5c5c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.884+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c15a1f0 msgr2=0x7f5c6c156180 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:42.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.884+0000 7f5c68ff9640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c15a1f0 0x7f5c6c156180 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.884+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5c6001a070 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.885+0000 7f5c68ff9640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5c58075d40 0x7f5c58078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.885+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5c60096a70 con 0x7f5c6c0c3f50 2024-05-09T05:16:42.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.885+0000 7f5c6bfff640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5c58075d40 0x7f5c58078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.886+0000 7f5c6bfff640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5c58075d40 0x7f5c58078250 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0780e0 tx=0x7f5c5c049040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.891+0000 7f5c72708640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c1568f0 0x7f5c6c1f0590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:42.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.891+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5c6c153140 con 0x7f5c6c1568f0 2024-05-09T05:16:42.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.891+0000 7f5c70ca5640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c1568f0 0x7f5c6c1f0590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:42.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.892+0000 7f5c70ca5640 1 -- 172.21.15.1:0/3605170589 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5c6c1566c0 con 0x7f5c6c1568f0 2024-05-09T05:16:42.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.892+0000 7f5c70ca5640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c1568f0 0x7f5c6c1f0590 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0b4e90 tx=0x7f5c6400d520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:42.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.892+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c6400dd90 con 0x7f5c6c1568f0 2024-05-09T05:16:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.892+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4550 (secure 0 0 0) 0x7f5c6400dd90 con 0x7f5c6c1568f0 2024-05-09T05:16:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.892+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c1568f0 msgr2=0x7f5c6c1f0590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.892+0000 7f5c68ff9640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c1568f0 0x7f5c6c1f0590 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0b4e90 tx=0x7f5c6400d520 comp rx=0 tx=0).stop 2024-05-09T05:16:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.892+0000 7f5c68ff9640 1 -- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c1568f0 msgr2=0x7f5c6c1f0590 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:42.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.892+0000 7f5c68ff9640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5c6c1568f0 0x7f5c6c1f0590 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 82, 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 24187, 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:16:42.892 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.893 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.894 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:16:42.895 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:16:42.896 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:42.897 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.898 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:16:42.899 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:16:42.900 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.901 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.902 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:42.903 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.904 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:16:42.905 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5c58075d40 msgr2=0x7f5c58078250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5c58075d40 0x7f5c58078250 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f5c6c0780e0 tx=0x7f5c5c049040 comp rx=0 tx=0).stop 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c0c3f50 msgr2=0x7f5c6c1e3c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c0c3f50 0x7f5c6c1e3c70 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f5c6000c3c0 tx=0x7f5c6000c880 comp rx=0 tx=0).stop 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c70ca5640 1 -- 172.21.15.1:0/3605170589 reap_dead start 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 shutdown_connections 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 --2- 172.21.15.1:0/3605170589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5c6c0c3f50 0x7f5c6c1e3c70 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 >> 172.21.15.1:0/3605170589 conn(0x7f5c6c0cb590 msgr2=0x7f5c6c0ce300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:42.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 shutdown_connections 2024-05-09T05:16:42.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:42.893+0000 7f5c72708640 1 -- 172.21.15.1:0/3605170589 wait complete. 2024-05-09T05:16:42.908 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:16:42.908 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:16:42.908 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:16:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.027+0000 7f8a2fd0d640 1 Processor -- start 2024-05-09T05:16:43.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.028+0000 7f8a2fd0d640 1 -- start start 2024-05-09T05:16:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.029+0000 7f8a2fd0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a2815d550 0x7f8a28156b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.029+0000 7f8a2fd0d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8a280746b0 0x7f8a280c8a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.030+0000 7f8a2fd0d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8a280ca3e0 con 0x7f8a280c5500 2024-05-09T05:16:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.030+0000 7f8a2fd0d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8a280c8fa0 con 0x7f8a280746b0 2024-05-09T05:16:43.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.030+0000 7f8a2fd0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8a280ca6c0 con 0x7f8a2815d550 2024-05-09T05:16:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.030+0000 7f8a2daa9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a2815d550 0x7f8a28156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.030+0000 7f8a2d2a8640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f8a280c5500 0x7f8a280c6f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:35966/0 (socket says 172.21.15.1:35966) 2024-05-09T05:16:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.030+0000 7f8a2d2a8640 1 -- 172.21.15.1:0/3716925621 learned_addr learned my addr 172.21.15.1:0/3716925621 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:16:43.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.030+0000 7f8a2e2aa640 1 --2- 172.21.15.1:0/3716925621 >> v2:172.21.15.196:3300/0 conn(0x7f8a280746b0 0x7f8a280c8a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.031+0000 7f8a2caa7640 1 -- 172.21.15.1:0/3716925621 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1560082923 0 0) 0x7f8a280ca3e0 con 0x7f8a280c5500 2024-05-09T05:16:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.031+0000 7f8a2caa7640 1 -- 172.21.15.1:0/3716925621 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8a14003520 con 0x7f8a280c5500 2024-05-09T05:16:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.031+0000 7f8a2daa9640 1 -- 172.21.15.1:0/3716925621 >> v2:172.21.15.196:3300/0 conn(0x7f8a280746b0 msgr2=0x7f8a280c8a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.031+0000 7f8a2daa9640 1 --2- 172.21.15.1:0/3716925621 >> v2:172.21.15.196:3300/0 conn(0x7f8a280746b0 0x7f8a280c8a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.031+0000 7f8a2daa9640 1 -- 172.21.15.1:0/3716925621 >> v1:172.21.15.196:6789/0 conn(0x7f8a280c5500 legacy=0x7f8a280c6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.031+0000 7f8a2daa9640 1 -- 172.21.15.1:0/3716925621 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2815ca70 con 0x7f8a2815d550 2024-05-09T05:16:43.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.032+0000 7f8a2daa9640 1 --2- 172.21.15.1:0/3716925621 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a2815d550 0x7f8a28156b70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8a240097a0 tx=0x7f8a2402a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4ec68f6a61037b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.033+0000 7f8a2caa7640 1 -- 172.21.15.1:0/3716925621 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24037070 con 0x7f8a2815d550 2024-05-09T05:16:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.033+0000 7f8a2caa7640 1 -- 172.21.15.1:0/3716925621 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a24032440 con 0x7f8a2815d550 2024-05-09T05:16:43.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.033+0000 7f8a2caa7640 1 -- 172.21.15.1:0/3716925621 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24032810 con 0x7f8a2815d550 2024-05-09T05:16:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.034+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/3716925621 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a2815d550 msgr2=0x7f8a28156b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.034+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/3716925621 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a2815d550 0x7f8a28156b70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f8a240097a0 tx=0x7f8a2402a210 comp rx=0 tx=0).stop 2024-05-09T05:16:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.034+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/3716925621 shutdown_connections 2024-05-09T05:16:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.034+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/3716925621 >> v2:172.21.15.196:3300/0 conn(0x7f8a280746b0 0x7f8a280c8a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.034+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/3716925621 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a2815d550 0x7f8a28156b70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.034+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/3716925621 >> 172.21.15.1:0/3716925621 conn(0x7f8a280ce0e0 msgr2=0x7f8a280ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.034+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/3716925621 shutdown_connections 2024-05-09T05:16:43.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.034+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/3716925621 wait complete. 2024-05-09T05:16:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.036+0000 7f8a2fd0d640 1 Processor -- start 2024-05-09T05:16:43.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.036+0000 7f8a2fd0d640 1 -- start start 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2fd0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a280746b0 0x7f8a28153690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2fd0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a280c5500 0x7f8a281511e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2fd0d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a2815d550 0x7f8a28151720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2fd0d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8a28074ac0 con 0x7f8a280746b0 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2fd0d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2815d030 con 0x7f8a2815d550 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2fd0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8a280740e0 con 0x7f8a280c5500 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2daa9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a280746b0 0x7f8a28153690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2daa9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a280746b0 0x7f8a28153690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44180/0 (socket says 172.21.15.1:44180) 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2daa9640 1 -- 172.21.15.1:0/2354928702 learned_addr learned my addr 172.21.15.1:0/2354928702 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.037+0000 7f8a2d2a8640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a280c5500 0x7f8a281511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2e2aa640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a2815d550 0x7f8a28151720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2daa9640 1 -- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a280c5500 msgr2=0x7f8a281511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:43.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2daa9640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a280c5500 0x7f8a281511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2daa9640 1 -- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a2815d550 msgr2=0x7f8a28151720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2daa9640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a2815d550 0x7f8a28151720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2daa9640 1 -- 172.21.15.1:0/2354928702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a28153bd0 con 0x7f8a280746b0 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2daa9640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a280746b0 0x7f8a28153690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8a240098c0 tx=0x7f8a240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a1e7fc640 1 -- 172.21.15.1:0/2354928702 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24041030 con 0x7f8a280746b0 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a1e7fc640 1 -- 172.21.15.1:0/2354928702 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a2403d4e0 con 0x7f8a280746b0 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a1e7fc640 1 -- 172.21.15.1:0/2354928702 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a24037040 con 0x7f8a280746b0 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a28152ed0 con 0x7f8a280746b0 2024-05-09T05:16:43.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.038+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a28153170 con 0x7f8a280746b0 2024-05-09T05:16:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.039+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a28050870 con 0x7f8a280746b0 2024-05-09T05:16:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.040+0000 7f8a1e7fc640 1 -- 172.21.15.1:0/2354928702 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8a24044070 con 0x7f8a280746b0 2024-05-09T05:16:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.041+0000 7f8a1e7fc640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a0c076300 0x7f8a0c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.041+0000 7f8a1e7fc640 1 -- 172.21.15.1:0/2354928702 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8a240b76b0 con 0x7f8a280746b0 2024-05-09T05:16:43.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.041+0000 7f8a2d2a8640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a0c076300 0x7f8a0c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:43.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.042+0000 7f8a2d2a8640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a0c076300 0x7f8a0c078810 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a280c8e40 tx=0x7f8a18006aa0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:43.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.046+0000 7f8a1e7fc640 1 -- 172.21.15.1:0/2354928702 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8a240808c0 con 0x7f8a280746b0 2024-05-09T05:16:43.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.368+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f8a280cbec0 con 0x7f8a280746b0 2024-05-09T05:16:43.367 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.369+0000 7f8a1e7fc640 1 -- 172.21.15.1:0/2354928702 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f8a24084bc0 con 0x7f8a280746b0 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a0c076300 msgr2=0x7f8a0c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a0c076300 0x7f8a0c078810 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a280c8e40 tx=0x7f8a18006aa0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a280746b0 msgr2=0x7f8a28153690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a280746b0 0x7f8a28153690 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8a240098c0 tx=0x7f8a240046c0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 shutdown_connections 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a0c076300 0x7f8a0c078810 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a2815d550 0x7f8a28151720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a280c5500 0x7f8a281511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 --2- 172.21.15.1:0/2354928702 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a280746b0 0x7f8a28153690 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 >> 172.21.15.1:0/2354928702 conn(0x7f8a280ce0e0 msgr2=0x7f8a280cc390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 shutdown_connections 2024-05-09T05:16:43.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:43.372+0000 7f8a2fd0d640 1 -- 172.21.15.1:0/2354928702 wait complete. 2024-05-09T05:16:43.383 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:16:53.385 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:16:53.385 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:16:53.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.516+0000 7fe0b8df3640 1 Processor -- start 2024-05-09T05:16:53.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.516+0000 7fe0b8df3640 1 -- start start 2024-05-09T05:16:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b8df3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b4154ab0 0x7fe0b415e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b8df3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe0b40c77a0 0x7fe0b40c7bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b8df3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe0b40cccf0 con 0x7fe0b406c910 2024-05-09T05:16:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b8df3640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe0b4068d10 con 0x7fe0b40c77a0 2024-05-09T05:16:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b8df3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b40ccfd0 con 0x7fe0b4154ab0 2024-05-09T05:16:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b2d9d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe0b40c77a0 0x7fe0b40c7bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b1d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b4154ab0 0x7fe0b415e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b259c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fe0b406c910 0x7fe0b40cbef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:50144/0 (socket says 172.21.15.1:50144) 2024-05-09T05:16:53.517 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.518+0000 7fe0b259c640 1 -- 172.21.15.1:0/4242833141 learned_addr learned my addr 172.21.15.1:0/4242833141 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:16:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.519+0000 7fe0b2d9d640 1 -- 172.21.15.1:0/4242833141 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b4154ab0 msgr2=0x7fe0b415e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.519+0000 7fe0b2d9d640 1 --2- 172.21.15.1:0/4242833141 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b4154ab0 0x7fe0b415e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.519+0000 7fe0b2d9d640 1 -- 172.21.15.1:0/4242833141 >> v1:172.21.15.196:6789/0 conn(0x7fe0b406c910 legacy=0x7fe0b40cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.519+0000 7fe0b2d9d640 1 -- 172.21.15.1:0/4242833141 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b40cd2b0 con 0x7fe0b40c77a0 2024-05-09T05:16:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.519+0000 7fe0b159a640 1 -- 172.21.15.1:0/4242833141 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 717325550 0 0) 0x7fe0b40cccf0 con 0x7fe0b406c910 2024-05-09T05:16:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.520+0000 7fe0b2d9d640 1 --2- 172.21.15.1:0/4242833141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b40c77a0 0x7fe0b40c7bb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009d90 tx=0x7fe0ac02a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=107a78eb6f49d66 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.520+0000 7fe0b159a640 1 -- 172.21.15.1:0/4242833141 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac007740 con 0x7fe0b40c77a0 2024-05-09T05:16:53.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.520+0000 7fe0b159a640 1 -- 172.21.15.1:0/4242833141 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0ac0078e0 con 0x7fe0b40c77a0 2024-05-09T05:16:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.520+0000 7fe0b159a640 1 -- 172.21.15.1:0/4242833141 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0ac007cb0 con 0x7fe0b40c77a0 2024-05-09T05:16:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.521+0000 7fe0b8df3640 1 -- 172.21.15.1:0/4242833141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b40c77a0 msgr2=0x7fe0b40c7bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.521+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/4242833141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b40c77a0 0x7fe0b40c7bb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe0ac009d90 tx=0x7fe0ac02a660 comp rx=0 tx=0).stop 2024-05-09T05:16:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.521+0000 7fe0b8df3640 1 -- 172.21.15.1:0/4242833141 shutdown_connections 2024-05-09T05:16:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.521+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/4242833141 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b40c77a0 0x7fe0b40c7bb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.521+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/4242833141 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b4154ab0 0x7fe0b415e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.521+0000 7fe0b8df3640 1 -- 172.21.15.1:0/4242833141 >> 172.21.15.1:0/4242833141 conn(0x7fe0b40ce110 msgr2=0x7fe0b40ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.521+0000 7fe0b8df3640 1 -- 172.21.15.1:0/4242833141 shutdown_connections 2024-05-09T05:16:53.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.521+0000 7fe0b8df3640 1 -- 172.21.15.1:0/4242833141 wait complete. 2024-05-09T05:16:53.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.523+0000 7fe0b8df3640 1 Processor -- start 2024-05-09T05:16:53.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.523+0000 7fe0b8df3640 1 -- start start 2024-05-09T05:16:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.524+0000 7fe0b8df3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b406c910 0x7fe0b414fe50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.524+0000 7fe0b8df3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b40c77a0 0x7fe0b41e5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.524+0000 7fe0b8df3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0b4154ab0 0x7fe0b414e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.524+0000 7fe0b8df3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b40cc990 con 0x7fe0b4154ab0 2024-05-09T05:16:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.524+0000 7fe0b8df3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe0b415dfb0 con 0x7fe0b406c910 2024-05-09T05:16:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.524+0000 7fe0b8df3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe0b40cd590 con 0x7fe0b40c77a0 2024-05-09T05:16:53.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.524+0000 7fe0b2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0b4154ab0 0x7fe0b414e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b2d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0b4154ab0 0x7fe0b414e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33356/0 (socket says 172.21.15.1:33356) 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b2d9d640 1 -- 172.21.15.1:0/1031280202 learned_addr learned my addr 172.21.15.1:0/1031280202 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b259c640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b406c910 0x7fe0b414fe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b1d9b640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b40c77a0 0x7fe0b41e5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b259c640 1 -- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b40c77a0 msgr2=0x7fe0b41e5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b259c640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b40c77a0 0x7fe0b41e5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b259c640 1 -- 172.21.15.1:0/1031280202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0b4154ab0 msgr2=0x7fe0b414e6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b259c640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0b4154ab0 0x7fe0b414e6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b259c640 1 -- 172.21.15.1:0/1031280202 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0b414de80 con 0x7fe0b406c910 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b2d9d640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0b4154ab0 0x7fe0b414e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b259c640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b406c910 0x7fe0b414fe50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400c3c0 tx=0x7fe0a400c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0a2ffd640 1 -- 172.21.15.1:0/1031280202 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a40074e0 con 0x7fe0b406c910 2024-05-09T05:16:53.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0a2ffd640 1 -- 172.21.15.1:0/1031280202 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0a40047e0 con 0x7fe0b406c910 2024-05-09T05:16:53.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.525+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0b4150390 con 0x7fe0b406c910 2024-05-09T05:16:53.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.526+0000 7fe0a2ffd640 1 -- 172.21.15.1:0/1031280202 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0a4004bd0 con 0x7fe0b406c910 2024-05-09T05:16:53.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.526+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0b41e3520 con 0x7fe0b406c910 2024-05-09T05:16:53.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.526+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0b4050870 con 0x7fe0b406c910 2024-05-09T05:16:53.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.528+0000 7fe0a2ffd640 1 -- 172.21.15.1:0/1031280202 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe0a4011070 con 0x7fe0b406c910 2024-05-09T05:16:53.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.528+0000 7fe0a2ffd640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0940762d0 0x7fe0940787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.529+0000 7fe0a2ffd640 1 -- 172.21.15.1:0/1031280202 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe0a40974c0 con 0x7fe0b406c910 2024-05-09T05:16:53.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.529+0000 7fe0b1d9b640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0940762d0 0x7fe0940787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.529+0000 7fe0b1d9b640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0940762d0 0x7fe0940787e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe0b41e5eb0 tx=0x7fe0a8009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:53.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.532+0000 7fe0a2ffd640 1 -- 172.21.15.1:0/1031280202 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe0a4060830 con 0x7fe0b406c910 2024-05-09T05:16:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.848+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fe0b406c6f0 con 0x7fe0b406c910 2024-05-09T05:16:53.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.849+0000 7fe0a2ffd640 1 -- 172.21.15.1:0/1031280202 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fe0a4064b30 con 0x7fe0b406c910 2024-05-09T05:16:53.848 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:16:53.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0940762d0 msgr2=0x7fe0940787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0940762d0 0x7fe0940787e0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fe0b41e5eb0 tx=0x7fe0a8009190 comp rx=0 tx=0).stop 2024-05-09T05:16:53.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b406c910 msgr2=0x7fe0b414fe50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b406c910 0x7fe0b414fe50 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe0a400c3c0 tx=0x7fe0a400c880 comp rx=0 tx=0).stop 2024-05-09T05:16:53.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 shutdown_connections 2024-05-09T05:16:53.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0940762d0 0x7fe0940787e0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe0b4154ab0 0x7fe0b414e6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe0b40c77a0 0x7fe0b41e5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 --2- 172.21.15.1:0/1031280202 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe0b406c910 0x7fe0b414fe50 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 >> 172.21.15.1:0/1031280202 conn(0x7fe0b40ce110 msgr2=0x7fe0b415ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 shutdown_connections 2024-05-09T05:16:53.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.851+0000 7fe0b8df3640 1 -- 172.21.15.1:0/1031280202 wait complete. 2024-05-09T05:16:53.862 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:16:53.862 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:16:53.862 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:16:53.862 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:16:53.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.988+0000 7f108ac14640 1 Processor -- start 2024-05-09T05:16:53.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.988+0000 7f108ac14640 1 -- start start 2024-05-09T05:16:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f108ac14640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f108415e300 0x7f108415e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f108ac14640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f10840c7810 0x7f10840c7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f108ac14640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f10840cd730 con 0x7f10841547d0 2024-05-09T05:16:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f108ac14640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f108415de70 con 0x7f10840c7810 2024-05-09T05:16:53.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f108ac14640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f10840cc970 con 0x7f108415e300 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f10891b1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f10840c7810 0x7f10840c7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f1083fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f108415e300 0x7f108415e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f10889b0640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f10841547d0 0x7f10840cbf60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:50148/0 (socket says 172.21.15.1:50148) 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f10889b0640 1 -- 172.21.15.1:0/1449301859 learned_addr learned my addr 172.21.15.1:0/1449301859 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f10837fe640 1 -- 172.21.15.1:0/1449301859 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2272350238 0 0) 0x7f1078001110 con 0x7f10841547d0 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.990+0000 7f10837fe640 1 -- 172.21.15.1:0/1449301859 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1074003520 con 0x7f10841547d0 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.991+0000 7f10891b1640 1 -- 172.21.15.1:0/1449301859 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f108415e300 msgr2=0x7f108415e710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.991+0000 7f10891b1640 1 --2- 172.21.15.1:0/1449301859 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f108415e300 0x7f108415e710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.991+0000 7f10891b1640 1 -- 172.21.15.1:0/1449301859 >> v1:172.21.15.196:6789/0 conn(0x7f10841547d0 legacy=0x7f10840cbf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.991+0000 7f10891b1640 1 -- 172.21.15.1:0/1449301859 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f108406c990 con 0x7f10840c7810 2024-05-09T05:16:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.991+0000 7f1083fff640 1 --2- 172.21.15.1:0/1449301859 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f108415e300 0x7f108415e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:16:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f10891b1640 1 --2- 172.21.15.1:0/1449301859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10840c7810 0x7f10840c7c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f107c009d90 tx=0x7f107c02a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e9959c63146a30c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f10837fe640 1 -- 172.21.15.1:0/1449301859 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c031600 con 0x7f10840c7810 2024-05-09T05:16:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f10837fe640 1 -- 172.21.15.1:0/1449301859 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107c0317a0 con 0x7f10840c7810 2024-05-09T05:16:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f10837fe640 1 -- 172.21.15.1:0/1449301859 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c031b70 con 0x7f10840c7810 2024-05-09T05:16:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f108ac14640 1 -- 172.21.15.1:0/1449301859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10840c7810 msgr2=0x7f10840c7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f108ac14640 1 --2- 172.21.15.1:0/1449301859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10840c7810 0x7f10840c7c20 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f107c009d90 tx=0x7f107c02a5d0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f108ac14640 1 -- 172.21.15.1:0/1449301859 shutdown_connections 2024-05-09T05:16:53.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f108ac14640 1 --2- 172.21.15.1:0/1449301859 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10840c7810 0x7f10840c7c20 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f108ac14640 1 --2- 172.21.15.1:0/1449301859 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f108415e300 0x7f108415e710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f108ac14640 1 -- 172.21.15.1:0/1449301859 >> 172.21.15.1:0/1449301859 conn(0x7f10840ce180 msgr2=0x7f10840ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.992+0000 7f108ac14640 1 -- 172.21.15.1:0/1449301859 shutdown_connections 2024-05-09T05:16:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.993+0000 7f108ac14640 1 -- 172.21.15.1:0/1449301859 wait complete. 2024-05-09T05:16:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.994+0000 7f108ac14640 1 Processor -- start 2024-05-09T05:16:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.995+0000 7f108ac14640 1 -- start start 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f108ac14640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10840c7810 0x7f10841552d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f108ac14640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841547d0 0x7f1084156250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f108ac14640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f108415e300 0x7f10840cb400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f108ac14640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f10840cce90 con 0x7f108415e300 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f108ac14640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f10840cc670 con 0x7f10841547d0 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f108ac14640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f10840cd170 con 0x7f10840c7810 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f10891b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f108415e300 0x7f10840cb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f10891b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f108415e300 0x7f10840cb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33372/0 (socket says 172.21.15.1:33372) 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f10891b1640 1 -- 172.21.15.1:0/1703976284 learned_addr learned my addr 172.21.15.1:0/1703976284 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:53.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.996+0000 7f1083fff640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841547d0 0x7f1084156250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f10889b0640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10840c7810 0x7f10841552d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f1083fff640 1 -- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10840c7810 msgr2=0x7f10841552d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f1083fff640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f10840c7810 0x7f10841552d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f1083fff640 1 -- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f108415e300 msgr2=0x7f10840cb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f1083fff640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f108415e300 0x7f10840cb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f1083fff640 1 -- 172.21.15.1:0/1703976284 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10840ca1d0 con 0x7f10841547d0 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f10891b1640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f108415e300 0x7f10840cb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f1083fff640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841547d0 0x7f1084156250 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f107000e870 tx=0x7f107000ed30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1070017070 con 0x7f10841547d0 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f107000c940 con 0x7f10841547d0 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107000cd30 con 0x7f10841547d0 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.997+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10840ca4d0 con 0x7f10841547d0 2024-05-09T05:16:53.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.998+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f10840caa90 con 0x7f10841547d0 2024-05-09T05:16:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.998+0000 7f108ac14640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e6f80 0x7f108414d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.998+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f10841f0eb0 con 0x7f10841e6f80 2024-05-09T05:16:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.998+0000 7f10889b0640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e6f80 0x7f108414d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:53.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.999+0000 7f10889b0640 1 -- 172.21.15.1:0/1703976284 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f10840c9f40 con 0x7f10841e6f80 2024-05-09T05:16:54.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.999+0000 7f10889b0640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e6f80 0x7f108414d7d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f107800f770 tx=0x7f107800fc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:53.999+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10780058b0 con 0x7f10841e6f80 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.000+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f1078018070 con 0x7f10841e6f80 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.000+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e6f80 msgr2=0x7f108414d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.000+0000 7f10817fa640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e6f80 0x7f108414d7d0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f107800f770 tx=0x7f107800fc30 comp rx=0 tx=0).stop 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.000+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e6f80 msgr2=0x7f108414d7d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.000+0000 7f10817fa640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e6f80 0x7f108414d7d0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.000+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f107001a070 con 0x7f10841547d0 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.000+0000 7f10817fa640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1068075d40 0x7f1068078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.001+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1070097800 con 0x7f10841547d0 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.001+0000 7f10889b0640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1068075d40 0x7f1068078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.001+0000 7f10889b0640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1068075d40 0x7f1068078250 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1078001eb0 tx=0x7f1078007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.008+0000 7f108ac14640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e7390 0x7f108414dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.008+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f108415ca00 con 0x7f10841e7390 2024-05-09T05:16:54.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.008+0000 7f10891b1640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e7390 0x7f108414dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.009+0000 7f10891b1640 1 -- 172.21.15.1:0/1703976284 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f10840cded0 con 0x7f10841e7390 2024-05-09T05:16:54.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.009+0000 7f10891b1640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e7390 0x7f108414dd10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f107c02a610 tx=0x7f107c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.009+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c03c970 con 0x7f10841e7390 2024-05-09T05:16:54.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.009+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7f107c03c970 con 0x7f10841e7390 2024-05-09T05:16:54.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.009+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e7390 msgr2=0x7f108414dd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.009+0000 7f10817fa640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e7390 0x7f108414dd10 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f107c02a610 tx=0x7f107c004a40 comp rx=0 tx=0).stop 2024-05-09T05:16:54.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.009+0000 7f10817fa640 1 -- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e7390 msgr2=0x7f108414dd10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:54.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.009+0000 7f10817fa640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f10841e7390 0x7f108414dd10 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.008 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 82, 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 216204, 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:16:54.009 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.010 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.011 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:16:54.012 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.013 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.014 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.015 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:16:54.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.019 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.020 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.021 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:54.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.022 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.022 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:16:54.022 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:16:54.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1068075d40 msgr2=0x7f1068078250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f108ac14640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1068075d40 0x7f1068078250 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f1078001eb0 tx=0x7f1078007000 comp rx=0 tx=0).stop 2024-05-09T05:16:54.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841547d0 msgr2=0x7f1084156250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f108ac14640 1 --2- 172.21.15.1:0/1703976284 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f10841547d0 0x7f1084156250 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f107000e870 tx=0x7f107000ed30 comp rx=0 tx=0).stop 2024-05-09T05:16:54.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f10891b1640 1 -- 172.21.15.1:0/1703976284 reap_dead start 2024-05-09T05:16:54.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f10891b1640 1 -- 172.21.15.1:0/1703976284 reap_dead start 2024-05-09T05:16:54.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 shutdown_connections 2024-05-09T05:16:54.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 >> 172.21.15.1:0/1703976284 conn(0x7f10840ce180 msgr2=0x7f10840cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:54.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 shutdown_connections 2024-05-09T05:16:54.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.010+0000 7f108ac14640 1 -- 172.21.15.1:0/1703976284 wait complete. 2024-05-09T05:16:54.024 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:16:54.024 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:16:54.024 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:16:54.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.159+0000 7fce4f5f3640 1 Processor -- start 2024-05-09T05:16:54.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.160+0000 7fce4f5f3640 1 -- start start 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4f5f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 0x7fce480cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4f5f3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fce480cb350 0x7fce48164ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4f5f3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fce480cbf90 con 0x7fce48072400 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4f5f3640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fce480cb760 con 0x7fce480cb350 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4f5f3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fce480ca3c0 con 0x7fce48154530 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4d38f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fce48072400 0x7fce48070f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55382/0 (socket says 172.21.15.1:55382) 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4d38f640 1 -- 172.21.15.1:0/3220421922 learned_addr learned my addr 172.21.15.1:0/3220421922 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4db90640 1 --2- 172.21.15.1:0/3220421922 >> v2:172.21.15.196:3300/0 conn(0x7fce480cb350 0x7fce48164ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce4cb8e640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 0x7fce480cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.161+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 777125701 0 0) 0x7fce480cbf90 con 0x7fce48072400 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fce34003520 con 0x7fce48072400 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 627778713 0 0) 0x7fce34003520 con 0x7fce48072400 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 msgr2=0x7fce480cae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 0x7fce480cae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 >> v2:172.21.15.196:3300/0 conn(0x7fce480cb350 msgr2=0x7fce48164ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 --2- 172.21.15.1:0/3220421922 >> v2:172.21.15.196:3300/0 conn(0x7fce480cb350 0x7fce48164ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce4815a780 con 0x7fce48072400 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fce440016f0 con 0x7fce48072400 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 >> v1:172.21.15.1:6789/0 conn(0x7fce48072400 legacy=0x7fce48070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce34003a00 0x7fce34003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce34004800 0x7fce34004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 0x7fce34005150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fce480ca3c0 con 0x7fce34003a00 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fce3400ba70 con 0x7fce34004800 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fce3400bbf0 con 0x7fce48154530 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4db90640 1 --2- 172.21.15.1:0/3220421922 >> v2:172.21.15.196:3300/0 conn(0x7fce480cb350 0x7fce48164ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4d38f640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce34003a00 0x7fce34003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4cb8e640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce34004800 0x7fce34004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4db90640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 0x7fce34005150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4d38f640 1 -- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 msgr2=0x7fce34005150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4d38f640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 0x7fce34005150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4d38f640 1 -- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce34004800 msgr2=0x7fce34004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4d38f640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce34004800 0x7fce34004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4d38f640 1 -- 172.21.15.1:0/3220421922 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce34003520 con 0x7fce34003a00 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.162+0000 7fce4db90640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48154530 0x7fce34005150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.163+0000 7fce4d38f640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce34003a00 0x7fce34003e70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c210 tx=0x7fce34005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c667b01e691be052 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.163+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44007d30 con 0x7fce34003a00 2024-05-09T05:16:54.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.163+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4402e070 con 0x7fce34003a00 2024-05-09T05:16:54.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.163+0000 7fce3ffff640 1 -- 172.21.15.1:0/3220421922 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44005d30 con 0x7fce34003a00 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.164+0000 7fce4f5f3640 1 -- 172.21.15.1:0/3220421922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce34003a00 msgr2=0x7fce34003e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.164+0000 7fce4f5f3640 1 --2- 172.21.15.1:0/3220421922 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce34003a00 0x7fce34003e70 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fce4400c210 tx=0x7fce34005830 comp rx=0 tx=0).stop 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.164+0000 7fce4db90640 1 -- 172.21.15.1:0/3220421922 reap_dead start 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.164+0000 7fce4f5f3640 1 -- 172.21.15.1:0/3220421922 shutdown_connections 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.164+0000 7fce4f5f3640 1 -- 172.21.15.1:0/3220421922 >> 172.21.15.1:0/3220421922 conn(0x7fce480d0470 msgr2=0x7fce480cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.164+0000 7fce4f5f3640 1 -- 172.21.15.1:0/3220421922 shutdown_connections 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.164+0000 7fce4f5f3640 1 -- 172.21.15.1:0/3220421922 wait complete. 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.166+0000 7fce4f5f3640 1 Processor -- start 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.166+0000 7fce4f5f3640 1 -- start start 2024-05-09T05:16:54.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.167+0000 7fce4f5f3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce34003a00 0x7fce481eff70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.167+0000 7fce4f5f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48156ba0 0x7fce48159220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.167+0000 7fce4f5f3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce4814b220 0x7fce4814b630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.167+0000 7fce4f5f3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fce48154940 con 0x7fce4814b220 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.167+0000 7fce4f5f3640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fce480ca8f0 con 0x7fce34003a00 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.167+0000 7fce4f5f3640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fce480b2630 con 0x7fce48156ba0 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce4814b220 0x7fce4814b630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce4814b220 0x7fce4814b630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33394/0 (socket says 172.21.15.1:33394) 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 -- 172.21.15.1:0/2049422836 learned_addr learned my addr 172.21.15.1:0/2049422836 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4d38f640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce34003a00 0x7fce481eff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4cb8e640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48156ba0 0x7fce48159220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 -- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48156ba0 msgr2=0x7fce48159220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fce48156ba0 0x7fce48159220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 -- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce34003a00 msgr2=0x7fce481eff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce34003a00 0x7fce481eff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 -- 172.21.15.1:0/2049422836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce481f04b0 con 0x7fce4814b220 2024-05-09T05:16:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4db90640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce4814b220 0x7fce4814b630 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fce48073f10 tx=0x7fce400117d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40003990 con 0x7fce4814b220 2024-05-09T05:16:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fce4001b070 con 0x7fce4814b220 2024-05-09T05:16:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce4814d5f0 con 0x7fce4814b220 2024-05-09T05:16:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.168+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce40003b90 con 0x7fce4814b220 2024-05-09T05:16:54.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.169+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce48155390 con 0x7fce4814b220 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.170+0000 7fce4f5f3640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48155590 0x7fce481559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.170+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fce480cf830 con 0x7fce48155590 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.170+0000 7fce4d38f640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48155590 0x7fce481559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.170+0000 7fce4d38f640 1 -- 172.21.15.1:0/2049422836 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fce48050870 con 0x7fce48155590 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.171+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fce40005d10 con 0x7fce4814b220 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.171+0000 7fce4d38f640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48155590 0x7fce481559a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fce44032000 tx=0x7fce4815a950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.171+0000 7fce3dffb640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fce2c0762d0 0x7fce2c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.171+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fce40097b00 con 0x7fce4814b220 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.172+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44007480 con 0x7fce48155590 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.172+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fce4402e050 con 0x7fce48155590 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.172+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48155590 msgr2=0x7fce481559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.172+0000 7fce3dffb640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48155590 0x7fce481559a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fce44032000 tx=0x7fce4815a950 comp rx=0 tx=0).stop 2024-05-09T05:16:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.172+0000 7fce4cb8e640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fce2c0762d0 0x7fce2c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.172+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48155590 msgr2=0x7fce481559a0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.172+0000 7fce3dffb640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48155590 0x7fce481559a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.172+0000 7fce4cb8e640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fce2c0762d0 0x7fce2c0787e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fce4814c380 tx=0x7fce38009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.178+0000 7fce4f5f3640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48157ce0 0x7fce481fc5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.178+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fce48151c90 con 0x7fce48157ce0 2024-05-09T05:16:54.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.179+0000 7fce4d38f640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48157ce0 0x7fce481fc5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.179+0000 7fce4d38f640 1 -- 172.21.15.1:0/2049422836 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fce480cbcd0 con 0x7fce48157ce0 2024-05-09T05:16:54.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.179+0000 7fce4d38f640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48157ce0 0x7fce481fc5b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce48155560 tx=0x7fce44009030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.179+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce44007d10 con 0x7fce48157ce0 2024-05-09T05:16:54.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4892 (secure 0 0 0) 0x7fce44007d10 con 0x7fce48157ce0 2024-05-09T05:16:54.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48157ce0 msgr2=0x7fce481fc5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce3dffb640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48157ce0 0x7fce481fc5b0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fce48155560 tx=0x7fce44009030 comp rx=0 tx=0).stop 2024-05-09T05:16:54.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce3dffb640 1 -- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48157ce0 msgr2=0x7fce481fc5b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:54.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce3dffb640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fce48157ce0 0x7fce481fc5b0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 82, 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 276459, 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.179 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.180 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.181 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:16:54.182 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.183 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.184 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:16:54.185 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:16:54.186 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.186 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.186 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.186 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:16:54.186 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.187 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:16:54.188 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.189 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:16:54.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.191 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.192 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:54.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.193 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:16:54.193 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:16:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fce2c0762d0 msgr2=0x7fce2c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fce2c0762d0 0x7fce2c0787e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fce4814c380 tx=0x7fce38009190 comp rx=0 tx=0).stop 2024-05-09T05:16:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce4814b220 msgr2=0x7fce4814b630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4db90640 1 -- 172.21.15.1:0/2049422836 reap_dead start 2024-05-09T05:16:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce4814b220 0x7fce4814b630 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fce48073f10 tx=0x7fce400117d0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 shutdown_connections 2024-05-09T05:16:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 --2- 172.21.15.1:0/2049422836 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fce4814b220 0x7fce4814b630 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 >> 172.21.15.1:0/2049422836 conn(0x7fce480d0470 msgr2=0x7fce48153b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 shutdown_connections 2024-05-09T05:16:54.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.180+0000 7fce4f5f3640 1 -- 172.21.15.1:0/2049422836 wait complete. 2024-05-09T05:16:54.195 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:16:54.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:16:54.196 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:16:54.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.316+0000 7f7710b34640 1 Processor -- start 2024-05-09T05:16:54.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.316+0000 7f7710b34640 1 -- start start 2024-05-09T05:16:54.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f7710b34640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f770c070f70 0x7f770c1568e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f7710b34640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f770c156e20 con 0x7f770c070230 2024-05-09T05:16:54.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f7710b34640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f770c0c3f50 con 0x7f770c15d2d0 2024-05-09T05:16:54.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f7710b34640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f770c15ce40 con 0x7f770c070f70 2024-05-09T05:16:54.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f770a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f770c070f70 0x7f770c1568e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f770a59c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f770c070f70 0x7f770c1568e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33408/0 (socket says 172.21.15.1:33408) 2024-05-09T05:16:54.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f770a59c640 1 -- 172.21.15.1:0/4095679773 learned_addr learned my addr 172.21.15.1:0/4095679773 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:54.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f770a59c640 1 -- 172.21.15.1:0/4095679773 >> v1:172.21.15.196:6789/0 conn(0x7f770c15d2d0 legacy=0x7f770c0c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f770a59c640 1 -- 172.21.15.1:0/4095679773 >> v1:172.21.15.1:6789/0 conn(0x7f770c070230 legacy=0x7f770c1672b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.318+0000 7f770a59c640 1 -- 172.21.15.1:0/4095679773 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c06c800 con 0x7f770c070f70 2024-05-09T05:16:54.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.319+0000 7f770a59c640 1 --2- 172.21.15.1:0/4095679773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 0x7f770c1568e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009870 tx=0x7f76fc02a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ef04801264f5f63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.319+0000 7f770959a640 1 -- 172.21.15.1:0/4095679773 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc037070 con 0x7f770c070f70 2024-05-09T05:16:54.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.319+0000 7f770959a640 1 -- 172.21.15.1:0/4095679773 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76fc032640 con 0x7f770c070f70 2024-05-09T05:16:54.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.320+0000 7f770959a640 1 -- 172.21.15.1:0/4095679773 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc032a10 con 0x7f770c070f70 2024-05-09T05:16:54.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.320+0000 7f7710b34640 1 -- 172.21.15.1:0/4095679773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 msgr2=0x7f770c1568e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.320+0000 7f7710b34640 1 --2- 172.21.15.1:0/4095679773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 0x7f770c1568e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009870 tx=0x7f76fc02a3c0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.320+0000 7f7710b34640 1 -- 172.21.15.1:0/4095679773 shutdown_connections 2024-05-09T05:16:54.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.320+0000 7f7710b34640 1 --2- 172.21.15.1:0/4095679773 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 0x7f770c1568e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.320+0000 7f7710b34640 1 -- 172.21.15.1:0/4095679773 >> 172.21.15.1:0/4095679773 conn(0x7f770c06d800 msgr2=0x7f770c06dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:54.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.320+0000 7f7710b34640 1 -- 172.21.15.1:0/4095679773 shutdown_connections 2024-05-09T05:16:54.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.320+0000 7f7710b34640 1 -- 172.21.15.1:0/4095679773 wait complete. 2024-05-09T05:16:54.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.322+0000 7f7710b34640 1 Processor -- start 2024-05-09T05:16:54.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.322+0000 7f7710b34640 1 -- start start 2024-05-09T05:16:54.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7710b34640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c070230 0x7f770c159a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7710b34640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 0x7f770c149930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7710b34640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f770c15d2d0 0x7f770c159160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7710b34640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f770c06cdc0 con 0x7f770c070f70 2024-05-09T05:16:54.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7710b34640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f770c06d0a0 con 0x7f770c15d2d0 2024-05-09T05:16:54.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7710b34640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f770c15cad0 con 0x7f770c070230 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 0x7f770c149930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 0x7f770c149930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33420/0 (socket says 172.21.15.1:33420) 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 -- 172.21.15.1:0/897823588 learned_addr learned my addr 172.21.15.1:0/897823588 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f770a59c640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c070230 0x7f770c159a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f770ad9d640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f770c15d2d0 0x7f770c159160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 -- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c070230 msgr2=0x7f770c159a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c070230 0x7f770c159a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 -- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f770c15d2d0 msgr2=0x7f770c159160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f770c15d2d0 0x7f770c159160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 -- 172.21.15.1:0/897823588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c15c070 con 0x7f770c070f70 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.324+0000 7f7709d9b640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 0x7f770c149930 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7700001eb0 tx=0x7f770000c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.325+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7700017070 con 0x7f770c070f70 2024-05-09T05:16:54.324 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.325+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f77000052d0 con 0x7f770c070f70 2024-05-09T05:16:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.325+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77000037f0 con 0x7f770c070f70 2024-05-09T05:16:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.325+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c15a820 con 0x7f770c070f70 2024-05-09T05:16:54.325 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.325+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c15c2f0 con 0x7f770c070f70 2024-05-09T05:16:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.326+0000 7f7710b34640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15b710 0x7f770c14e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.326 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.326+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f770c157b00 con 0x7f770c15b710 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.326+0000 7f770a59c640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15b710 0x7f770c14e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.326+0000 7f770a59c640 1 -- 172.21.15.1:0/897823588 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f770c0696d0 con 0x7f770c15b710 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.326+0000 7f770a59c640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15b710 0x7f770c14e8e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f770c06d720 tx=0x7f76fc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.327+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc042070 con 0x7f770c15b710 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.327+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f770001a070 con 0x7f770c070f70 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.328+0000 7f76faffd640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76ec076250 0x7f76ec078760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.328+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f77000968d0 con 0x7f770c070f70 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.328+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f76fc03e870 con 0x7f770c15b710 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.328+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15b710 msgr2=0x7f770c14e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.328+0000 7f76faffd640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15b710 0x7f770c14e8e0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f770c06d720 tx=0x7f76fc0046c0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.328+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15b710 msgr2=0x7f770c14e8e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:54.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.328+0000 7f76faffd640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15b710 0x7f770c14e8e0 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f770c06d720 tx=0x7f76fc0046c0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.328+0000 7f770ad9d640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76ec076250 0x7f76ec078760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.329 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.329+0000 7f770ad9d640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76ec076250 0x7f76ec078760 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f770c1f0440 tx=0x7f7704009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.336+0000 7f7710b34640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15bb20 0x7f770c14ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.336+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f770c0c4cf0 con 0x7f770c15bb20 2024-05-09T05:16:54.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.336+0000 7f770a59c640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15bb20 0x7f770c14ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.337+0000 7f770a59c640 1 -- 172.21.15.1:0/897823588 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f770c050870 con 0x7f770c15bb20 2024-05-09T05:16:54.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.337+0000 7f770a59c640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15bb20 0x7f770c14ee20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f76fc000c00 tx=0x7f76fc009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.337+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76fc037040 con 0x7f770c15bb20 2024-05-09T05:16:54.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.337+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f76fc03cde0 con 0x7f770c15bb20 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.337+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15bb20 msgr2=0x7f770c14ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.337+0000 7f76faffd640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15bb20 0x7f770c14ee20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f76fc000c00 tx=0x7f76fc009240 comp rx=0 tx=0).stop 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f76faffd640 1 -- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15bb20 msgr2=0x7f770c14ee20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f76faffd640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f770c15bb20 0x7f770c14ee20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 82, 2024-05-09T05:16:54.337 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 35632, 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.338 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.339 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:16:54.340 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:16:54.341 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:16:54.342 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.343 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.344 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.345 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:16:54.346 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:16:54.347 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:54.348 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:16:54.349 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:16:54.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76ec076250 msgr2=0x7f76ec078760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f7710b34640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76ec076250 0x7f76ec078760 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f770c1f0440 tx=0x7f7704009040 comp rx=0 tx=0).stop 2024-05-09T05:16:54.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 msgr2=0x7f770c149930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f7710b34640 1 --2- 172.21.15.1:0/897823588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f770c070f70 0x7f770c149930 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7700001eb0 tx=0x7f770000c890 comp rx=0 tx=0).stop 2024-05-09T05:16:54.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f770ad9d640 1 -- 172.21.15.1:0/897823588 reap_dead start 2024-05-09T05:16:54.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f770ad9d640 1 -- 172.21.15.1:0/897823588 reap_dead start 2024-05-09T05:16:54.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 shutdown_connections 2024-05-09T05:16:54.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 >> 172.21.15.1:0/897823588 conn(0x7f770c06d800 msgr2=0x7f770c070640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:54.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 shutdown_connections 2024-05-09T05:16:54.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.338+0000 7f7710b34640 1 -- 172.21.15.1:0/897823588 wait complete. 2024-05-09T05:16:54.352 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:16:54.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-05-09T05:16:54.352 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:16:54.352 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-05-09T05:16:54.352 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:16:54.353 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T05:16:54.353 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:16:54.368 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T05:16:54.368 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:16:54.368 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:16:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.488+0000 7f64bfbb2640 1 Processor -- start 2024-05-09T05:16:54.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.488+0000 7f64bfbb2640 1 -- start start 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bfbb2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64b8072120 0x7f64b8156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bfbb2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b80c5e00 con 0x7f64b80caab0 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bfbb2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f64b80c60e0 con 0x7f64b815d1d0 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bfbb2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f64b80ca340 con 0x7f64b8072120 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bd94e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64b8072120 0x7f64b8156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bd94e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f64b8072120 0x7f64b8156a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33444/0 (socket says 172.21.15.1:33444) 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bd94e640 1 -- 172.21.15.1:0/2540424460 learned_addr learned my addr 172.21.15.1:0/2540424460 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bd14d640 1 -- 172.21.15.1:0/2540424460 >> v1:172.21.15.196:6789/0 conn(0x7f64b815d1d0 legacy=0x7f64b8070e20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bd94e640 1 -- 172.21.15.1:0/2540424460 >> v1:172.21.15.196:6789/0 conn(0x7f64b815d1d0 legacy=0x7f64b8070e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:16:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bd94e640 1 -- 172.21.15.1:0/2540424460 >> v1:172.21.15.1:6789/0 conn(0x7f64b80caab0 legacy=0x7f64b80caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.490+0000 7f64bd94e640 1 -- 172.21.15.1:0/2540424460 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b80c63c0 con 0x7f64b8072120 2024-05-09T05:16:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.491+0000 7f64bd94e640 1 --2- 172.21.15.1:0/2540424460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b8072120 0x7f64b8156a80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009870 tx=0x7f64b402a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69de841e77ef8128 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.491+0000 7f64bc94c640 1 -- 172.21.15.1:0/2540424460 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b4037070 con 0x7f64b8072120 2024-05-09T05:16:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.491+0000 7f64bc94c640 1 -- 172.21.15.1:0/2540424460 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64b4032440 con 0x7f64b8072120 2024-05-09T05:16:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.491+0000 7f64bc94c640 1 -- 172.21.15.1:0/2540424460 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b4032810 con 0x7f64b8072120 2024-05-09T05:16:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.492+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2540424460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b8072120 msgr2=0x7f64b8156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.492+0000 7f64bfbb2640 1 --2- 172.21.15.1:0/2540424460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b8072120 0x7f64b8156a80 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f64b4009870 tx=0x7f64b402a250 comp rx=0 tx=0).stop 2024-05-09T05:16:54.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.492+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2540424460 shutdown_connections 2024-05-09T05:16:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.492+0000 7f64bfbb2640 1 --2- 172.21.15.1:0/2540424460 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b8072120 0x7f64b8156a80 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.492+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2540424460 >> 172.21.15.1:0/2540424460 conn(0x7f64b80731d0 msgr2=0x7f64b80cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.492+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2540424460 shutdown_connections 2024-05-09T05:16:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.492+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2540424460 wait complete. 2024-05-09T05:16:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.494+0000 7f64bfbb2640 1 Processor -- start 2024-05-09T05:16:54.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.494+0000 7f64bfbb2640 1 -- start start 2024-05-09T05:16:54.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.495+0000 7f64bfbb2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b8072120 0x7f64b81f62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bfbb2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b80caab0 0x7f64b814c220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bfbb2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f64b815d1d0 0x7f64b814c760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bfbb2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f64b815d5e0 con 0x7f64b80caab0 2024-05-09T05:16:54.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bfbb2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f64b80ca620 con 0x7f64b8072120 2024-05-09T05:16:54.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bfbb2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f64b8071530 con 0x7f64b815d1d0 2024-05-09T05:16:54.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd94e640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b8072120 msgr2=0x7f64b81f62c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:16:54.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b80caab0 0x7f64b814c220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd94e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b8072120 0x7f64b81f62c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b80caab0 0x7f64b814c220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33448/0 (socket says 172.21.15.1:33448) 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 -- 172.21.15.1:0/2312434809 learned_addr learned my addr 172.21.15.1:0/2312434809 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64be14f640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f64b815d1d0 0x7f64b814c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 -- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f64b815d1d0 msgr2=0x7f64b814c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f64b815d1d0 0x7f64b814c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 -- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b8072120 msgr2=0x7f64b81f62c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b8072120 0x7f64b81f62c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 -- 172.21.15.1:0/2312434809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64b81f6c10 con 0x7f64b80caab0 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64bd14d640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b80caab0 0x7f64b814c220 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f64a800b580 tx=0x7f64a800ba40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.496+0000 7f64ae7fc640 1 -- 172.21.15.1:0/2312434809 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a8013020 con 0x7f64b80caab0 2024-05-09T05:16:54.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.497+0000 7f64ae7fc640 1 -- 172.21.15.1:0/2312434809 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f64a8003de0 con 0x7f64b80caab0 2024-05-09T05:16:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.497+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64b814cca0 con 0x7f64b80caab0 2024-05-09T05:16:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.497+0000 7f64ae7fc640 1 -- 172.21.15.1:0/2312434809 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64a80052b0 con 0x7f64b80caab0 2024-05-09T05:16:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.497+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64b81f6e80 con 0x7f64b80caab0 2024-05-09T05:16:54.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.498+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64b80766d0 con 0x7f64b80caab0 2024-05-09T05:16:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.499+0000 7f64ae7fc640 1 -- 172.21.15.1:0/2312434809 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f64a8016020 con 0x7f64b80caab0 2024-05-09T05:16:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.499+0000 7f64ae7fc640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f649c0762b0 0x7f649c0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.499+0000 7f64ae7fc640 1 -- 172.21.15.1:0/2312434809 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f64a80973f0 con 0x7f64b80caab0 2024-05-09T05:16:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.499+0000 7f64bd94e640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f649c0762b0 0x7f649c0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:54.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.500+0000 7f64bd94e640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f649c0762b0 0x7f649c0787c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f64b4034000 tx=0x7f64b402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:54.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.505+0000 7f64ae7fc640 1 -- 172.21.15.1:0/2312434809 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f64b80766d0 con 0x7f64b80caab0 2024-05-09T05:16:54.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.859+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f64b8050870 con 0x7f64b80caab0 2024-05-09T05:16:54.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.859+0000 7f64ae7fc640 1 -- 172.21.15.1:0/2312434809 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f64a801d020 con 0x7f64b80caab0 2024-05-09T05:16:54.858 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:16:54.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f649c0762b0 msgr2=0x7f649c0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f649c0762b0 0x7f649c0787c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f64b4034000 tx=0x7f64b402e000 comp rx=0 tx=0).stop 2024-05-09T05:16:54.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b80caab0 msgr2=0x7f64b814c220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:54.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b80caab0 0x7f64b814c220 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f64a800b580 tx=0x7f64a800ba40 comp rx=0 tx=0).stop 2024-05-09T05:16:54.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 shutdown_connections 2024-05-09T05:16:54.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f649c0762b0 0x7f649c0787c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f64b815d1d0 0x7f64b814c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f64b80caab0 0x7f64b814c220 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.861+0000 7f64bfbb2640 1 --2- 172.21.15.1:0/2312434809 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f64b8072120 0x7f64b81f62c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:54.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.862+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 >> 172.21.15.1:0/2312434809 conn(0x7f64b80731d0 msgr2=0x7f64b8150be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:54.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.862+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 shutdown_connections 2024-05-09T05:16:54.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:54.862+0000 7f64bfbb2640 1 -- 172.21.15.1:0/2312434809 wait complete. 2024-05-09T05:16:57.872 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:16:57.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.000+0000 7f1aba2e0640 1 Processor -- start 2024-05-09T05:16:57.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.000+0000 7f1aba2e0640 1 -- start start 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.001+0000 7f1aba2e0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1ab406f1b0 0x7f1ab40ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.001+0000 7f1aba2e0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 0x7f1ab415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.001+0000 7f1aba2e0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1ab406f6d0 con 0x7f1ab40d16b0 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.001+0000 7f1aba2e0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f1ab40cd0e0 con 0x7f1ab406f1b0 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.001+0000 7f1aba2e0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab40cc4d0 con 0x7f1ab415ca90 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.001+0000 7f1ab37fe640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f1ab406f1b0 msgr2=0x7f1ab40ce740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab37fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1ab406f1b0 0x7f1ab40ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab3fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1ab40d16b0 0x7f1ab40d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55414/0 (socket says 172.21.15.1:55414) 2024-05-09T05:16:58.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab3fff640 1 -- 172.21.15.1:0/2103386478 learned_addr learned my addr 172.21.15.1:0/2103386478 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab887d640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 0x7f1ab415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3536441741 0 0) 0x7f1ab406f6d0 con 0x7f1ab40d16b0 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a9c003520 con 0x7f1ab40d16b0 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1262799860 0 0) 0x7f1a9c003520 con 0x7f1ab40d16b0 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 msgr2=0x7f1ab415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 0x7f1ab415cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 >> v2:172.21.15.196:3300/0 conn(0x7f1ab406f1b0 msgr2=0x7f1ab40ce740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 --2- 172.21.15.1:0/2103386478 >> v2:172.21.15.196:3300/0 conn(0x7f1ab406f1b0 0x7f1ab40ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab40cc7b0 con 0x7f1ab40d16b0 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab887d640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 0x7f1ab415cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f1aa40016f0 con 0x7f1ab40d16b0 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 >> v1:172.21.15.1:6789/0 conn(0x7f1ab40d16b0 legacy=0x7f1ab40d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:58.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1a9c003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a9c004800 0x7f1a9c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 0x7f1a9c0051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab40cd0e0 con 0x7f1a9c0039b0 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab40cc4d0 con 0x7f1a9c004800 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1a9c00bbe0 con 0x7f1ab415ca90 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab37fe640 1 -- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a9c004800 msgr2=0x7f1a9c004c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.002+0000 7f1ab37fe640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a9c004800 0x7f1a9c004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.003+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1a9c003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.003+0000 7f1ab887d640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 0x7f1a9c0051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.003+0000 7f1ab3fff640 1 -- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 msgr2=0x7f1a9c0051e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.003+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab415ca90 0x7f1a9c0051e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.003+0000 7f1ab3fff640 1 -- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a9c004800 msgr2=0x7f1a9c004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.003+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a9c004800 0x7f1a9c004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.003+0000 7f1ab3fff640 1 -- 172.21.15.1:0/2103386478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a9c003520 con 0x7f1a9c0039b0 2024-05-09T05:16:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.004+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1a9c003e20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c210 tx=0x7f1a9c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a85d7f728341287d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.004+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4007d50 con 0x7f1a9c0039b0 2024-05-09T05:16:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.004+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa4005ad0 con 0x7f1a9c0039b0 2024-05-09T05:16:58.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.004+0000 7f1ab2ffd640 1 -- 172.21.15.1:0/2103386478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4007070 con 0x7f1a9c0039b0 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.005+0000 7f1aba2e0640 1 -- 172.21.15.1:0/2103386478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 msgr2=0x7f1a9c003e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.005+0000 7f1aba2e0640 1 --2- 172.21.15.1:0/2103386478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1a9c003e20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c210 tx=0x7f1a9c005830 comp rx=0 tx=0).stop 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.005+0000 7f1ab887d640 1 -- 172.21.15.1:0/2103386478 reap_dead start 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.005+0000 7f1aba2e0640 1 -- 172.21.15.1:0/2103386478 shutdown_connections 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.005+0000 7f1aba2e0640 1 -- 172.21.15.1:0/2103386478 >> 172.21.15.1:0/2103386478 conn(0x7f1ab40b3660 msgr2=0x7f1ab40b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.005+0000 7f1aba2e0640 1 -- 172.21.15.1:0/2103386478 shutdown_connections 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.005+0000 7f1aba2e0640 1 -- 172.21.15.1:0/2103386478 wait complete. 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.007+0000 7f1aba2e0640 1 Processor -- start 2024-05-09T05:16:58.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.007+0000 7f1aba2e0640 1 -- start start 2024-05-09T05:16:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.008+0000 7f1aba2e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1ab40ca110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.008+0000 7f1aba2e0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab41583d0 0x7f1ab41e5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.008+0000 7f1aba2e0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ab4158920 0x7f1ab4151cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.008+0000 7f1aba2e0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab40cec80 con 0x7f1a9c0039b0 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.008+0000 7f1aba2e0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1ab40ccd70 con 0x7f1ab4158920 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.008+0000 7f1aba2e0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1ab40cee00 con 0x7f1ab41583d0 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.008+0000 7f1ab3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1ab40ca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1ab40ca110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33470/0 (socket says 172.21.15.1:33470) 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab3fff640 1 -- 172.21.15.1:0/4029182347 learned_addr learned my addr 172.21.15.1:0/4029182347 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab887d640 1 -- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ab4158920 msgr2=0x7f1ab4151cd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab887d640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ab4158920 0x7f1ab4151cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab37fe640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab41583d0 0x7f1ab41e5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab3fff640 1 -- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab41583d0 msgr2=0x7f1ab41e5000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab41583d0 0x7f1ab41e5000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab3fff640 1 -- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ab4158920 msgr2=0x7f1ab4151cd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ab4158920 0x7f1ab4151cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab3fff640 1 -- 172.21.15.1:0/4029182347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ab40ca650 con 0x7f1a9c0039b0 2024-05-09T05:16:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab3fff640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1ab40ca110 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c270 tx=0x7f1ab415c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/4029182347 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4043070 con 0x7f1a9c0039b0 2024-05-09T05:16:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/4029182347 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1aa4038040 con 0x7f1a9c0039b0 2024-05-09T05:16:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ab4157590 con 0x7f1a9c0039b0 2024-05-09T05:16:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/4029182347 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1aa4007290 con 0x7f1a9c0039b0 2024-05-09T05:16:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.009+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ab4157b50 con 0x7f1a9c0039b0 2024-05-09T05:16:58.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.010+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ab4050870 con 0x7f1a9c0039b0 2024-05-09T05:16:58.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.011+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/4029182347 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1aa4007430 con 0x7f1a9c0039b0 2024-05-09T05:16:58.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.012+0000 7f1ab0ff9640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1aa0076280 0x7f1aa0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:16:58.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.012+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/4029182347 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1aa40b7220 con 0x7f1a9c0039b0 2024-05-09T05:16:58.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.012+0000 7f1ab37fe640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1aa0076280 0x7f1aa0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:16:58.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.013+0000 7f1ab37fe640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1aa0076280 0x7f1aa0078790 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40d2200 tx=0x7f1aa8003b90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:16:58.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.018+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/4029182347 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1aa407a990 con 0x7f1a9c0039b0 2024-05-09T05:16:58.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.312+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1ab41570b0 con 0x7f1a9c0039b0 2024-05-09T05:16:58.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.312+0000 7f1ab0ff9640 1 -- 172.21.15.1:0/4029182347 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f1aa40016b0 con 0x7f1a9c0039b0 2024-05-09T05:16:58.311 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":82,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1aa0076280 msgr2=0x7f1aa0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1aa0076280 0x7f1aa0078790 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f1ab40d2200 tx=0x7f1aa8003b90 comp rx=0 tx=0).stop 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 msgr2=0x7f1ab40ca110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1ab40ca110 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f1aa400c270 tx=0x7f1ab415c740 comp rx=0 tx=0).stop 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 shutdown_connections 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1aa0076280 0x7f1aa0078790 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1ab4158920 0x7f1ab4151cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1ab41583d0 0x7f1ab41e5000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 --2- 172.21.15.1:0/4029182347 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a9c0039b0 0x7f1ab40ca110 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:16:58.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 >> 172.21.15.1:0/4029182347 conn(0x7f1ab40b3660 msgr2=0x7f1ab4075e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:16:58.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 shutdown_connections 2024-05-09T05:16:58.315 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:16:58.315+0000 7f1aba2e0640 1 -- 172.21.15.1:0/4029182347 wait complete. 2024-05-09T05:17:01.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:17:01.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.534+0000 7ffadda0c640 1 Processor -- start 2024-05-09T05:17:01.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.534+0000 7ffadda0c640 1 -- start start 2024-05-09T05:17:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.535+0000 7ffadda0c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 0x7ffad80ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.535+0000 7ffadda0c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad80c86a0 con 0x7ffad8156e90 2024-05-09T05:17:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.536+0000 7ffadda0c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad80c8980 con 0x7ffad8156a80 2024-05-09T05:17:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.536+0000 7ffadda0c640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ffad80cb910 con 0x7ffad80c5ca0 2024-05-09T05:17:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.536+0000 7fface7fc640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7ffad8156a80 legacy=0x7ffad806df20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:01.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.536+0000 7ffad6ffd640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 msgr2=0x7ffad80ca4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.536+0000 7ffad6ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 0x7ffad80ca4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.536+0000 7ffad77fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffad8156e90 0x7ffad80c7750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41446/0 (socket says 172.21.15.1:41446) 2024-05-09T05:17:01.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.536+0000 7ffad77fe640 1 -- 172.21.15.1:0/1089006427 learned_addr learned my addr 172.21.15.1:0/1089006427 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:17:01.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.737+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.196:6789/0 conn(0x7ffad8156a80 legacy=0x7ffad806df20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:01.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.737+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 msgr2=0x7ffad80ca4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:01.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:01.737+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 0x7ffad80ca4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:17:02.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:02.138+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 msgr2=0x7ffad80ca4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:02.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:02.138+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.196:6789/0 conn(0x7ffad8156a80 legacy=0x7ffad806df20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:02.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:02.138+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 0x7ffad80ca4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:17:02.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:02.939+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.196:6789/0 conn(0x7ffad8156a80 legacy=0x7ffad806df20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:02.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:02.939+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 msgr2=0x7ffad80ca4a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:02.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:02.939+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 0x7ffad80ca4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:17:04.302 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:17:04.304 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:17:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.535+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 msgr2=0x7ffad80ca4a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad57fa640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.196:3300/0 conn(0x7ffad80c5ca0 0x7ffad80ca4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.196:6789/0 conn(0x7ffad8156a80 legacy=0x7ffad806df20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.1:6789/0 conn(0x7ffad8156e90 legacy=0x7ffad80c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad57fa640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad57fa640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ffab8008da0 con 0x7ffab80036c0 2024-05-09T05:17:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffab8008f50 con 0x7ffab80061f0 2024-05-09T05:17:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:04.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:17:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:17:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.536+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:04.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:17:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:17:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:17:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:04.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:04.737+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:17:05.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:05.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.138+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:17:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:17:05.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:17:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:17:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:05.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:05.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:05.939+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 msgr2=0x7ffab8008700 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad57fa640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab80061f0 0x7ffab8008700 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 msgr2=0x7ffab8005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad57fa640 1 --2- 172.21.15.1:0/1089006427 >> v2:172.21.15.1:3300/0 conn(0x7ffab80036c0 0x7ffab8005b90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad77fe640 1 -- 172.21.15.1:0/1089006427 reap_dead start 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad57fa640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad80c8c60 con 0x7ffab8006b60 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffad80c8980 con 0x7ffab8008780 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad57fa640 1 -- 172.21.15.1:0/1089006427 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffab8008da0 con 0x7ffab8003ad0 2024-05-09T05:17:07.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad77fe640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.196:6789/0 conn(0x7ffab8008780 legacy=0x7ffab800d410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 msgr2=0x7ffab8006620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:17:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 msgr2=0x7ffab8006620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:07.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.536+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:07.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.737+0000 7ffad77fe640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.196:6789/0 conn(0x7ffab8008780 legacy=0x7ffab800d410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:07.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.738+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:07.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.738+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 msgr2=0x7ffab8006620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:17:07.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.738+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 msgr2=0x7ffab8006620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:07.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.738+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:07.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:07.738+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:17:08.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.138+0000 7ffad77fe640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.196:6789/0 conn(0x7ffab8008780 legacy=0x7ffab800d410 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:08.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.139+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:08.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.139+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 msgr2=0x7ffab8006620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:17:08.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.139+0000 7ffad6ffd640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 msgr2=0x7ffab8006620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:08.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.139+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:08.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.139+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:17:08.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.813+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1233967472 0 0) 0x7ffad80c8c60 con 0x7ffab8006b60 2024-05-09T05:17:08.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.813+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffac4003520 con 0x7ffab8006b60 2024-05-09T05:17:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1213979710 0 0) 0x7ffac4003520 con 0x7ffab8006b60 2024-05-09T05:17:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 msgr2=0x7ffab8006620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffab8006620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.196:6789/0 conn(0x7ffab8008780 legacy=0x7ffab800d410 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffab8010ef0 con 0x7ffab8006b60 2024-05-09T05:17:08.816 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7ffac00098b0 con 0x7ffab8006b60 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 >> v1:172.21.15.1:6789/0 conn(0x7ffab8006b60 legacy=0x7ffab8006f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffac40039e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac4003ff0 0x7ffac4004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffac4004d40 0x7ffac4005210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffab8008da0 con 0x7ffac4003ff0 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffac400bc70 con 0x7ffac4004d40 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffac400be70 con 0x7ffab8003ad0 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad77fe640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffac4004d40 msgr2=0x7ffac4005210 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac4003ff0 0x7ffac4004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:08.817 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad77fe640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffac4004d40 0x7ffac4005210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffac40039e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 msgr2=0x7ffac40039e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffab8003ad0 0x7ffac40039e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffac4004d40 msgr2=0x7ffac4005210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffac4004d40 0x7ffac4005210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.817+0000 7fface7fc640 1 -- 172.21.15.1:0/1089006427 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffac4003520 con 0x7ffac4003ff0 2024-05-09T05:17:08.818 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.818+0000 7fface7fc640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac4003ff0 0x7ffac4004800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffac0003930 tx=0x7ffac4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c52556c318550c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.818+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0037070 con 0x7ffac4003ff0 2024-05-09T05:17:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.818+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffac0032600 con 0x7ffac4003ff0 2024-05-09T05:17:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.818+0000 7ffad67fc640 1 -- 172.21.15.1:0/1089006427 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0005050 con 0x7ffac4003ff0 2024-05-09T05:17:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.819+0000 7ffadda0c640 1 -- 172.21.15.1:0/1089006427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac4003ff0 msgr2=0x7ffac4004800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:08.819 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.819+0000 7ffadda0c640 1 --2- 172.21.15.1:0/1089006427 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffac4003ff0 0x7ffac4004800 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7ffac0003930 tx=0x7ffac4005830 comp rx=0 tx=0).stop 2024-05-09T05:17:08.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.819+0000 7ffad77fe640 1 -- 172.21.15.1:0/1089006427 reap_dead start 2024-05-09T05:17:08.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.819+0000 7ffadda0c640 1 -- 172.21.15.1:0/1089006427 shutdown_connections 2024-05-09T05:17:08.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.819+0000 7ffadda0c640 1 -- 172.21.15.1:0/1089006427 >> 172.21.15.1:0/1089006427 conn(0x7ffad80745a0 msgr2=0x7ffad8071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:08.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.820+0000 7ffadda0c640 1 -- 172.21.15.1:0/1089006427 shutdown_connections 2024-05-09T05:17:08.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.820+0000 7ffadda0c640 1 -- 172.21.15.1:0/1089006427 wait complete. 2024-05-09T05:17:08.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.822+0000 7ffadda0c640 1 Processor -- start 2024-05-09T05:17:08.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.822+0000 7ffadda0c640 1 -- start start 2024-05-09T05:17:08.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffadda0c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffac4003ff0 0x7ffad80cf130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffadda0c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad81f63e0 0x7ffad81f2b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffadda0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffad8149ca0 0x7ffad814a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffadda0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffad80cad60 con 0x7ffad8149ca0 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffadda0c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffad80cb5a0 con 0x7ffad81f63e0 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffadda0c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffad80ca4a0 con 0x7ffac4003ff0 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7fface7fc640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad81f63e0 msgr2=0x7ffad81f2b70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7fface7fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad81f63e0 0x7ffad81f2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffad77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffad8149ca0 0x7ffad814a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffad77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffad8149ca0 0x7ffad814a0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41478/0 (socket says 172.21.15.1:41478) 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffad77fe640 1 -- 172.21.15.1:0/3462097981 learned_addr learned my addr 172.21.15.1:0/3462097981 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.824+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffac4003ff0 0x7ffad80cf130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:08.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad77fe640 1 -- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffac4003ff0 msgr2=0x7ffad80cf130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad77fe640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffac4003ff0 0x7ffad80cf130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad77fe640 1 -- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad81f63e0 msgr2=0x7ffad81f2b70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad77fe640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad81f63e0 0x7ffad81f2b70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad77fe640 1 -- 172.21.15.1:0/3462097981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffad81f30b0 con 0x7ffad8149ca0 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad77fe640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffad8149ca0 0x7ffad814a0b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffad80c76b0 tx=0x7ffad000ce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad4ff9640 1 -- 172.21.15.1:0/3462097981 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad0017070 con 0x7ffad8149ca0 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad4ff9640 1 -- 172.21.15.1:0/3462097981 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffad000aa50 con 0x7ffad8149ca0 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffad81f67f0 con 0x7ffad8149ca0 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffad4ff9640 1 -- 172.21.15.1:0/3462097981 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffad000ae20 con 0x7ffad8149ca0 2024-05-09T05:17:08.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.825+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffad8155780 con 0x7ffad8149ca0 2024-05-09T05:17:08.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.826+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffad8155210 con 0x7ffad8149ca0 2024-05-09T05:17:08.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.827+0000 7ffad4ff9640 1 -- 172.21.15.1:0/3462097981 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffad001a070 con 0x7ffad8149ca0 2024-05-09T05:17:08.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.828+0000 7ffad4ff9640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffab8077630 0x7ffab8079b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:08.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.828+0000 7ffad4ff9640 1 -- 172.21.15.1:0/3462097981 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ffad0097050 con 0x7ffad8149ca0 2024-05-09T05:17:08.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.828+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffab8077630 0x7ffab8079b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:08.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.829+0000 7ffad6ffd640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffab8077630 0x7ffab8079b40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ffac8003fd0 tx=0x7ffac800d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:08.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:08.833+0000 7ffad4ff9640 1 -- 172.21.15.1:0/3462097981 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffad00608a0 con 0x7ffad8149ca0 2024-05-09T05:17:09.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.132+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ffad8050870 con 0x7ffad8149ca0 2024-05-09T05:17:09.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.133+0000 7ffad4ff9640 1 -- 172.21.15.1:0/3462097981 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7ffad00601a0 con 0x7ffad8149ca0 2024-05-09T05:17:09.131 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":86,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffab8077630 msgr2=0x7ffab8079b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffab8077630 0x7ffab8079b40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7ffac8003fd0 tx=0x7ffac800d040 comp rx=0 tx=0).stop 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffad8149ca0 msgr2=0x7ffad814a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffad8149ca0 0x7ffad814a0b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7ffad80c76b0 tx=0x7ffad000ce40 comp rx=0 tx=0).stop 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 shutdown_connections 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffab8077630 0x7ffab8079b40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffad81f63e0 0x7ffad81f2b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffad8149ca0 0x7ffad814a0b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 --2- 172.21.15.1:0/3462097981 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffac4003ff0 0x7ffad80cf130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 >> 172.21.15.1:0/3462097981 conn(0x7ffad80745a0 msgr2=0x7ffad815c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 shutdown_connections 2024-05-09T05:17:09.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.135+0000 7ffadda0c640 1 -- 172.21.15.1:0/3462097981 wait complete. 2024-05-09T05:17:09.146 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:17:09.146 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:17:09.147 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:17:09.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.287+0000 7f765dbc7640 1 Processor -- start 2024-05-09T05:17:09.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.287+0000 7f765dbc7640 1 -- start start 2024-05-09T05:17:09.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f765dbc7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76580c5600 0x7f765815d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f765dbc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76580cc1b0 con 0x7f765806dde0 2024-05-09T05:17:09.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f765dbc7640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76580cc490 con 0x7f76580736b0 2024-05-09T05:17:09.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f765dbc7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f76580cbf60 con 0x7f76580c5600 2024-05-09T05:17:09.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76577fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f765806dde0 0x7f76580c70b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41476/0 (socket says 172.21.15.1:41476) 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76577fe640 1 -- 172.21.15.1:0/553022920 learned_addr learned my addr 172.21.15.1:0/553022920 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f7656ffd640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f76580736b0 legacy=0x7f7658165220 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f7657fff640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76580c5600 0x7f765815d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 370166205 0 0) 0x7f76580cc1b0 con 0x7f765806dde0 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7640003520 con 0x7f765806dde0 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2869036603 0 0) 0x7f7640003520 con 0x7f765806dde0 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76580c5600 msgr2=0x7f765815d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76567fc640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76580c5600 0x7f765815d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 >> v1:172.21.15.196:6789/0 conn(0x7f76580736b0 legacy=0x7f7658165220 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765806c6f0 con 0x7f765806dde0 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.289+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f76480016f0 con 0x7f765806dde0 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 >> v1:172.21.15.1:6789/0 conn(0x7f765806dde0 legacy=0x7f76580c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76567fc640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76400039e0 0x7f7640003e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76567fc640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76580c5600 0x7f7640004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76567fc640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7640004d40 0x7f7640005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76580cbf60 con 0x7f7640004d40 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f764000bb50 con 0x7f76400039e0 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f764000bd20 con 0x7f76580c5600 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76577fe640 1 -- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76400039e0 msgr2=0x7f7640003e30 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f76577fe640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76400039e0 0x7f7640003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f7657fff640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7640004d40 0x7f7640005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f7656ffd640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76580c5600 0x7f7640004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f7657fff640 1 -- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76580c5600 msgr2=0x7f7640004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f7657fff640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76580c5600 0x7f7640004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f7657fff640 1 -- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76400039e0 msgr2=0x7f7640003e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f7657fff640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76400039e0 0x7f7640003e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.290+0000 7f7657fff640 1 -- 172.21.15.1:0/553022920 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7640003520 con 0x7f7640004d40 2024-05-09T05:17:09.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.291+0000 7f7657fff640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7640004d40 0x7f7640005190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7650009fc0 tx=0x7f7640005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d01108c4dc90a4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.291+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650034020 con 0x7f7640004d40 2024-05-09T05:17:09.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.291+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76500305f0 con 0x7f7640004d40 2024-05-09T05:17:09.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.291+0000 7f76567fc640 1 -- 172.21.15.1:0/553022920 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650009c10 con 0x7f7640004d40 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.292+0000 7f765dbc7640 1 -- 172.21.15.1:0/553022920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7640004d40 msgr2=0x7f7640005190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.292+0000 7f765dbc7640 1 --2- 172.21.15.1:0/553022920 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7640004d40 0x7f7640005190 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f7650009fc0 tx=0x7f7640005830 comp rx=0 tx=0).stop 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.292+0000 7f7657fff640 1 -- 172.21.15.1:0/553022920 reap_dead start 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.292+0000 7f765dbc7640 1 -- 172.21.15.1:0/553022920 shutdown_connections 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.292+0000 7f765dbc7640 1 -- 172.21.15.1:0/553022920 >> 172.21.15.1:0/553022920 conn(0x7f76580ce180 msgr2=0x7f76580ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.292+0000 7f765dbc7640 1 -- 172.21.15.1:0/553022920 shutdown_connections 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.292+0000 7f765dbc7640 1 -- 172.21.15.1:0/553022920 wait complete. 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.294+0000 7f765dbc7640 1 Processor -- start 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.294+0000 7f765dbc7640 1 -- start start 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f765dbc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f765815da70 0x7f76581f5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f765dbc7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f765815e5e0 0x7f765814cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f765dbc7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f765815e9f0 0x7f765814d250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f765dbc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76580c77c0 con 0x7f765815da70 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f765dbc7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f765806ccb0 con 0x7f765815e9f0 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f765dbc7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f76580c7940 con 0x7f765815e5e0 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f76577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f765815da70 0x7f76581f5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f7657fff640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f765815e9f0 msgr2=0x7f765814d250 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f7657fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f765815e9f0 0x7f765814d250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f76577fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f765815da70 0x7f76581f5d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41496/0 (socket says 172.21.15.1:41496) 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.295+0000 7f76577fe640 1 -- 172.21.15.1:0/1771607215 learned_addr learned my addr 172.21.15.1:0/1771607215 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f7656ffd640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f765815e5e0 0x7f765814cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f76577fe640 1 -- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f765815e5e0 msgr2=0x7f765814cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f76577fe640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f765815e5e0 0x7f765814cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f76577fe640 1 -- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f765815e9f0 msgr2=0x7f765814d250 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f76577fe640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f765815e9f0 0x7f765814d250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f76577fe640 1 -- 172.21.15.1:0/1771607215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7658159e20 con 0x7f765815da70 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f76577fe640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f765815da70 0x7f76581f5d70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f764800cad0 tx=0x7f764800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648018070 con 0x7f765815da70 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7648004cb0 con 0x7f765815da70 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648004e60 con 0x7f765815da70 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7658159b50 con 0x7f765815da70 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.296+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7658160690 con 0x7f765815da70 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.297+0000 7f765dbc7640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658160970 0x7f76581f2800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.297+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76580c9630 con 0x7f7658160970 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.297+0000 7f7656ffd640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658160970 0x7f76581f2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.298+0000 7f7656ffd640 1 -- 172.21.15.1:0/1771607215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7658050870 con 0x7f7658160970 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.298+0000 7f7656ffd640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658160970 0x7f76581f2800 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7658072230 tx=0x7f764c009ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.298+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764c011860 con 0x7f7658160970 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.298+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f764c011a00 con 0x7f7658160970 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.298+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658160970 msgr2=0x7f76581f2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.298+0000 7f763ffff640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658160970 0x7f76581f2800 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f7658072230 tx=0x7f764c009ac0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.298+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658160970 msgr2=0x7f76581f2800 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.298+0000 7f763ffff640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658160970 0x7f76581f2800 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.299+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f764801b070 con 0x7f765815da70 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.299+0000 7f763ffff640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7644075c80 0x7f7644078190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.300+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7648096a80 con 0x7f765815da70 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.300+0000 7f7656ffd640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7644075c80 0x7f7644078190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.300+0000 7f7656ffd640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7644075c80 0x7f7644078190 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f764c000c00 tx=0x7f764c008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.305+0000 7f765dbc7640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658158dd0 0x7f76581f2d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.306+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76580748f0 con 0x7f7658158dd0 2024-05-09T05:17:09.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.306+0000 7f7657fff640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658158dd0 0x7f76581f2d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.306+0000 7f7657fff640 1 -- 172.21.15.1:0/1771607215 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f765815a770 con 0x7f7658158dd0 2024-05-09T05:17:09.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.306+0000 7f7657fff640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658158dd0 0x7f76581f2d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7650009f90 tx=0x7f7650009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.306+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650034040 con 0x7f7658158dd0 2024-05-09T05:17:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5052 (secure 0 0 0) 0x7f7650003990 con 0x7f7658158dd0 2024-05-09T05:17:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658158dd0 msgr2=0x7f76581f2d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f763ffff640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658158dd0 0x7f76581f2d40 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7650009f90 tx=0x7f7650009750 comp rx=0 tx=0).stop 2024-05-09T05:17:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f763ffff640 1 -- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658158dd0 msgr2=0x7f76581f2d40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:09.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f763ffff640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7658158dd0 0x7f76581f2d40 secure :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f7650009f90 tx=0x7f7650009750 comp rx=0 tx=0).stop 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 231501, 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:17:09.306 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:09.307 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:09.308 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:17:09.309 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:09.310 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:09.311 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:17:09.312 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.313 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.315 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.316 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:17:09.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:17:09.318 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:17:09.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.320 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:17:09.320 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:17:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7644075c80 msgr2=0x7f7644078190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7644075c80 0x7f7644078190 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f764c000c00 tx=0x7f764c008030 comp rx=0 tx=0).stop 2024-05-09T05:17:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f765815da70 msgr2=0x7f76581f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f7657fff640 1 -- 172.21.15.1:0/1771607215 reap_dead start 2024-05-09T05:17:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f765815da70 0x7f76581f5d70 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f764800cad0 tx=0x7f764800cf90 comp rx=0 tx=0).stop 2024-05-09T05:17:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 shutdown_connections 2024-05-09T05:17:09.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 --2- 172.21.15.1:0/1771607215 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f765815da70 0x7f76581f5d70 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 >> 172.21.15.1:0/1771607215 conn(0x7f76580ce180 msgr2=0x7f7658155fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:09.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 shutdown_connections 2024-05-09T05:17:09.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.307+0000 7f765dbc7640 1 -- 172.21.15.1:0/1771607215 wait complete. 2024-05-09T05:17:09.322 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:17:09.322 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:17:09.322 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:17:09.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.448+0000 7fa4ae813640 1 Processor -- start 2024-05-09T05:17:09.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.448+0000 7fa4ae813640 1 -- start start 2024-05-09T05:17:09.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4ae813640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa4a80c8330 0x7fa4a81534a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4ae813640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4a80c5670 0x7fa4a815d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4ae813640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa4a8159fb0 con 0x7fa4a815d7d0 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4ae813640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa4a815cb20 con 0x7fa4a80c5670 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4ae813640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fa4a815ce00 con 0x7fa4a80c8330 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4acdb0640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fa4a815d7d0 legacy=0x7fa4a80cd740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a7fff640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7fa4a80c8330 msgr2=0x7fa4a81534a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4a80c5670 0x7fa4a815d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a7fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa4a80c8330 0x7fa4a81534a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4a80c5670 0x7fa4a815d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41518/0 (socket says 172.21.15.1:41518) 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a77fe640 1 -- 172.21.15.1:0/1165313588 learned_addr learned my addr 172.21.15.1:0/1165313588 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a77fe640 1 -- 172.21.15.1:0/1165313588 >> v2:172.21.15.196:3300/0 conn(0x7fa4a80c8330 msgr2=0x7fa4a81534a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a77fe640 1 --2- 172.21.15.1:0/1165313588 >> v2:172.21.15.196:3300/0 conn(0x7fa4a80c8330 0x7fa4a81534a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a77fe640 1 -- 172.21.15.1:0/1165313588 >> v1:172.21.15.196:6789/0 conn(0x7fa4a815d7d0 legacy=0x7fa4a80cd740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:09.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.450+0000 7fa4a77fe640 1 -- 172.21.15.1:0/1165313588 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a81539e0 con 0x7fa4a80c5670 2024-05-09T05:17:09.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.451+0000 7fa4a77fe640 1 --2- 172.21.15.1:0/1165313588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c5670 0x7fa4a815d290 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8072280 tx=0x7fa49802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6d47a2033ee93a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.452+0000 7fa4a6ffd640 1 -- 172.21.15.1:0/1165313588 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498037070 con 0x7fa4a80c5670 2024-05-09T05:17:09.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.452+0000 7fa4a6ffd640 1 -- 172.21.15.1:0/1165313588 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa498032440 con 0x7fa4a80c5670 2024-05-09T05:17:09.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.452+0000 7fa4a6ffd640 1 -- 172.21.15.1:0/1165313588 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498032810 con 0x7fa4a80c5670 2024-05-09T05:17:09.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.453+0000 7fa4ae813640 1 -- 172.21.15.1:0/1165313588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c5670 msgr2=0x7fa4a815d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.453+0000 7fa4ae813640 1 --2- 172.21.15.1:0/1165313588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c5670 0x7fa4a815d290 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8072280 tx=0x7fa49802a1e0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.453+0000 7fa4ae813640 1 -- 172.21.15.1:0/1165313588 shutdown_connections 2024-05-09T05:17:09.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.453+0000 7fa4ae813640 1 --2- 172.21.15.1:0/1165313588 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c5670 0x7fa4a815d290 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.453+0000 7fa4ae813640 1 --2- 172.21.15.1:0/1165313588 >> v2:172.21.15.196:3300/0 conn(0x7fa4a80c8330 0x7fa4a81534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.453+0000 7fa4ae813640 1 -- 172.21.15.1:0/1165313588 >> 172.21.15.1:0/1165313588 conn(0x7fa4a80cac70 msgr2=0x7fa4a80cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:09.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.453+0000 7fa4ae813640 1 -- 172.21.15.1:0/1165313588 shutdown_connections 2024-05-09T05:17:09.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.453+0000 7fa4ae813640 1 -- 172.21.15.1:0/1165313588 wait complete. 2024-05-09T05:17:09.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.455+0000 7fa4ae813640 1 Processor -- start 2024-05-09T05:17:09.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.455+0000 7fa4ae813640 1 -- start start 2024-05-09T05:17:09.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4ae813640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a80c5670 0x7fa4a81f3130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4ae813640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c8330 0x7fa4a81f68a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4ae813640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4a815d7d0 0x7fa4a81e8390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4ae813640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a815a250 con 0x7fa4a80c8330 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4ae813640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa4a815a510 con 0x7fa4a815d7d0 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4ae813640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a80cc390 con 0x7fa4a80c5670 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c8330 0x7fa4a81f68a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c8330 0x7fa4a81f68a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41528/0 (socket says 172.21.15.1:41528) 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a77fe640 1 -- 172.21.15.1:0/1685582197 learned_addr learned my addr 172.21.15.1:0/1685582197 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4acdb0640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4a815d7d0 msgr2=0x7fa4a81e8390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4acdb0640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4a815d7d0 0x7fa4a81e8390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:09.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a7fff640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a80c5670 0x7fa4a81f3130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a77fe640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a80c5670 msgr2=0x7fa4a81f3130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a77fe640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a80c5670 0x7fa4a81f3130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a77fe640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4a815d7d0 msgr2=0x7fa4a81e8390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a77fe640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4a815d7d0 0x7fa4a81e8390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.457+0000 7fa4a77fe640 1 -- 172.21.15.1:0/1685582197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4a81f3670 con 0x7fa4a80c8330 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.458+0000 7fa4a77fe640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c8330 0x7fa4a81f68a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa49802a220 tx=0x7fa4980046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.458+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498041070 con 0x7fa4a80c8330 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.458+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa4a81f2950 con 0x7fa4a80c8330 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.458+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa49803c870 con 0x7fa4a80c8330 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.458+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa498037040 con 0x7fa4a80c8330 2024-05-09T05:17:09.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.458+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4a81524f0 con 0x7fa4a80c8330 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.459+0000 7fa4ae813640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a814fe00 0x7fa4a81e88d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.459+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a81f5f70 con 0x7fa4a814fe00 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.459+0000 7fa4a7fff640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a814fe00 0x7fa4a81e88d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.460+0000 7fa4a7fff640 1 -- 172.21.15.1:0/1685582197 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa4a8159090 con 0x7fa4a814fe00 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.460+0000 7fa4a7fff640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a814fe00 0x7fa4a81e88d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00d700 tx=0x7fa49c00dbc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.460+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa498044070 con 0x7fa4a80c8330 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.460+0000 7fa4a4ff9640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4940762b0 0x7fa4940787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa4980b80f0 con 0x7fa4a80c8330 2024-05-09T05:17:09.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c016030 con 0x7fa4a814fe00 2024-05-09T05:17:09.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa49c003890 con 0x7fa4a814fe00 2024-05-09T05:17:09.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a814fe00 msgr2=0x7fa4a81e88d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4a4ff9640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a814fe00 0x7fa4a81e88d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00d700 tx=0x7fa49c00dbc0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4acdb0640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4940762b0 0x7fa4940787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a814fe00 msgr2=0x7fa4a81e88d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:09.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4a4ff9640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a814fe00 0x7fa4a81e88d0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fa49c00d700 tx=0x7fa49c00dbc0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.461+0000 7fa4acdb0640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4940762b0 0x7fa4940787c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8159570 tx=0x7fa4a0009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.470+0000 7fa4ae813640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a81527d0 0x7fa4a81e72c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:09.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.470+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa4a8152be0 con 0x7fa4a81527d0 2024-05-09T05:17:09.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.470+0000 7fa4a7fff640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a81527d0 0x7fa4a81e72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:09.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.470+0000 7fa4a7fff640 1 -- 172.21.15.1:0/1685582197 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4a814ab80 con 0x7fa4a81527d0 2024-05-09T05:17:09.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4a7fff640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a81527d0 0x7fa4a81e72c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa49c016000 tx=0x7fa49c005940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:09.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa49c0052e0 con 0x7fa4a81527d0 2024-05-09T05:17:09.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7fa49c0052e0 con 0x7fa4a81527d0 2024-05-09T05:17:09.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a81527d0 msgr2=0x7fa4a81e72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4a4ff9640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a81527d0 0x7fa4a81e72c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa49c016000 tx=0x7fa49c005940 comp rx=0 tx=0).stop 2024-05-09T05:17:09.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4a4ff9640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a81527d0 msgr2=0x7fa4a81e72c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:09.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4a4ff9640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4a81527d0 0x7fa4a81e72c0 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa49c016000 tx=0x7fa49c005940 comp rx=0 tx=0).stop 2024-05-09T05:17:09.470 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:17:09.470 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:09.470 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:09.470 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 86, 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 50765, 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:09.471 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:09.472 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:17:09.473 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:17:09.475 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:09.476 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:09.478 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.479 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.480 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:09.481 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:17:09.482 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:09.483 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:17:09.484 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4940762b0 msgr2=0x7fa4940787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4ae813640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4940762b0 0x7fa4940787c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fa4a8159570 tx=0x7fa4a0009270 comp rx=0 tx=0).stop 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c8330 msgr2=0x7fa4a81f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4ae813640 1 --2- 172.21.15.1:0/1685582197 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4a80c8330 0x7fa4a81f68a0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fa49802a220 tx=0x7fa4980046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4acdb0640 1 -- 172.21.15.1:0/1685582197 reap_dead start 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.471+0000 7fa4acdb0640 1 -- 172.21.15.1:0/1685582197 reap_dead start 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.472+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 shutdown_connections 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.472+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 >> 172.21.15.1:0/1685582197 conn(0x7fa4a80cac70 msgr2=0x7fa4a806c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.472+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 shutdown_connections 2024-05-09T05:17:09.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:09.472+0000 7fa4ae813640 1 -- 172.21.15.1:0/1685582197 wait complete. 2024-05-09T05:17:09.486 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:17:09.486 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:17:29.486 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:17:29.487 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-05-09T05:17:29.487 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T05:17:29.487 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T05:17:29.491 INFO:tasks.ceph.mon.b:Started 2024-05-09T05:17:29.491 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:17:29.491 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:17:29.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.758+0000 7f78f85c7640 1 Processor -- start 2024-05-09T05:17:29.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.758+0000 7f78f85c7640 1 -- start start 2024-05-09T05:17:29.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f85c7640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f78f00d16b0 0x7f78f00d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:29.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f85c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78f015ca90 0x7f78f015cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f85c7640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78f006f6d0 con 0x7f78f006f1b0 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f85c7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f78f00cee50 con 0x7f78f015ca90 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f85c7640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f78f00cd0e0 con 0x7f78f00d16b0 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f6b64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78f015ca90 0x7f78f015cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f6b64640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f78f015ca90 0x7f78f015cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43898/0 (socket says 172.21.15.1:43898) 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f6b64640 1 -- 172.21.15.1:0/2783777420 learned_addr learned my addr 172.21.15.1:0/2783777420 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f6363640 1 --2- 172.21.15.1:0/2783777420 >> v2:172.21.15.196:3300/0 conn(0x7f78f00d16b0 0x7f78f00d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f6b64640 1 -- 172.21.15.1:0/2783777420 >> v2:172.21.15.196:3300/0 conn(0x7f78f00d16b0 msgr2=0x7f78f00d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f6b64640 1 --2- 172.21.15.1:0/2783777420 >> v2:172.21.15.196:3300/0 conn(0x7f78f00d16b0 0x7f78f00d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f6b64640 1 -- 172.21.15.1:0/2783777420 >> v1:172.21.15.196:6789/0 conn(0x7f78f006f1b0 legacy=0x7f78f00ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:29.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.760+0000 7f78f6b64640 1 -- 172.21.15.1:0/2783777420 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f00cc4d0 con 0x7f78f015ca90 2024-05-09T05:17:29.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.761+0000 7f78f6b64640 1 --2- 172.21.15.1:0/2783777420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f015cea0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f78f006d120 tx=0x7f78e802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2518253a14081a1d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:29.760 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.761+0000 7f78f5361640 1 -- 172.21.15.1:0/2783777420 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8036070 con 0x7f78f015ca90 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.761+0000 7f78f5361640 1 -- 172.21.15.1:0/2783777420 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e8031600 con 0x7f78f015ca90 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.761+0000 7f78f5361640 1 -- 172.21.15.1:0/2783777420 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e80319f0 con 0x7f78f015ca90 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.762+0000 7f78f85c7640 1 -- 172.21.15.1:0/2783777420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 msgr2=0x7f78f015cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.762+0000 7f78f85c7640 1 --2- 172.21.15.1:0/2783777420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f015cea0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f78f006d120 tx=0x7f78e802a7c0 comp rx=0 tx=0).stop 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.762+0000 7f78f85c7640 1 -- 172.21.15.1:0/2783777420 shutdown_connections 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.762+0000 7f78f85c7640 1 --2- 172.21.15.1:0/2783777420 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f015cea0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.762+0000 7f78f85c7640 1 --2- 172.21.15.1:0/2783777420 >> v2:172.21.15.196:3300/0 conn(0x7f78f00d16b0 0x7f78f00d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.762+0000 7f78f85c7640 1 -- 172.21.15.1:0/2783777420 >> 172.21.15.1:0/2783777420 conn(0x7f78f00b3660 msgr2=0x7f78f00b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.762+0000 7f78f85c7640 1 -- 172.21.15.1:0/2783777420 shutdown_connections 2024-05-09T05:17:29.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.762+0000 7f78f85c7640 1 -- 172.21.15.1:0/2783777420 wait complete. 2024-05-09T05:17:29.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.764+0000 7f78f85c7640 1 Processor -- start 2024-05-09T05:17:29.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.764+0000 7f78f85c7640 1 -- start start 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.765+0000 7f78f85c7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 0x7f78f01f3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.765+0000 7f78f85c7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78f00d16b0 0x7f78f01e4040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.765+0000 7f78f85c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f01e4580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.765+0000 7f78f85c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f78f00ccd70 con 0x7f78f015ca90 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.765+0000 7f78f85c7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f78f00cca90 con 0x7f78f006f1b0 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.765+0000 7f78f85c7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f78f00d0910 con 0x7f78f00d16b0 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6363640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 0x7f78f01f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6b64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f01e4580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:29.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f5b62640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78f00d16b0 0x7f78f01e4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6363640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 0x7f78f01f3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:32902/0 (socket says 172.21.15.1:32902) 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6363640 1 -- 172.21.15.1:0/196585738 learned_addr learned my addr 172.21.15.1:0/196585738 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6b64640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f01e4580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43902/0 (socket says 172.21.15.1:43902) 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6363640 1 -- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 msgr2=0x7f78f01f3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6363640 1 -- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 msgr2=0x7f78f01f3110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6363640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 0x7f78f01f3110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6363640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 0x7f78f01f3110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6b64640 1 -- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78f00d16b0 msgr2=0x7f78f01e4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6b64640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78f00d16b0 0x7f78f01e4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6b64640 1 -- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 msgr2=0x7f78f01f3110 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6b64640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 0x7f78f01f3110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6b64640 1 -- 172.21.15.1:0/196585738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78f01f3650 con 0x7f78f015ca90 2024-05-09T05:17:29.766 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f5b62640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78f00d16b0 0x7f78f01e4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:17:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78f6b64640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f01e4580 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f78e8030000 tx=0x7f78e8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.766+0000 7f78e6ffd640 1 -- 172.21.15.1:0/196585738 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8036050 con 0x7f78f015ca90 2024-05-09T05:17:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.767+0000 7f78e6ffd640 1 -- 172.21.15.1:0/196585738 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78e8042070 con 0x7f78f015ca90 2024-05-09T05:17:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.767+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78f00c9fb0 con 0x7f78f015ca90 2024-05-09T05:17:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.767+0000 7f78e6ffd640 1 -- 172.21.15.1:0/196585738 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78e8009510 con 0x7f78f015ca90 2024-05-09T05:17:29.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.767+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78f01e4dc0 con 0x7f78f015ca90 2024-05-09T05:17:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.768+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78f006ec30 con 0x7f78f015ca90 2024-05-09T05:17:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.769+0000 7f78e6ffd640 1 -- 172.21.15.1:0/196585738 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f78e8004490 con 0x7f78f015ca90 2024-05-09T05:17:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.769+0000 7f78e6ffd640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78d40762d0 0x7f78d40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.769+0000 7f78e6ffd640 1 -- 172.21.15.1:0/196585738 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f78e80b8410 con 0x7f78f015ca90 2024-05-09T05:17:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.770+0000 7f78f6363640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78d40762d0 0x7f78d40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:29.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.770+0000 7f78f6363640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78d40762d0 0x7f78d40787e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f78ec006fd0 tx=0x7f78ec008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:29.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:29.774+0000 7f78e6ffd640 1 -- 172.21.15.1:0/196585738 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f78f006ec30 con 0x7f78f015ca90 2024-05-09T05:17:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.068+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f78f015a930 con 0x7f78f015ca90 2024-05-09T05:17:30.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.068+0000 7f78e6ffd640 1 -- 172.21.15.1:0/196585738 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f78e8081cf0 con 0x7f78f015ca90 2024-05-09T05:17:30.066 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":86,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78d40762d0 msgr2=0x7f78d40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78d40762d0 0x7f78d40787e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f78ec006fd0 tx=0x7f78ec008040 comp rx=0 tx=0).stop 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 msgr2=0x7f78f01e4580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f01e4580 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f78e8030000 tx=0x7f78e8004a40 comp rx=0 tx=0).stop 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 shutdown_connections 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78d40762d0 0x7f78d40787e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78f015ca90 0x7f78f01e4580 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78f00d16b0 0x7f78f01e4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:30.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 --2- 172.21.15.1:0/196585738 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78f006f1b0 0x7f78f01f3110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:30.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 >> 172.21.15.1:0/196585738 conn(0x7f78f00b3660 msgr2=0x7f78f00ce740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:30.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 shutdown_connections 2024-05-09T05:17:30.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:30.071+0000 7f78f85c7640 1 -- 172.21.15.1:0/196585738 wait complete. 2024-05-09T05:17:33.081 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:17:33.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.234+0000 7f823f3c7640 1 Processor -- start 2024-05-09T05:17:33.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.235+0000 7f823f3c7640 1 -- start start 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823f3c7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82380c5ca0 0x7f82380ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823f3c7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f823806df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823f3c7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f82380c86a0 con 0x7f8238156e90 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823f3c7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f82380cb910 con 0x7f82380c5ca0 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823f3c7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f82380c8980 con 0x7f8238156a80 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823d163640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82380c5ca0 0x7f82380ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823d163640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f82380c5ca0 0x7f82380ca4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43908/0 (socket says 172.21.15.1:43908) 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823d163640 1 -- 172.21.15.1:0/1453540365 learned_addr learned my addr 172.21.15.1:0/1453540365 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823c962640 1 --2- 172.21.15.1:0/1453540365 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f823806df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823d163640 1 -- 172.21.15.1:0/1453540365 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 msgr2=0x7f823806df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:33.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.236+0000 7f823d163640 1 --2- 172.21.15.1:0/1453540365 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f823806df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.237+0000 7f823d163640 1 -- 172.21.15.1:0/1453540365 >> v1:172.21.15.1:6789/0 conn(0x7f8238156e90 legacy=0x7f82380c7750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:33.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.237+0000 7f823d163640 1 -- 172.21.15.1:0/1453540365 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82380c8c60 con 0x7f82380c5ca0 2024-05-09T05:17:33.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.237+0000 7f823c962640 1 --2- 172.21.15.1:0/1453540365 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f823806df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:17:33.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.237+0000 7f823d163640 1 --2- 172.21.15.1:0/1453540365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82380c5ca0 0x7f82380ca4a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8234009870 tx=0x7f823402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a1bb9f37c2d2062 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:33.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f822ffff640 1 -- 172.21.15.1:0/1453540365 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8234037070 con 0x7f82380c5ca0 2024-05-09T05:17:33.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f822ffff640 1 -- 172.21.15.1:0/1453540365 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f823402ae50 con 0x7f82380c5ca0 2024-05-09T05:17:33.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f822ffff640 1 -- 172.21.15.1:0/1453540365 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8234032860 con 0x7f82380c5ca0 2024-05-09T05:17:33.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f823f3c7640 1 -- 172.21.15.1:0/1453540365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82380c5ca0 msgr2=0x7f82380ca4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:33.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f823f3c7640 1 --2- 172.21.15.1:0/1453540365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82380c5ca0 0x7f82380ca4a0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8234009870 tx=0x7f823402a300 comp rx=0 tx=0).stop 2024-05-09T05:17:33.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f823f3c7640 1 -- 172.21.15.1:0/1453540365 shutdown_connections 2024-05-09T05:17:33.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f823f3c7640 1 --2- 172.21.15.1:0/1453540365 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f823806df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f823f3c7640 1 --2- 172.21.15.1:0/1453540365 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f82380c5ca0 0x7f82380ca4a0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f823f3c7640 1 -- 172.21.15.1:0/1453540365 >> 172.21.15.1:0/1453540365 conn(0x7f82380745a0 msgr2=0x7f8238071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:33.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f823f3c7640 1 -- 172.21.15.1:0/1453540365 shutdown_connections 2024-05-09T05:17:33.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.238+0000 7f823f3c7640 1 -- 172.21.15.1:0/1453540365 wait complete. 2024-05-09T05:17:33.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.241+0000 7f823f3c7640 1 Processor -- start 2024-05-09T05:17:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.241+0000 7f823f3c7640 1 -- start start 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.242+0000 7f823f3c7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 0x7f82381f6860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.242+0000 7f823f3c7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f8238150e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.242+0000 7f823f3c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238156e90 0x7f823814b190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.242+0000 7f823f3c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82380cad60 con 0x7f8238156e90 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823f3c7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f82380cb5a0 con 0x7f82380c5ca0 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823f3c7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f82380ca9e0 con 0x7f8238156a80 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238156e90 0x7f823814b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238156e90 0x7f823814b190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43914/0 (socket says 172.21.15.1:43914) 2024-05-09T05:17:33.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 -- 172.21.15.1:0/2426279417 learned_addr learned my addr 172.21.15.1:0/2426279417 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823c962640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f8238150e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d163640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 0x7f82381f6860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d163640 1 -- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 msgr2=0x7f82381f6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d163640 1 -- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 msgr2=0x7f82381f6860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d163640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 0x7f82381f6860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d163640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 0x7f82381f6860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 -- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 msgr2=0x7f8238150e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f8238150e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 -- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 msgr2=0x7f82381f6860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 0x7f82381f6860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 -- 172.21.15.1:0/2426279417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82381599c0 con 0x7f8238156e90 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f823d964640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238156e90 0x7f823814b190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f823000c700 tx=0x7f823000cbc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.243+0000 7f822dffb640 1 -- 172.21.15.1:0/2426279417 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8230003b70 con 0x7f8238156e90 2024-05-09T05:17:33.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.244+0000 7f822dffb640 1 -- 172.21.15.1:0/2426279417 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8230003d10 con 0x7f8238156e90 2024-05-09T05:17:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.244+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82381f2b30 con 0x7f8238156e90 2024-05-09T05:17:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.244+0000 7f822dffb640 1 -- 172.21.15.1:0/2426279417 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f823000aa50 con 0x7f8238156e90 2024-05-09T05:17:33.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.244+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f82381f30f0 con 0x7f8238156e90 2024-05-09T05:17:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.245+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f823806dc90 con 0x7f8238156e90 2024-05-09T05:17:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.246+0000 7f822dffb640 1 -- 172.21.15.1:0/2426279417 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f823000abf0 con 0x7f8238156e90 2024-05-09T05:17:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.246+0000 7f822dffb640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f821c076300 0x7f821c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.246+0000 7f822dffb640 1 -- 172.21.15.1:0/2426279417 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f82300974e0 con 0x7f8238156e90 2024-05-09T05:17:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.246+0000 7f823d163640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f821c076300 0x7f821c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:33.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.247+0000 7f823d163640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f821c076300 0x7f821c078810 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8234034000 tx=0x7f823402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:33.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.252+0000 7f822dffb640 1 -- 172.21.15.1:0/2426279417 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f823806dc90 con 0x7f8238156e90 2024-05-09T05:17:33.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.553+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8238050870 con 0x7f8238156e90 2024-05-09T05:17:33.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.554+0000 7f822dffb640 1 -- 172.21.15.1:0/2426279417 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f8230060d30 con 0x7f8238156e90 2024-05-09T05:17:33.552 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":86,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:17:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f821c076300 msgr2=0x7f821c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f821c076300 0x7f821c078810 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8234034000 tx=0x7f823402e000 comp rx=0 tx=0).stop 2024-05-09T05:17:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238156e90 msgr2=0x7f823814b190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:33.554 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238156e90 0x7f823814b190 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f823000c700 tx=0x7f823000cbc0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 shutdown_connections 2024-05-09T05:17:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f821c076300 0x7f821c078810 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8238156e90 0x7f823814b190 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8238156a80 0x7f8238150e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 --2- 172.21.15.1:0/2426279417 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82380c5ca0 0x7f82381f6860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 >> 172.21.15.1:0/2426279417 conn(0x7f82380745a0 msgr2=0x7f8238154140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 shutdown_connections 2024-05-09T05:17:33.555 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:33.556+0000 7f823f3c7640 1 -- 172.21.15.1:0/2426279417 wait complete. 2024-05-09T05:17:34.431 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:17:34.434 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:17:36.569 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:17:36.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.715+0000 7f85eab96640 1 Processor -- start 2024-05-09T05:17:36.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.715+0000 7f85eab96640 1 -- start start 2024-05-09T05:17:36.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.716+0000 7f85eab96640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85e40cd7f0 0x7f85e40c7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:36.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.716+0000 7f85eab96640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 0x7f85e40c7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:36.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.716+0000 7f85eab96640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f85e40cd0a0 con 0x7f85e406dd70 2024-05-09T05:17:36.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.716+0000 7f85eab96640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f85e40cce00 con 0x7f85e40cd7f0 2024-05-09T05:17:36.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.716+0000 7f85eab96640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f85e40c56a0 con 0x7f85e40cd2b0 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.716+0000 7f85e3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85e40cd7f0 0x7f85e40c7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e3fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f85e40cd7f0 0x7f85e40c7040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43918/0 (socket says 172.21.15.1:43918) 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e3fff640 1 -- 172.21.15.1:0/981500288 learned_addr learned my addr 172.21.15.1:0/981500288 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e9133640 1 --2- 172.21.15.1:0/981500288 >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 0x7f85e40c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e9133640 1 -- 172.21.15.1:0/981500288 >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 msgr2=0x7f85e40c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e9133640 1 -- 172.21.15.1:0/981500288 >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 msgr2=0x7f85e40c7580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e9133640 1 --2- 172.21.15.1:0/981500288 >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 0x7f85e40c7580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e9133640 1 --2- 172.21.15.1:0/981500288 >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 0x7f85e40c7580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e3fff640 1 -- 172.21.15.1:0/981500288 >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 msgr2=0x7f85e40c7580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e3fff640 1 --2- 172.21.15.1:0/981500288 >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 0x7f85e40c7580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e3fff640 1 -- 172.21.15.1:0/981500288 >> v1:172.21.15.196:6789/0 conn(0x7f85e406dd70 legacy=0x7f85e40cbff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:36.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.717+0000 7f85e3fff640 1 -- 172.21.15.1:0/981500288 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e415d320 con 0x7f85e40cd7f0 2024-05-09T05:17:36.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.718+0000 7f85e3fff640 1 --2- 172.21.15.1:0/981500288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd7f0 0x7f85e40c7040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85e40cab20 tx=0x7f85d002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c0f4f54f207d931 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:36.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.719+0000 7f85e37fe640 1 -- 172.21.15.1:0/981500288 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d0037070 con 0x7f85e40cd7f0 2024-05-09T05:17:36.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.719+0000 7f85e37fe640 1 -- 172.21.15.1:0/981500288 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d0032440 con 0x7f85e40cd7f0 2024-05-09T05:17:36.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.719+0000 7f85e37fe640 1 -- 172.21.15.1:0/981500288 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d0032810 con 0x7f85e40cd7f0 2024-05-09T05:17:36.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.720+0000 7f85eab96640 1 -- 172.21.15.1:0/981500288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd7f0 msgr2=0x7f85e40c7040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:36.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.720+0000 7f85eab96640 1 --2- 172.21.15.1:0/981500288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd7f0 0x7f85e40c7040 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f85e40cab20 tx=0x7f85d002a270 comp rx=0 tx=0).stop 2024-05-09T05:17:36.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.720+0000 7f85eab96640 1 -- 172.21.15.1:0/981500288 shutdown_connections 2024-05-09T05:17:36.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.720+0000 7f85eab96640 1 --2- 172.21.15.1:0/981500288 >> v2:172.21.15.196:3300/0 conn(0x7f85e40cd2b0 0x7f85e40c7580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:36.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.720+0000 7f85eab96640 1 --2- 172.21.15.1:0/981500288 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd7f0 0x7f85e40c7040 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.720+0000 7f85eab96640 1 -- 172.21.15.1:0/981500288 >> 172.21.15.1:0/981500288 conn(0x7f85e40ce110 msgr2=0x7f85e40ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.720+0000 7f85eab96640 1 -- 172.21.15.1:0/981500288 shutdown_connections 2024-05-09T05:17:36.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.720+0000 7f85eab96640 1 -- 172.21.15.1:0/981500288 wait complete. 2024-05-09T05:17:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.722+0000 7f85eab96640 1 Processor -- start 2024-05-09T05:17:36.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.722+0000 7f85eab96640 1 -- start start 2024-05-09T05:17:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.723+0000 7f85eab96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd2b0 0x7f85e4152320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.723+0000 7f85eab96640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 0x7f85e4150f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.723+0000 7f85eab96640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f85e40c8df0 0x7f85e4154e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.723+0000 7f85eab96640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f85e4074ba0 con 0x7f85e40cd2b0 2024-05-09T05:17:36.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.723+0000 7f85eab96640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f85e4074a20 con 0x7f85e40cd7f0 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.723+0000 7f85eab96640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f85e4156c60 con 0x7f85e40c8df0 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd2b0 0x7f85e4152320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd2b0 0x7f85e4152320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43922/0 (socket says 172.21.15.1:43922) 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 -- 172.21.15.1:0/31520925 learned_addr learned my addr 172.21.15.1:0/31520925 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e3fff640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 0x7f85e4150f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e9133640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f85e40c8df0 0x7f85e4154e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e3fff640 1 -- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 msgr2=0x7f85e4150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e3fff640 1 -- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 msgr2=0x7f85e4150f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e3fff640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 0x7f85e4150f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e3fff640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 0x7f85e4150f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 -- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f85e40c8df0 msgr2=0x7f85e4154e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f85e40c8df0 0x7f85e4154e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 -- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 msgr2=0x7f85e4150f90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:36.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 0x7f85e4150f90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 -- 172.21.15.1:0/31520925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e4152860 con 0x7f85e40cd2b0 2024-05-09T05:17:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.724+0000 7f85e8932640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd2b0 0x7f85e4152320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f85d800c400 tx=0x7f85d800c8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.725+0000 7f85e17fa640 1 -- 172.21.15.1:0/31520925 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8017070 con 0x7f85e40cd2b0 2024-05-09T05:17:36.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.725+0000 7f85e17fa640 1 -- 172.21.15.1:0/31520925 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f85d80052d0 con 0x7f85e40cd2b0 2024-05-09T05:17:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.725+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e41514d0 con 0x7f85e40cd2b0 2024-05-09T05:17:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.725+0000 7f85e17fa640 1 -- 172.21.15.1:0/31520925 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85d8003830 con 0x7f85e40cd2b0 2024-05-09T05:17:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.725+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e41542e0 con 0x7f85e40cd2b0 2024-05-09T05:17:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.726+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e40ca390 con 0x7f85e40cd2b0 2024-05-09T05:17:36.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.727+0000 7f85e17fa640 1 -- 172.21.15.1:0/31520925 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f85d801a070 con 0x7f85e40cd2b0 2024-05-09T05:17:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.728+0000 7f85e17fa640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85c8076300 0x7f85c8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.728+0000 7f85e17fa640 1 -- 172.21.15.1:0/31520925 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f85d80969b0 con 0x7f85e40cd2b0 2024-05-09T05:17:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.728+0000 7f85e3fff640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85c8076300 0x7f85c8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:36.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.729+0000 7f85e3fff640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85c8076300 0x7f85c8078810 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f85d0034000 tx=0x7f85d002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:36.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:36.733+0000 7f85e17fa640 1 -- 172.21.15.1:0/31520925 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f85d805fbb0 con 0x7f85e40cd2b0 2024-05-09T05:17:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.117+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f85e4050870 con 0x7f85e40cd2b0 2024-05-09T05:17:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.117+0000 7f85e17fa640 1 -- 172.21.15.1:0/31520925 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7f85d8068280 con 0x7f85e40cd2b0 2024-05-09T05:17:37.116 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:17:37.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85c8076300 msgr2=0x7f85c8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85c8076300 0x7f85c8078810 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f85d0034000 tx=0x7f85d002e000 comp rx=0 tx=0).stop 2024-05-09T05:17:37.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd2b0 msgr2=0x7f85e4152320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd2b0 0x7f85e4152320 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f85d800c400 tx=0x7f85d800c8c0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 shutdown_connections 2024-05-09T05:17:37.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f85c8076300 0x7f85c8078810 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f85e40cd7f0 0x7f85e4150f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f85e40c8df0 0x7f85e4154e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 --2- 172.21.15.1:0/31520925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f85e40cd2b0 0x7f85e4152320 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 >> 172.21.15.1:0/31520925 conn(0x7f85e40ce110 msgr2=0x7f85e4073d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 shutdown_connections 2024-05-09T05:17:37.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.120+0000 7f85eab96640 1 -- 172.21.15.1:0/31520925 wait complete. 2024-05-09T05:17:37.131 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:17:37.131 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:17:37.131 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:17:37.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.273+0000 7f20f1a2b640 1 Processor -- start 2024-05-09T05:17:37.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.273+0000 7f20f1a2b640 1 -- start start 2024-05-09T05:17:37.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20f1a2b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20ec0736b0 0x7f20ec15d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20f1a2b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec0cc1b0 con 0x7f20ec0c5600 2024-05-09T05:17:37.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20f1a2b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20ec0cc490 con 0x7f20ec06dde0 2024-05-09T05:17:37.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20f1a2b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f20ec0cbf60 con 0x7f20ec0736b0 2024-05-09T05:17:37.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20ea7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20ec0736b0 0x7f20ec15d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20ea7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20ec0736b0 0x7f20ec15d330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43932/0 (socket says 172.21.15.1:43932) 2024-05-09T05:17:37.273 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20ea7fc640 1 -- 172.21.15.1:0/2362832450 learned_addr learned my addr 172.21.15.1:0/2362832450 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:37.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20ea7fc640 1 -- 172.21.15.1:0/2362832450 >> v1:172.21.15.196:6789/0 conn(0x7f20ec06dde0 legacy=0x7f20ec0c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20ea7fc640 1 -- 172.21.15.1:0/2362832450 >> v1:172.21.15.1:6789/0 conn(0x7f20ec0c5600 legacy=0x7f20ec167230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.275+0000 7f20ea7fc640 1 -- 172.21.15.1:0/2362832450 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ec06c6f0 con 0x7f20ec0736b0 2024-05-09T05:17:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.276+0000 7f20ea7fc640 1 --2- 172.21.15.1:0/2362832450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 0x7f20ec15d330 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009870 tx=0x7f20e002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81000b74cfe23d0a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.276+0000 7f20e9ffb640 1 -- 172.21.15.1:0/2362832450 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0037070 con 0x7f20ec0736b0 2024-05-09T05:17:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.276+0000 7f20e9ffb640 1 -- 172.21.15.1:0/2362832450 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e002ae50 con 0x7f20ec0736b0 2024-05-09T05:17:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.276+0000 7f20e9ffb640 1 -- 172.21.15.1:0/2362832450 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0032860 con 0x7f20ec0736b0 2024-05-09T05:17:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.277+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2362832450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 msgr2=0x7f20ec15d330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.277+0000 7f20f1a2b640 1 --2- 172.21.15.1:0/2362832450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 0x7f20ec15d330 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f20e0009870 tx=0x7f20e002a300 comp rx=0 tx=0).stop 2024-05-09T05:17:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.277+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2362832450 shutdown_connections 2024-05-09T05:17:37.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.277+0000 7f20f1a2b640 1 --2- 172.21.15.1:0/2362832450 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 0x7f20ec15d330 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.277+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2362832450 >> 172.21.15.1:0/2362832450 conn(0x7f20ec0ce180 msgr2=0x7f20ec0ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:37.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.277+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2362832450 shutdown_connections 2024-05-09T05:17:37.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.277+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2362832450 wait complete. 2024-05-09T05:17:37.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.279+0000 7f20f1a2b640 1 Processor -- start 2024-05-09T05:17:37.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.279+0000 7f20f1a2b640 1 -- start start 2024-05-09T05:17:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20f1a2b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20ec06dde0 0x7f20ec150dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20f1a2b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 0x7f20ec14a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20f1a2b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20ec0c5600 0x7f20ec152bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.280 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20f1a2b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec0c77c0 con 0x7f20ec0736b0 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20f1a2b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec06ccb0 con 0x7f20ec06dde0 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20f1a2b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f20ec0c7940 con 0x7f20ec0c5600 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20ea7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 0x7f20ec14a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20ea7fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 0x7f20ec14a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43936/0 (socket says 172.21.15.1:43936) 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20ea7fc640 1 -- 172.21.15.1:0/2976181454 learned_addr learned my addr 172.21.15.1:0/2976181454 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20eaffd640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20ec06dde0 0x7f20ec150dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.281+0000 7f20eb7fe640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20ec0c5600 0x7f20ec152bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20ea7fc640 1 -- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20ec0c5600 msgr2=0x7f20ec152bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20ea7fc640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20ec0c5600 0x7f20ec152bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20ea7fc640 1 -- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20ec06dde0 msgr2=0x7f20ec150dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20ea7fc640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20ec06dde0 0x7f20ec150dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20ea7fc640 1 -- 172.21.15.1:0/2976181454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20ec151300 con 0x7f20ec0736b0 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20ea7fc640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 0x7f20ec14a070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000c00 tx=0x7f20e00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0041070 con 0x7f20ec0736b0 2024-05-09T05:17:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20e003c890 con 0x7f20ec0736b0 2024-05-09T05:17:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20e0037040 con 0x7f20ec0736b0 2024-05-09T05:17:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20ec160ca0 con 0x7f20ec0736b0 2024-05-09T05:17:37.282 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.282+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20ec1606e0 con 0x7f20ec0736b0 2024-05-09T05:17:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.283+0000 7f20f1a2b640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec15ee80 0x7f20ec153130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.283+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec14fb40 con 0x7f20ec15ee80 2024-05-09T05:17:37.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.283+0000 7f20eaffd640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec15ee80 0x7f20ec153130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.284+0000 7f20eaffd640 1 -- 172.21.15.1:0/2976181454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20ec050870 con 0x7f20ec15ee80 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.284+0000 7f20eaffd640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec15ee80 0x7f20ec153130 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00f520 tx=0x7f20dc00f9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.284+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc0052e0 con 0x7f20ec15ee80 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.284+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f20e0044070 con 0x7f20ec0736b0 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.285+0000 7f20d3fff640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20d8076200 0x7f20d8078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.285+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f20e00b7f10 con 0x7f20ec0736b0 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.285+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f20dc018070 con 0x7f20ec15ee80 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.285+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec15ee80 msgr2=0x7f20ec153130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.285+0000 7f20d3fff640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec15ee80 0x7f20ec153130 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f20dc00f520 tx=0x7f20dc00f9e0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.285+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec15ee80 msgr2=0x7f20ec153130 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.285+0000 7f20d3fff640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec15ee80 0x7f20ec153130 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.285+0000 7f20eb7fe640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20d8076200 0x7f20d8078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.286+0000 7f20eb7fe640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20d8076200 0x7f20d8078710 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f20ec1501a0 tx=0x7f20e4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.294+0000 7f20f1a2b640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec14d180 0x7f20ec1f6420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.294+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f20ec0748f0 con 0x7f20ec14d180 2024-05-09T05:17:37.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.294+0000 7f20eaffd640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec14d180 0x7f20ec1f6420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.294+0000 7f20eaffd640 1 -- 172.21.15.1:0/2976181454 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20ec0c9e20 con 0x7f20ec14d180 2024-05-09T05:17:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.294+0000 7f20eaffd640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec14d180 0x7f20ec1f6420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f20dc017000 tx=0x7f20dc001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20dc005480 con 0x7f20ec14d180 2024-05-09T05:17:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7f20dc003dd0 con 0x7f20ec14d180 2024-05-09T05:17:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec14d180 msgr2=0x7f20ec1f6420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20d3fff640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec14d180 0x7f20ec1f6420 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f20dc017000 tx=0x7f20dc001080 comp rx=0 tx=0).stop 2024-05-09T05:17:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20d3fff640 1 -- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec14d180 msgr2=0x7f20ec1f6420 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:37.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20d3fff640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec14d180 0x7f20ec1f6420 secure :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f20dc017000 tx=0x7f20dc001080 comp rx=0 tx=0).stop 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 88, 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 259490, 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:17:37.294 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.295 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.296 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:37.297 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.298 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:37.299 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.300 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:17:37.301 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.302 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.303 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:17:37.304 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.305 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.306 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.307 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.307 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:17:37.307 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.307 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.307 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.307 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:17:37.307 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:17:37.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20d8076200 msgr2=0x7f20d8078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20f1a2b640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20d8076200 0x7f20d8078710 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f20ec1501a0 tx=0x7f20e4009040 comp rx=0 tx=0).stop 2024-05-09T05:17:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 msgr2=0x7f20ec14a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20f1a2b640 1 --2- 172.21.15.1:0/2976181454 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20ec0736b0 0x7f20ec14a070 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f20e0000c00 tx=0x7f20e00046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20eb7fe640 1 -- 172.21.15.1:0/2976181454 reap_dead start 2024-05-09T05:17:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20eb7fe640 1 -- 172.21.15.1:0/2976181454 reap_dead start 2024-05-09T05:17:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 shutdown_connections 2024-05-09T05:17:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 >> 172.21.15.1:0/2976181454 conn(0x7f20ec0ce180 msgr2=0x7f20ec154220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.295+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 shutdown_connections 2024-05-09T05:17:37.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.296+0000 7f20f1a2b640 1 -- 172.21.15.1:0/2976181454 wait complete. 2024-05-09T05:17:37.309 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:17:37.309 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:17:37.310 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:17:37.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.446+0000 7f26d6a70640 1 Processor -- start 2024-05-09T05:17:37.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.446+0000 7f26d6a70640 1 -- start start 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.447+0000 7f26d6a70640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f26d015d130 0x7f26d01569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.447+0000 7f26d6a70640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26d006c6f0 0x7f26d00716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.447+0000 7f26d6a70640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d00c3ce0 con 0x7f26d0071be0 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.447+0000 7f26d6a70640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f26d015d540 con 0x7f26d006c6f0 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.447+0000 7f26d6a70640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f26d00c3fc0 con 0x7f26d015d130 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.447+0000 7f26c7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26d006c6f0 0x7f26d00716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.447+0000 7f26c7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26d006c6f0 0x7f26d00716a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43952/0 (socket says 172.21.15.1:43952) 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.447+0000 7f26c7fff640 1 -- 172.21.15.1:0/2262247731 learned_addr learned my addr 172.21.15.1:0/2262247731 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:37.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.448+0000 7f26d480c640 1 --2- 172.21.15.1:0/2262247731 >> v2:172.21.15.196:3300/0 conn(0x7f26d015d130 0x7f26d01569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.448+0000 7f26c7fff640 1 -- 172.21.15.1:0/2262247731 >> v2:172.21.15.196:3300/0 conn(0x7f26d015d130 msgr2=0x7f26d01569c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:37.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.448+0000 7f26c7fff640 1 --2- 172.21.15.1:0/2262247731 >> v2:172.21.15.196:3300/0 conn(0x7f26d015d130 0x7f26d01569c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.448+0000 7f26c7fff640 1 -- 172.21.15.1:0/2262247731 >> v1:172.21.15.1:6789/0 conn(0x7f26d0071be0 legacy=0x7f26d00c48c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.448+0000 7f26c7fff640 1 -- 172.21.15.1:0/2262247731 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d006e4a0 con 0x7f26d006c6f0 2024-05-09T05:17:37.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.448+0000 7f26d480c640 1 --2- 172.21.15.1:0/2262247731 >> v2:172.21.15.196:3300/0 conn(0x7f26d015d130 0x7f26d01569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:17:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.449+0000 7f26c7fff640 1 --2- 172.21.15.1:0/2262247731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 0x7f26d00716a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f26d00cf650 tx=0x7f26c002a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9160d195913e7661 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.449+0000 7f26c77fe640 1 -- 172.21.15.1:0/2262247731 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c0037070 con 0x7f26d006c6f0 2024-05-09T05:17:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.449+0000 7f26c77fe640 1 -- 172.21.15.1:0/2262247731 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26c0032440 con 0x7f26d006c6f0 2024-05-09T05:17:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.449+0000 7f26c77fe640 1 -- 172.21.15.1:0/2262247731 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c0032810 con 0x7f26d006c6f0 2024-05-09T05:17:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.450+0000 7f26d6a70640 1 -- 172.21.15.1:0/2262247731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 msgr2=0x7f26d00716a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.450+0000 7f26d6a70640 1 --2- 172.21.15.1:0/2262247731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 0x7f26d00716a0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f26d00cf650 tx=0x7f26c002a270 comp rx=0 tx=0).stop 2024-05-09T05:17:37.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.450+0000 7f26d6a70640 1 -- 172.21.15.1:0/2262247731 shutdown_connections 2024-05-09T05:17:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.450+0000 7f26d6a70640 1 --2- 172.21.15.1:0/2262247731 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 0x7f26d00716a0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.450+0000 7f26d6a70640 1 --2- 172.21.15.1:0/2262247731 >> v2:172.21.15.196:3300/0 conn(0x7f26d015d130 0x7f26d01569c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.450+0000 7f26d6a70640 1 -- 172.21.15.1:0/2262247731 >> 172.21.15.1:0/2262247731 conn(0x7f26d00747b0 msgr2=0x7f26d00ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.450+0000 7f26d6a70640 1 -- 172.21.15.1:0/2262247731 shutdown_connections 2024-05-09T05:17:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.450+0000 7f26d6a70640 1 -- 172.21.15.1:0/2262247731 wait complete. 2024-05-09T05:17:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.452+0000 7f26d6a70640 1 Processor -- start 2024-05-09T05:17:37.450 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.452+0000 7f26d6a70640 1 -- start start 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.453+0000 7f26d6a70640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 0x7f26d014e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.453+0000 7f26d6a70640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0071be0 0x7f26d014de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.453+0000 7f26d6a70640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d015d130 0x7f26d0155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.453+0000 7f26d6a70640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f26d006cb00 con 0x7f26d006c6f0 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.453+0000 7f26d6a70640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26d006e1a0 con 0x7f26d0071be0 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.453+0000 7f26d6a70640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f26d006cc80 con 0x7f26d015d130 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 0x7f26d014e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 0x7f26d014e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43960/0 (socket says 172.21.15.1:43960) 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 -- 172.21.15.1:0/2482549552 learned_addr learned my addr 172.21.15.1:0/2482549552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d500d640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d015d130 0x7f26d0155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26c7fff640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0071be0 0x7f26d014de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 -- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d015d130 msgr2=0x7f26d0155830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d015d130 0x7f26d0155830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 -- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0071be0 msgr2=0x7f26d014de30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0071be0 0x7f26d014de30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 -- 172.21.15.1:0/2482549552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d0155d70 con 0x7f26d006c6f0 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d480c640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 0x7f26d014e5c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f26cc007ef0 tx=0x7f26cc00d4a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc01a070 con 0x7f26d006c6f0 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26cc003b90 con 0x7f26d006c6f0 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d0155020 con 0x7f26d006c6f0 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.454+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc004030 con 0x7f26d006c6f0 2024-05-09T05:17:37.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.455+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d015bc90 con 0x7f26d006c6f0 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.455+0000 7f26d6a70640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0155f40 0x7f26d015bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.455+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26d015b740 con 0x7f26d0155f40 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.456+0000 7f26c7fff640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0155f40 0x7f26d015bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.456+0000 7f26c7fff640 1 -- 172.21.15.1:0/2482549552 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26d01554b0 con 0x7f26d0155f40 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.456+0000 7f26c7fff640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0155f40 0x7f26d015bed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26d00c43d0 tx=0x7f26c00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.456+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c0042070 con 0x7f26d0155f40 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f26cc01d070 con 0x7f26d006c6f0 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26c57fa640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26bc076200 0x7f26bc078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f26cc0977c0 con 0x7f26d006c6f0 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26d500d640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26bc076200 0x7f26bc078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26c003e660 con 0x7f26d0155f40 2024-05-09T05:17:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0155f40 msgr2=0x7f26d015bed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26c57fa640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0155f40 0x7f26d015bed0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f26d00c43d0 tx=0x7f26c00046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0155f40 msgr2=0x7f26d015bed0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:37.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.457+0000 7f26c57fa640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d0155f40 0x7f26d015bed0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.458+0000 7f26d500d640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26bc076200 0x7f26bc078710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f26d014edb0 tx=0x7f26c8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.464+0000 7f26d6a70640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d014c720 0x7f26d0152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.464+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26d0050870 con 0x7f26d014c720 2024-05-09T05:17:37.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.465+0000 7f26c7fff640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d014c720 0x7f26d0152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.465+0000 7f26c7fff640 1 -- 172.21.15.1:0/2482549552 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26d015a120 con 0x7f26d014c720 2024-05-09T05:17:37.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.465+0000 7f26c7fff640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d014c720 0x7f26d0152ba0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26c002a2b0 tx=0x7f26c0008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.465+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26c0037040 con 0x7f26d014c720 2024-05-09T05:17:37.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7f26c003ee10 con 0x7f26d014c720 2024-05-09T05:17:37.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d014c720 msgr2=0x7f26d0152ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26c57fa640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d014c720 0x7f26d0152ba0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f26c002a2b0 tx=0x7f26c0008030 comp rx=0 tx=0).stop 2024-05-09T05:17:37.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26c57fa640 1 -- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d014c720 msgr2=0x7f26d0152ba0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:37.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26c57fa640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d014c720 0x7f26d0152ba0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.464 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 88, 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 7870, 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:17:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:17:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:17:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.475 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:17:37.476 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.477 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:17:37.477 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:17:37.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26bc076200 msgr2=0x7f26bc078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d6a70640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26bc076200 0x7f26bc078710 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f26d014edb0 tx=0x7f26c8009040 comp rx=0 tx=0).stop 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 msgr2=0x7f26d014e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d6a70640 1 --2- 172.21.15.1:0/2482549552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d006c6f0 0x7f26d014e5c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f26cc007ef0 tx=0x7f26cc00d4a0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d500d640 1 -- 172.21.15.1:0/2482549552 reap_dead start 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d500d640 1 -- 172.21.15.1:0/2482549552 reap_dead start 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 shutdown_connections 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 >> 172.21.15.1:0/2482549552 conn(0x7f26d00747b0 msgr2=0x7f26d00c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 shutdown_connections 2024-05-09T05:17:37.479 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.466+0000 7f26d6a70640 1 -- 172.21.15.1:0/2482549552 wait complete. 2024-05-09T05:17:37.479 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:17:37.480 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:17:37.480 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:17:37.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.620+0000 7f0d15061640 1 Processor -- start 2024-05-09T05:17:37.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.620+0000 7f0d15061640 1 -- start start 2024-05-09T05:17:37.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d15061640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d100c89a0 0x7f0d100ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d15061640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0d100c4af0 0x7f0d10153490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d15061640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0d100c3be0 con 0x7f0d100c46e0 2024-05-09T05:17:37.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d15061640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f0d100c9290 con 0x7f0d100c89a0 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d15061640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0d1006e890 con 0x7f0d100c4af0 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d0ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d100c89a0 0x7f0d100ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d0ed9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f0d100c89a0 0x7f0d100ca930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43962/0 (socket says 172.21.15.1:43962) 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d0ed9d640 1 -- 172.21.15.1:0/327373059 learned_addr learned my addr 172.21.15.1:0/327373059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d0f59e640 1 --2- 172.21.15.1:0/327373059 >> v2:172.21.15.196:3300/0 conn(0x7f0d100c4af0 0x7f0d10153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d0ed9d640 1 -- 172.21.15.1:0/327373059 >> v2:172.21.15.196:3300/0 conn(0x7f0d100c4af0 msgr2=0x7f0d10153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d0ed9d640 1 --2- 172.21.15.1:0/327373059 >> v2:172.21.15.196:3300/0 conn(0x7f0d100c4af0 0x7f0d10153490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d0ed9d640 1 -- 172.21.15.1:0/327373059 >> v1:172.21.15.1:6789/0 conn(0x7f0d100c46e0 legacy=0x7f0d10159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.622+0000 7f0d0ed9d640 1 -- 172.21.15.1:0/327373059 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d100c3e60 con 0x7f0d100c89a0 2024-05-09T05:17:37.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.623+0000 7f0d0ed9d640 1 --2- 172.21.15.1:0/327373059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c89a0 0x7f0d100ca930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0d100c9410 tx=0x7f0d0002a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=134a3b1f8fb68b0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.623+0000 7f0d0dd9b640 1 -- 172.21.15.1:0/327373059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00037070 con 0x7f0d100c89a0 2024-05-09T05:17:37.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.623+0000 7f0d0dd9b640 1 -- 172.21.15.1:0/327373059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d00032440 con 0x7f0d100c89a0 2024-05-09T05:17:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.623+0000 7f0d0dd9b640 1 -- 172.21.15.1:0/327373059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00032810 con 0x7f0d100c89a0 2024-05-09T05:17:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.624+0000 7f0d15061640 1 -- 172.21.15.1:0/327373059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c89a0 msgr2=0x7f0d100ca930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.624+0000 7f0d15061640 1 --2- 172.21.15.1:0/327373059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c89a0 0x7f0d100ca930 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f0d100c9410 tx=0x7f0d0002a1e0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.624+0000 7f0d15061640 1 -- 172.21.15.1:0/327373059 shutdown_connections 2024-05-09T05:17:37.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.624+0000 7f0d15061640 1 --2- 172.21.15.1:0/327373059 >> v2:172.21.15.196:3300/0 conn(0x7f0d100c4af0 0x7f0d10153490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.624+0000 7f0d15061640 1 --2- 172.21.15.1:0/327373059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c89a0 0x7f0d100ca930 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.624+0000 7f0d15061640 1 -- 172.21.15.1:0/327373059 >> 172.21.15.1:0/327373059 conn(0x7f0d100747b0 msgr2=0x7f0d100ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.624+0000 7f0d15061640 1 -- 172.21.15.1:0/327373059 shutdown_connections 2024-05-09T05:17:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.624+0000 7f0d15061640 1 -- 172.21.15.1:0/327373059 wait complete. 2024-05-09T05:17:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.627+0000 7f0d15061640 1 Processor -- start 2024-05-09T05:17:37.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.627+0000 7f0d15061640 1 -- start start 2024-05-09T05:17:37.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.628+0000 7f0d15061640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d100c46e0 0x7f0d101f6770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.628+0000 7f0d15061640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c4af0 0x7f0d1014e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.628+0000 7f0d15061640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d100c89a0 0x7f0d101452b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.628+0000 7f0d15061640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0d100c97c0 con 0x7f0d100c4af0 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.628+0000 7f0d15061640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0d100c94e0 con 0x7f0d100c46e0 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.628+0000 7f0d15061640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0d100cae70 con 0x7f0d100c89a0 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c4af0 0x7f0d1014e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c4af0 0x7f0d1014e760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43978/0 (socket says 172.21.15.1:43978) 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 -- 172.21.15.1:0/408100684 learned_addr learned my addr 172.21.15.1:0/408100684 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0ed9d640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d100c46e0 0x7f0d101f6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0f59e640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d100c89a0 0x7f0d101452b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 -- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d100c89a0 msgr2=0x7f0d101452b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d100c89a0 0x7f0d101452b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 -- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d100c46e0 msgr2=0x7f0d101f6770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0d100c46e0 0x7f0d101f6770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 -- 172.21.15.1:0/408100684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d101f6cb0 con 0x7f0d100c4af0 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d0e59c640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c4af0 0x7f0d1014e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400c3d0 tx=0x7f0d0400c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d04017070 con 0x7f0d100c4af0 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0d040052d0 con 0x7f0d100c4af0 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d10157e50 con 0x7f0d100c4af0 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.629+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d04003830 con 0x7f0d100c4af0 2024-05-09T05:17:37.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.630+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d1014b050 con 0x7f0d100c4af0 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.630+0000 7f0d15061640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d101590c0 0x7f0d101457f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.630+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0d10050870 con 0x7f0d101590c0 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.631+0000 7f0d0ed9d640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d101590c0 0x7f0d101457f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.631+0000 7f0d0ed9d640 1 -- 172.21.15.1:0/408100684 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0d100ca360 con 0x7f0d101590c0 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.631+0000 7f0d0ed9d640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d101590c0 0x7f0d101457f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0d100739e0 tx=0x7f0d000046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.631+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d00042070 con 0x7f0d101590c0 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.632+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0d0003e660 con 0x7f0d101590c0 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.632+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d101590c0 msgr2=0x7f0d101457f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.632+0000 7f0cf77fe640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d101590c0 0x7f0d101457f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0d100739e0 tx=0x7f0d000046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.632+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d101590c0 msgr2=0x7f0d101457f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:37.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.632+0000 7f0cf77fe640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d101590c0 0x7f0d101457f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.632+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0d0401a070 con 0x7f0d100c4af0 2024-05-09T05:17:37.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.632+0000 7f0cf77fe640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cfc075d40 0x7f0cfc078250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.632+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0d04096960 con 0x7f0d100c4af0 2024-05-09T05:17:37.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.633+0000 7f0d0ed9d640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cfc075d40 0x7f0cfc078250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.633+0000 7f0d0ed9d640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cfc075d40 0x7f0cfc078250 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0d00041000 tx=0x7f0d0003d000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.639+0000 7f0d15061640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d10149fb0 0x7f0d1014a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.639+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0d1006d6b0 con 0x7f0d10149fb0 2024-05-09T05:17:37.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.640+0000 7f0d0f59e640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d10149fb0 0x7f0d1014a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.640+0000 7f0d0f59e640 1 -- 172.21.15.1:0/408100684 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0d1006ca70 con 0x7f0d10149fb0 2024-05-09T05:17:37.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.640+0000 7f0d0f59e640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d10149fb0 0x7f0d1014a3c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009d90 tx=0x7f0d0800f580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.640+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d08003b90 con 0x7f0d10149fb0 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4550 (secure 0 0 0) 0x7f0d08003b90 con 0x7f0d10149fb0 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d10149fb0 msgr2=0x7f0d1014a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0cf77fe640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d10149fb0 0x7f0d1014a3c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009d90 tx=0x7f0d0800f580 comp rx=0 tx=0).stop 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0cf77fe640 1 -- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d10149fb0 msgr2=0x7f0d1014a3c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0cf77fe640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0d10149fb0 0x7f0d1014a3c0 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0d08009d90 tx=0x7f0d0800f580 comp rx=0 tx=0).stop 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:37.639 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 88, 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 78936, 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 2, 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.640 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.641 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.642 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:17:37.643 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:37.644 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.645 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.646 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:17:37.647 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:37.648 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:17:37.649 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:37.650 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:17:37.651 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:17:37.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cfc075d40 msgr2=0x7f0cfc078250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d15061640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cfc075d40 0x7f0cfc078250 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f0d00041000 tx=0x7f0d0003d000 comp rx=0 tx=0).stop 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c4af0 msgr2=0x7f0d1014e760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d15061640 1 --2- 172.21.15.1:0/408100684 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0d100c4af0 0x7f0d1014e760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0d0400c3d0 tx=0x7f0d0400c890 comp rx=0 tx=0).stop 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d0f59e640 1 -- 172.21.15.1:0/408100684 reap_dead start 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d0f59e640 1 -- 172.21.15.1:0/408100684 reap_dead start 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 shutdown_connections 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 >> 172.21.15.1:0/408100684 conn(0x7f0d100747b0 msgr2=0x7f0d10072070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 shutdown_connections 2024-05-09T05:17:37.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.641+0000 7f0d15061640 1 -- 172.21.15.1:0/408100684 wait complete. 2024-05-09T05:17:37.653 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:17:37.654 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:17:37.654 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:17:37.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.795+0000 7faf25e3b640 1 Processor -- start 2024-05-09T05:17:37.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.795+0000 7faf25e3b640 1 -- start start 2024-05-09T05:17:37.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf25e3b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7faf200c5ca0 0x7faf200ca4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf25e3b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf20156e90 0x7faf200c7750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf25e3b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7faf200c86a0 con 0x7faf20156a80 2024-05-09T05:17:37.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf25e3b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7faf200cb910 con 0x7faf20156e90 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf25e3b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7faf200c8980 con 0x7faf200c5ca0 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf1ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf20156e90 0x7faf200c7750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf1ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7faf20156e90 0x7faf200c7750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43984/0 (socket says 172.21.15.1:43984) 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf1ffff640 1 -- 172.21.15.1:0/2548570144 learned_addr learned my addr 172.21.15.1:0/2548570144 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf1f7fe640 1 --2- 172.21.15.1:0/2548570144 >> v2:172.21.15.196:3300/0 conn(0x7faf200c5ca0 0x7faf200ca4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf1ffff640 1 -- 172.21.15.1:0/2548570144 >> v2:172.21.15.196:3300/0 conn(0x7faf200c5ca0 msgr2=0x7faf200ca4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf1ffff640 1 --2- 172.21.15.1:0/2548570144 >> v2:172.21.15.196:3300/0 conn(0x7faf200c5ca0 0x7faf200ca4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf1ffff640 1 -- 172.21.15.1:0/2548570144 >> v1:172.21.15.196:6789/0 conn(0x7faf20156a80 legacy=0x7faf2006df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:37.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.797+0000 7faf1ffff640 1 -- 172.21.15.1:0/2548570144 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf200c8c60 con 0x7faf20156e90 2024-05-09T05:17:37.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.798+0000 7faf1ffff640 1 --2- 172.21.15.1:0/2548570144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf200c7750 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faf20072d60 tx=0x7faf1802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e16dffa873ba714 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.798+0000 7faf1e7fc640 1 -- 172.21.15.1:0/2548570144 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf18036070 con 0x7faf20156e90 2024-05-09T05:17:37.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.798+0000 7faf1e7fc640 1 -- 172.21.15.1:0/2548570144 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf18031600 con 0x7faf20156e90 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.798+0000 7faf1e7fc640 1 -- 172.21.15.1:0/2548570144 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf180319d0 con 0x7faf20156e90 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.799+0000 7faf25e3b640 1 -- 172.21.15.1:0/2548570144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 msgr2=0x7faf200c7750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.799+0000 7faf25e3b640 1 --2- 172.21.15.1:0/2548570144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf200c7750 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7faf20072d60 tx=0x7faf1802a730 comp rx=0 tx=0).stop 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.799+0000 7faf25e3b640 1 -- 172.21.15.1:0/2548570144 shutdown_connections 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.799+0000 7faf25e3b640 1 --2- 172.21.15.1:0/2548570144 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf200c7750 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.799+0000 7faf25e3b640 1 --2- 172.21.15.1:0/2548570144 >> v2:172.21.15.196:3300/0 conn(0x7faf200c5ca0 0x7faf200ca4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.799+0000 7faf25e3b640 1 -- 172.21.15.1:0/2548570144 >> 172.21.15.1:0/2548570144 conn(0x7faf200745a0 msgr2=0x7faf20071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.799+0000 7faf25e3b640 1 -- 172.21.15.1:0/2548570144 shutdown_connections 2024-05-09T05:17:37.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.799+0000 7faf25e3b640 1 -- 172.21.15.1:0/2548570144 wait complete. 2024-05-09T05:17:37.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.802+0000 7faf25e3b640 1 Processor -- start 2024-05-09T05:17:37.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.802+0000 7faf25e3b640 1 -- start start 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf25e3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7faf200c5ca0 0x7faf2015aea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf25e3b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf20156a80 0x7faf201574c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf25e3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf20155a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf25e3b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7faf200cad60 con 0x7faf20156e90 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf25e3b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7faf200cb5a0 con 0x7faf20156a80 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf25e3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7faf200ca9e0 con 0x7faf200c5ca0 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf20155a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf20155a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43988/0 (socket says 172.21.15.1:43988) 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 -- 172.21.15.1:0/1656442060 learned_addr learned my addr 172.21.15.1:0/1656442060 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:37.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1f7fe640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7faf200c5ca0 0x7faf2015aea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1effd640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf20156a80 0x7faf201574c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 -- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7faf200c5ca0 msgr2=0x7faf2015aea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7faf200c5ca0 0x7faf2015aea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 -- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf20156a80 msgr2=0x7faf201574c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf20156a80 0x7faf201574c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 -- 172.21.15.1:0/1656442060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faf2015b3e0 con 0x7faf20156e90 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.804+0000 7faf1ffff640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf20155a90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faf18009f40 tx=0x7faf18004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.805+0000 7faf07fff640 1 -- 172.21.15.1:0/1656442060 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf18036050 con 0x7faf20156e90 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.805+0000 7faf07fff640 1 -- 172.21.15.1:0/1656442060 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7faf18042070 con 0x7faf20156e90 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.805+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faf20151c30 con 0x7faf20156e90 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.805+0000 7faf07fff640 1 -- 172.21.15.1:0/1656442060 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faf18009510 con 0x7faf20156e90 2024-05-09T05:17:37.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.805+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faf201f35f0 con 0x7faf20156e90 2024-05-09T05:17:37.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.806+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faf20050870 con 0x7faf20156e90 2024-05-09T05:17:37.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.807+0000 7faf07fff640 1 -- 172.21.15.1:0/1656442060 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7faf18007bf0 con 0x7faf20156e90 2024-05-09T05:17:37.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.807+0000 7faf07fff640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7faf0c0762d0 0x7faf0c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:37.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.807+0000 7faf07fff640 1 -- 172.21.15.1:0/1656442060 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7faf180b82d0 con 0x7faf20156e90 2024-05-09T05:17:37.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.807+0000 7faf1f7fe640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7faf0c0762d0 0x7faf0c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:37.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.808+0000 7faf1f7fe640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7faf0c0762d0 0x7faf0c0787e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7faf100079a0 tx=0x7faf10008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:37.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:37.813+0000 7faf07fff640 1 -- 172.21.15.1:0/1656442060 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7faf180815f0 con 0x7faf20156e90 2024-05-09T05:17:38.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.125+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7faf201f3060 con 0x7faf20156e90 2024-05-09T05:17:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.126+0000 7faf07fff640 1 -- 172.21.15.1:0/1656442060 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7faf180858f0 con 0x7faf20156e90 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7faf0c0762d0 msgr2=0x7faf0c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7faf0c0762d0 0x7faf0c0787e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7faf100079a0 tx=0x7faf10008040 comp rx=0 tx=0).stop 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 msgr2=0x7faf20155a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf20155a90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7faf18009f40 tx=0x7faf18004a40 comp rx=0 tx=0).stop 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 shutdown_connections 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7faf0c0762d0 0x7faf0c0787e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7faf20156a80 0x7faf201574c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7faf20156e90 0x7faf20155a90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 --2- 172.21.15.1:0/1656442060 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7faf200c5ca0 0x7faf2015aea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 >> 172.21.15.1:0/1656442060 conn(0x7faf200745a0 msgr2=0x7faf200c9870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 shutdown_connections 2024-05-09T05:17:38.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:38.129+0000 7faf25e3b640 1 -- 172.21.15.1:0/1656442060 wait complete. 2024-05-09T05:17:38.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:17:44.504 INFO:tasks.workunit.client.0.smithi001.stdout:fs setup took 546 secs 2024-05-09T05:17:46.026 INFO:tasks.workunit.client.0.smithi001.stdout:Syncing()...1 sec 2024-05-09T05:17:46.027 INFO:tasks.workunit.client.0.smithi001.stdout:Starting Actual Benchmark At: Thu May 9 05:17:46 2024 2024-05-09T05:17:46.027 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:17:48.143 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:17:48.143 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:17:48.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.345+0000 7f9c2649f640 1 Processor -- start 2024-05-09T05:17:48.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.345+0000 7f9c2649f640 1 -- start start 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.347+0000 7f9c2649f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f9c200c9520 0x7f9c20070c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.347+0000 7f9c2649f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c20154ae0 0x7f9c2006ff10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.347+0000 7f9c2649f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c2006cc50 con 0x7f9c2006c6f0 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.347+0000 7f9c2649f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f9c200c8530 con 0x7f9c200c9520 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.347+0000 7f9c2649f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9c2006cf30 con 0x7f9c20154ae0 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c1f7fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f9c200c9520 0x7f9c20070c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c1ffff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9c2006c6f0 0x7f9c2015a7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52902/0 (socket says 172.21.15.1:52902) 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c1ffff640 1 -- 172.21.15.1:0/1992736696 learned_addr learned my addr 172.21.15.1:0/1992736696 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c24a3c640 1 --2- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c20154ae0 0x7f9c2006ff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:48.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c20154ae0 msgr2=0x7f9c2006ff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c1f7fe640 1 --2- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c20154ae0 0x7f9c2006ff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/1992736696 >> v1:172.21.15.1:6789/0 conn(0x7f9c2006c6f0 legacy=0x7f9c2015a7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c1effd640 1 -- 172.21.15.1:0/1992736696 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 734415296 0 0) 0x7f9c2006cc50 con 0x7f9c2006c6f0 2024-05-09T05:17:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/1992736696 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c200c7a50 con 0x7f9c200c9520 2024-05-09T05:17:48.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.348+0000 7f9c24a3c640 1 --2- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c20154ae0 0x7f9c2006ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:17:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.349+0000 7f9c1f7fe640 1 --2- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c200c9520 0x7f9c20070c50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c140097a0 tx=0x7f9c1402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15dfa4b1c3ed098b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.349+0000 7f9c1effd640 1 -- 172.21.15.1:0/1992736696 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14037070 con 0x7f9c200c9520 2024-05-09T05:17:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.350+0000 7f9c1effd640 1 -- 172.21.15.1:0/1992736696 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c14032440 con 0x7f9c200c9520 2024-05-09T05:17:48.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.350+0000 7f9c1effd640 1 -- 172.21.15.1:0/1992736696 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14032810 con 0x7f9c200c9520 2024-05-09T05:17:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.351+0000 7f9c2649f640 1 -- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c200c9520 msgr2=0x7f9c20070c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.351+0000 7f9c2649f640 1 --2- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c200c9520 0x7f9c20070c50 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f9c140097a0 tx=0x7f9c1402a180 comp rx=0 tx=0).stop 2024-05-09T05:17:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.351+0000 7f9c2649f640 1 -- 172.21.15.1:0/1992736696 shutdown_connections 2024-05-09T05:17:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.351+0000 7f9c2649f640 1 --2- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c20154ae0 0x7f9c2006ff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.351+0000 7f9c2649f640 1 --2- 172.21.15.1:0/1992736696 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c200c9520 0x7f9c20070c50 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.351+0000 7f9c2649f640 1 -- 172.21.15.1:0/1992736696 >> 172.21.15.1:0/1992736696 conn(0x7f9c200ceb20 msgr2=0x7f9c200cef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.351+0000 7f9c2649f640 1 -- 172.21.15.1:0/1992736696 shutdown_connections 2024-05-09T05:17:48.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.351+0000 7f9c2649f640 1 -- 172.21.15.1:0/1992736696 wait complete. 2024-05-09T05:17:48.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.355+0000 7f9c2649f640 1 Processor -- start 2024-05-09T05:17:48.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.356+0000 7f9c2649f640 1 -- start start 2024-05-09T05:17:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.358+0000 7f9c2649f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2006c6f0 0x7f9c200ce530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.358+0000 7f9c2649f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c200c9520 0x7f9c2014c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c2649f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c20154ae0 0x7f9c200c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2006c6f0 0x7f9c200ce530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c2649f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9c200c7750 con 0x7f9c20154ae0 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c2649f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9c200c8010 con 0x7f9c2006c6f0 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c2649f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9c20154ef0 con 0x7f9c200c9520 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2006c6f0 0x7f9c200ce530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:49620/0 (socket says 172.21.15.1:49620) 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1ffff640 1 -- 172.21.15.1:0/3693103522 learned_addr learned my addr 172.21.15.1:0/3693103522 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1f7fe640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c200c9520 0x7f9c2014c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c24a3c640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c20154ae0 0x7f9c200c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2006c6f0 msgr2=0x7f9c200ce530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1f7fe640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2006c6f0 0x7f9c200ce530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/3693103522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c20154ae0 msgr2=0x7f9c200c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1f7fe640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c20154ae0 0x7f9c200c3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1f7fe640 1 -- 172.21.15.1:0/3693103522 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c2014d100 con 0x7f9c200c9520 2024-05-09T05:17:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.359+0000 7f9c1ffff640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2006c6f0 0x7f9c200ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:17:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.360+0000 7f9c1f7fe640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c200c9520 0x7f9c2014c360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9c14031000 tx=0x7f9c140046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.360+0000 7f9c1cff9640 1 -- 172.21.15.1:0/3693103522 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14041070 con 0x7f9c200c9520 2024-05-09T05:17:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.360+0000 7f9c1cff9640 1 -- 172.21.15.1:0/3693103522 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c1403c8b0 con 0x7f9c200c9520 2024-05-09T05:17:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.360+0000 7f9c1cff9640 1 -- 172.21.15.1:0/3693103522 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c14037040 con 0x7f9c200c9520 2024-05-09T05:17:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.360+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c201f32e0 con 0x7f9c200c9520 2024-05-09T05:17:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.360+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c200cd7b0 con 0x7f9c200c9520 2024-05-09T05:17:48.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.362+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c200c8a60 con 0x7f9c200c9520 2024-05-09T05:17:48.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.364+0000 7f9c1cff9640 1 -- 172.21.15.1:0/3693103522 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9c14044070 con 0x7f9c200c9520 2024-05-09T05:17:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.364+0000 7f9c1cff9640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c0c076320 0x7f9c0c078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.365+0000 7f9c1cff9640 1 -- 172.21.15.1:0/3693103522 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9c140b7ee0 con 0x7f9c200c9520 2024-05-09T05:17:48.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.365+0000 7f9c1ffff640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c0c076320 0x7f9c0c078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:48.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.365+0000 7f9c1ffff640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c0c076320 0x7f9c0c078830 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9c10000f60 tx=0x7f9c10007e70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:48.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.371+0000 7f9c1cff9640 1 -- 172.21.15.1:0/3693103522 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9c200c8a60 con 0x7f9c200c9520 2024-05-09T05:17:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.836+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9c200c69b0 con 0x7f9c200c9520 2024-05-09T05:17:48.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.837+0000 7f9c1cff9640 1 -- 172.21.15.1:0/3693103522 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f9c200c69b0 con 0x7f9c200c9520 2024-05-09T05:17:48.835 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.840+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c0c076320 msgr2=0x7f9c0c078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.840+0000 7f9c2649f640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c0c076320 0x7f9c0c078830 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f9c10000f60 tx=0x7f9c10007e70 comp rx=0 tx=0).stop 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.840+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c200c9520 msgr2=0x7f9c2014c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.840+0000 7f9c2649f640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c200c9520 0x7f9c2014c360 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f9c14031000 tx=0x7f9c140046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.841+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 shutdown_connections 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.841+0000 7f9c2649f640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c0c076320 0x7f9c0c078830 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.841+0000 7f9c2649f640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c20154ae0 0x7f9c200c3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.841+0000 7f9c2649f640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c200c9520 0x7f9c2014c360 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.841+0000 7f9c2649f640 1 --2- 172.21.15.1:0/3693103522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c2006c6f0 0x7f9c200ce530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.841+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 >> 172.21.15.1:0/3693103522 conn(0x7f9c200ceb20 msgr2=0x7f9c200b47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.841+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 shutdown_connections 2024-05-09T05:17:48.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:48.841+0000 7f9c2649f640 1 -- 172.21.15.1:0/3693103522 wait complete. 2024-05-09T05:17:48.861 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:17:48.861 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:17:48.861 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:17:48.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:17:49.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.060+0000 7f20baaf9640 1 Processor -- start 2024-05-09T05:17:49.058 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.060+0000 7f20baaf9640 1 -- start start 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20baaf9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f20b415b1d0 0x7f20b40cd380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20baaf9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f20b40c8130 0x7f20b40c71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20baaf9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f20b40c5810 con 0x7f20b40c7710 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20baaf9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f20b40c6e10 con 0x7f20b40c8130 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20baaf9640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f20b40c5af0 con 0x7f20b415b1d0 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20b9096640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f20b40c7710 0x7f20b4161320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52910/0 (socket says 172.21.15.1:52910) 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20b9096640 1 -- 172.21.15.1:0/2020794986 learned_addr learned my addr 172.21.15.1:0/2020794986 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20b8895640 1 --2- 172.21.15.1:0/2020794986 >> v2:172.21.15.196:3300/0 conn(0x7f20b415b1d0 0x7f20b40cd380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.062+0000 7f20b3fff640 1 --2- 172.21.15.1:0/2020794986 >> v2:172.21.15.1:3300/0 conn(0x7f20b40c8130 0x7f20b40c71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.063+0000 7f20b37fe640 1 -- 172.21.15.1:0/2020794986 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4070650107 0 0) 0x7f20b40c5810 con 0x7f20b40c7710 2024-05-09T05:17:49.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.063+0000 7f20b37fe640 1 -- 172.21.15.1:0/2020794986 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f20a4003520 con 0x7f20b40c7710 2024-05-09T05:17:49.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.063+0000 7f20b8895640 1 -- 172.21.15.1:0/2020794986 >> v2:172.21.15.1:3300/0 conn(0x7f20b40c8130 msgr2=0x7f20b40c71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:49.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.063+0000 7f20b8895640 1 --2- 172.21.15.1:0/2020794986 >> v2:172.21.15.1:3300/0 conn(0x7f20b40c8130 0x7f20b40c71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.063+0000 7f20b8895640 1 -- 172.21.15.1:0/2020794986 >> v1:172.21.15.1:6789/0 conn(0x7f20b40c7710 legacy=0x7f20b4161320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.062 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.063+0000 7f20b8895640 1 -- 172.21.15.1:0/2020794986 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b40cd930 con 0x7f20b415b1d0 2024-05-09T05:17:49.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.064+0000 7f20b8895640 1 --2- 172.21.15.1:0/2020794986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 0x7f20b40cd380 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f20b40c5540 tx=0x7f20a002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=10ebd75f9362731d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.064+0000 7f20b37fe640 1 -- 172.21.15.1:0/2020794986 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a0037070 con 0x7f20b415b1d0 2024-05-09T05:17:49.063 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.064+0000 7f20b37fe640 1 -- 172.21.15.1:0/2020794986 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20a0032440 con 0x7f20b415b1d0 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.064+0000 7f20b37fe640 1 -- 172.21.15.1:0/2020794986 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a002ae60 con 0x7f20b415b1d0 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.065+0000 7f20baaf9640 1 -- 172.21.15.1:0/2020794986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 msgr2=0x7f20b40cd380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.065+0000 7f20baaf9640 1 --2- 172.21.15.1:0/2020794986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 0x7f20b40cd380 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f20b40c5540 tx=0x7f20a002a110 comp rx=0 tx=0).stop 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.065+0000 7f20baaf9640 1 -- 172.21.15.1:0/2020794986 shutdown_connections 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.065+0000 7f20baaf9640 1 --2- 172.21.15.1:0/2020794986 >> v2:172.21.15.1:3300/0 conn(0x7f20b40c8130 0x7f20b40c71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.065+0000 7f20baaf9640 1 --2- 172.21.15.1:0/2020794986 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 0x7f20b40cd380 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.065+0000 7f20baaf9640 1 -- 172.21.15.1:0/2020794986 >> 172.21.15.1:0/2020794986 conn(0x7f20b40cac70 msgr2=0x7f20b40cb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.065+0000 7f20baaf9640 1 -- 172.21.15.1:0/2020794986 shutdown_connections 2024-05-09T05:17:49.064 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.066+0000 7f20baaf9640 1 -- 172.21.15.1:0/2020794986 wait complete. 2024-05-09T05:17:49.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.068+0000 7f20baaf9640 1 Processor -- start 2024-05-09T05:17:49.066 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.069+0000 7f20baaf9640 1 -- start start 2024-05-09T05:17:49.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20baaf9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b40c7710 0x7f20b41fabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20baaf9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20b40c8130 0x7f20b4153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20baaf9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 0x7f20b4164450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.069 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20baaf9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f20b406de10 con 0x7f20b40c7710 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20baaf9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f20b40cdef0 con 0x7f20b415b1d0 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20baaf9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f20b406df90 con 0x7f20b40c8130 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20b8895640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b40c7710 0x7f20b41fabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20b8895640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b40c7710 0x7f20b41fabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56340/0 (socket says 172.21.15.1:56340) 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20b8895640 1 -- 172.21.15.1:0/1950265522 learned_addr learned my addr 172.21.15.1:0/1950265522 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20b9096640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 0x7f20b4164450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.070+0000 7f20b3fff640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20b40c8130 0x7f20b4153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b8895640 1 -- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20b40c8130 msgr2=0x7f20b4153970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b8895640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20b40c8130 0x7f20b4153970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b8895640 1 -- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 msgr2=0x7f20b4164450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b8895640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 0x7f20b4164450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b8895640 1 -- 172.21.15.1:0/1950265522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b406d160 con 0x7f20b40c7710 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b3fff640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f20b40c8130 0x7f20b4153970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:17:49.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b9096640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f20b415b1d0 0x7f20b4164450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:17:49.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b8895640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b40c7710 0x7f20b41fabf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009890 tx=0x7f20a00046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.071+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a0041020 con 0x7f20b40c7710 2024-05-09T05:17:49.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.072+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f20a003d640 con 0x7f20b40c7710 2024-05-09T05:17:49.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.072+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a0037040 con 0x7f20b40c7710 2024-05-09T05:17:49.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.072+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b41520b0 con 0x7f20b40c7710 2024-05-09T05:17:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.072+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b4152630 con 0x7f20b40c7710 2024-05-09T05:17:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.073+0000 7f20baaf9640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b415b6a0 0x7f20b4151320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.073+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f20b4154220 con 0x7f20b415b6a0 2024-05-09T05:17:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.074+0000 7f20b3fff640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b415b6a0 0x7f20b4151320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.074+0000 7f20b3fff640 1 -- 172.21.15.1:0/1950265522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f20b40c5810 con 0x7f20b415b6a0 2024-05-09T05:17:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.074+0000 7f20b3fff640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b415b6a0 0x7f20b4151320 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f20a80115d0 tx=0x7f20a8011a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.074+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f20a0044020 con 0x7f20b40c7710 2024-05-09T05:17:49.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.075+0000 7f20b17fa640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20980762d0 0x7f20980787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.075+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f20a00b7db0 con 0x7f20b40c7710 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.075+0000 7f20b9096640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20980762d0 0x7f20980787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.076+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a800f910 con 0x7f20b415b6a0 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.076+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f20a800fab0 con 0x7f20b415b6a0 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.076+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b415b6a0 msgr2=0x7f20b4151320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.076+0000 7f20b17fa640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b415b6a0 0x7f20b4151320 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f20a80115d0 tx=0x7f20a8011a90 comp rx=0 tx=0).stop 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.076+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b415b6a0 msgr2=0x7f20b4151320 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.076+0000 7f20b17fa640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b415b6a0 0x7f20b4151320 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.076+0000 7f20b9096640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20980762d0 0x7f20980787e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0012e0 tx=0x7f20ac001600 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.088+0000 7f20baaf9640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b4151860 0x7f20b41f6c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.089+0000 7f20b3fff640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b4151860 0x7f20b41f6c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.089+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f20b4152800 con 0x7f20b4151860 2024-05-09T05:17:49.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.090+0000 7f20b3fff640 1 -- 172.21.15.1:0/1950265522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f20b4151c70 con 0x7f20b4151860 2024-05-09T05:17:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.090+0000 7f20b3fff640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b4151860 0x7f20b41f6c30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f20a80115d0 tx=0x7f20a80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a8006040 con 0x7f20b4151860 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7f20a80061e0 con 0x7f20b4151860 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b4151860 msgr2=0x7f20b41f6c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20b17fa640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b4151860 0x7f20b41f6c30 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f20a80115d0 tx=0x7f20a80046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 88, 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 271284, 2024-05-09T05:17:49.089 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 14, 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.090 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.091 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:49.092 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.093 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.094 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:17:49.095 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.096 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.097 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.098 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.099 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.100 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.101 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:17:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.102 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:17:49.102 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20b17fa640 1 -- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b4151860 msgr2=0x7f20b41f6c30 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20b17fa640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b4151860 0x7f20b41f6c30 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20980762d0 msgr2=0x7f20980787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20baaf9640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f20980762d0 0x7f20980787e0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0012e0 tx=0x7f20ac001600 comp rx=0 tx=0).stop 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20b9096640 1 -- 172.21.15.1:0/1950265522 reap_dead start 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b40c7710 msgr2=0x7f20b41fabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20baaf9640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b40c7710 0x7f20b41fabf0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f20a0009890 tx=0x7f20a00046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 shutdown_connections 2024-05-09T05:17:49.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20baaf9640 1 --2- 172.21.15.1:0/1950265522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f20b40c7710 0x7f20b41fabf0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.091+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 >> 172.21.15.1:0/1950265522 conn(0x7f20b40cac70 msgr2=0x7f20b40cbfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:49.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.092+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 shutdown_connections 2024-05-09T05:17:49.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.092+0000 7f20baaf9640 1 -- 172.21.15.1:0/1950265522 wait complete. 2024-05-09T05:17:49.107 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:17:49.107 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:17:49.107 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:17:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.294+0000 7f26db772640 1 Processor -- start 2024-05-09T05:17:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.294+0000 7f26db772640 1 -- start start 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26db772640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40b4ec0 0x7f26d40cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26db772640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26d40ce010 0x7f26d415fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26db772640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26d415c7b0 con 0x7f26d4160290 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26db772640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f26d415d0c0 con 0x7f26d40ce010 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26db772640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f26d40cca50 con 0x7f26d40b4ec0 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26d8d0d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f26d40ce010 0x7f26d415fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26d950e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40b4ec0 0x7f26d40cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26d9d0f640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f26d4160290 0x7f26d41606a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:44438/0 (socket says 172.21.15.1:44438) 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.296+0000 7f26d9d0f640 1 -- 172.21.15.1:0/3361993521 learned_addr learned my addr 172.21.15.1:0/3361993521 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.297+0000 7f26cbfff640 1 -- 172.21.15.1:0/3361993521 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4258329779 0 0) 0x7f26d415c7b0 con 0x7f26d4160290 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.297+0000 7f26cbfff640 1 -- 172.21.15.1:0/3361993521 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f26c0003520 con 0x7f26d4160290 2024-05-09T05:17:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.297+0000 7f26d950e640 1 -- 172.21.15.1:0/3361993521 >> v2:172.21.15.1:3300/0 conn(0x7f26d40ce010 msgr2=0x7f26d415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:49.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.297+0000 7f26d950e640 1 --2- 172.21.15.1:0/3361993521 >> v2:172.21.15.1:3300/0 conn(0x7f26d40ce010 0x7f26d415fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.297+0000 7f26d950e640 1 -- 172.21.15.1:0/3361993521 >> v1:172.21.15.196:6789/0 conn(0x7f26d4160290 legacy=0x7f26d41606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.297+0000 7f26d950e640 1 -- 172.21.15.1:0/3361993521 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d40ccd30 con 0x7f26d40b4ec0 2024-05-09T05:17:49.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.298+0000 7f26d950e640 1 --2- 172.21.15.1:0/3361993521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40b4ec0 0x7f26d40cf9d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f26d415d240 tx=0x7f26d002a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20e18b92eb9ef4df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.298+0000 7f26cbfff640 1 -- 172.21.15.1:0/3361993521 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0037070 con 0x7f26d40b4ec0 2024-05-09T05:17:49.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.298+0000 7f26cbfff640 1 -- 172.21.15.1:0/3361993521 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26d0032440 con 0x7f26d40b4ec0 2024-05-09T05:17:49.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.299+0000 7f26cbfff640 1 -- 172.21.15.1:0/3361993521 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0032810 con 0x7f26d40b4ec0 2024-05-09T05:17:49.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.300+0000 7f26db772640 1 -- 172.21.15.1:0/3361993521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40b4ec0 msgr2=0x7f26d40cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.300+0000 7f26db772640 1 --2- 172.21.15.1:0/3361993521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40b4ec0 0x7f26d40cf9d0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f26d415d240 tx=0x7f26d002a210 comp rx=0 tx=0).stop 2024-05-09T05:17:49.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.300+0000 7f26db772640 1 -- 172.21.15.1:0/3361993521 shutdown_connections 2024-05-09T05:17:49.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.300+0000 7f26db772640 1 --2- 172.21.15.1:0/3361993521 >> v2:172.21.15.1:3300/0 conn(0x7f26d40ce010 0x7f26d415fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.300+0000 7f26db772640 1 --2- 172.21.15.1:0/3361993521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40b4ec0 0x7f26d40cf9d0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.300+0000 7f26db772640 1 -- 172.21.15.1:0/3361993521 >> 172.21.15.1:0/3361993521 conn(0x7f26d40d2360 msgr2=0x7f26d40ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:49.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.300+0000 7f26db772640 1 -- 172.21.15.1:0/3361993521 shutdown_connections 2024-05-09T05:17:49.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.300+0000 7f26db772640 1 -- 172.21.15.1:0/3361993521 wait complete. 2024-05-09T05:17:49.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.302+0000 7f26db772640 1 Processor -- start 2024-05-09T05:17:49.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.302+0000 7f26db772640 1 -- start start 2024-05-09T05:17:49.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.304+0000 7f26db772640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d40b4ec0 0x7f26d415da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.304+0000 7f26db772640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40ce010 0x7f26d40cb390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.304+0000 7f26db772640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d4160290 0x7f26d4154e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d950e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d40b4ec0 0x7f26d415da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d950e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d40b4ec0 0x7f26d415da70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:49638/0 (socket says 172.21.15.1:49638) 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d8d0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40ce010 0x7f26d40cb390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d8d0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40ce010 0x7f26d40cb390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:51516/0 (socket says 172.21.15.1:51516) 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26db772640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f26d40cc750 con 0x7f26d4160290 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26db772640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26d415cd50 con 0x7f26d40b4ec0 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26db772640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f26d40cff10 con 0x7f26d40ce010 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d9d0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d4160290 0x7f26d4154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d9d0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d4160290 0x7f26d4154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56382/0 (socket says 172.21.15.1:56382) 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d9d0f640 1 -- 172.21.15.1:0/2880120609 learned_addr learned my addr 172.21.15.1:0/2880120609 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d9d0f640 1 -- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40ce010 msgr2=0x7f26d40cb390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d9d0f640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40ce010 0x7f26d40cb390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d9d0f640 1 -- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d40b4ec0 msgr2=0x7f26d415da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d9d0f640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d40b4ec0 0x7f26d415da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d9d0f640 1 -- 172.21.15.1:0/2880120609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26d415dfb0 con 0x7f26d4160290 2024-05-09T05:17:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d8d0d640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26d40ce010 0x7f26d40cb390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:17:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.305+0000 7f26d950e640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d40b4ec0 0x7f26d415da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:17:49.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.306+0000 7f26d9d0f640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d4160290 0x7f26d4154e10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00cad0 tx=0x7f26cc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.306+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc00abd0 con 0x7f26d4160290 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.306+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26cc00ad70 con 0x7f26d4160290 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.306+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26d415f8d0 con 0x7f26d4160290 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.306+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc012600 con 0x7f26d4160290 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.306+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26d40cb8d0 con 0x7f26d4160290 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.307+0000 7f26db772640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d415e180 0x7f26d4155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.307+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26d4050870 con 0x7f26d415e180 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.308+0000 7f26d950e640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d415e180 0x7f26d4155350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.308+0000 7f26d950e640 1 -- 172.21.15.1:0/2880120609 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f26d40cd2b0 con 0x7f26d415e180 2024-05-09T05:17:49.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.309+0000 7f26d950e640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d415e180 0x7f26d4155350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b4a70 tx=0x7f26d00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.309+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f26cc0127a0 con 0x7f26d4160290 2024-05-09T05:17:49.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26c9ffb640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26b8076320 0x7f26b8078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f26cc097980 con 0x7f26d4160290 2024-05-09T05:17:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26d8d0d640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26b8076320 0x7f26b8078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0053070 con 0x7f26d415e180 2024-05-09T05:17:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f26d0048040 con 0x7f26d415e180 2024-05-09T05:17:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d415e180 msgr2=0x7f26d4155350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26c9ffb640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d415e180 0x7f26d4155350 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b4a70 tx=0x7f26d00046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d415e180 msgr2=0x7f26d4155350 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26c9ffb640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d415e180 0x7f26d4155350 secure :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f26d40b4a70 tx=0x7f26d00046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.310+0000 7f26d8d0d640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26b8076320 0x7f26b8078830 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f26d40cfd70 tx=0x7f26c4009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.322+0000 7f26db772640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d4158350 0x7f26d4157a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.322+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26d40cc000 con 0x7f26d4158350 2024-05-09T05:17:49.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.322+0000 7f26d950e640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d4158350 0x7f26d4157a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.323+0000 7f26d950e640 1 -- 172.21.15.1:0/2880120609 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f26d415d540 con 0x7f26d4158350 2024-05-09T05:17:49.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.323+0000 7f26d950e640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d4158350 0x7f26d4157a60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f26d0000c00 tx=0x7f26d0008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0003820 con 0x7f26d4158350 2024-05-09T05:17:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f26d0003820 con 0x7f26d4158350 2024-05-09T05:17:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d4158350 msgr2=0x7f26d4157a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26c9ffb640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d4158350 0x7f26d4157a60 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f26d0000c00 tx=0x7f26d0008030 comp rx=0 tx=0).stop 2024-05-09T05:17:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26c9ffb640 1 -- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d4158350 msgr2=0x7f26d4157a60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:49.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26c9ffb640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26d4158350 0x7f26d4157a60 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f26d0000c00 tx=0x7f26d0008030 comp rx=0 tx=0).stop 2024-05-09T05:17:49.322 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 88, 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 19728, 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 14, 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:49.323 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.324 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.325 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:17:49.326 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.327 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.328 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.329 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.330 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.331 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.332 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:17:49.333 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.334 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.334 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.334 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:49.334 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.334 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.334 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.334 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:17:49.334 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:17:49.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26b8076320 msgr2=0x7f26b8078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26db772640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26b8076320 0x7f26b8078830 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f26d40cfd70 tx=0x7f26c4009190 comp rx=0 tx=0).stop 2024-05-09T05:17:49.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d4160290 msgr2=0x7f26d4154e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26db772640 1 --2- 172.21.15.1:0/2880120609 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26d4160290 0x7f26d4154e10 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f26cc00cad0 tx=0x7f26cc00cf90 comp rx=0 tx=0).stop 2024-05-09T05:17:49.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26d9d0f640 1 -- 172.21.15.1:0/2880120609 reap_dead start 2024-05-09T05:17:49.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.324+0000 7f26d9d0f640 1 -- 172.21.15.1:0/2880120609 reap_dead start 2024-05-09T05:17:49.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.325+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 shutdown_connections 2024-05-09T05:17:49.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.325+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 >> 172.21.15.1:0/2880120609 conn(0x7f26d40d2360 msgr2=0x7f26d40cf060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:49.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.325+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 shutdown_connections 2024-05-09T05:17:49.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.325+0000 7f26db772640 1 -- 172.21.15.1:0/2880120609 wait complete. 2024-05-09T05:17:49.344 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:17:49.344 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:17:49.344 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:17:49.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.535+0000 7f23e9c29640 1 Processor -- start 2024-05-09T05:17:49.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.535+0000 7f23e9c29640 1 -- start start 2024-05-09T05:17:49.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e9c29640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f23e41569c0 0x7f23e40716a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e9c29640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e4071be0 0x7f23e40c48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e9c29640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23e4156dd0 con 0x7f23e415d130 2024-05-09T05:17:49.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e9c29640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f23e415d540 con 0x7f23e41569c0 2024-05-09T05:17:49.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e9c29640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23e40c3ce0 con 0x7f23e4071be0 2024-05-09T05:17:49.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e4071be0 0x7f23e40c48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e4071be0 0x7f23e40c48c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:51522/0 (socket says 172.21.15.1:51522) 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e3fff640 1 -- 172.21.15.1:0/681634938 learned_addr learned my addr 172.21.15.1:0/681634938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e37fe640 1 --1- 172.21.15.1:0/681634938 >> v1:172.21.15.1:6789/0 conn(0x7f23e415d130 0x7f23e406c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52926/0 (socket says 172.21.15.1:52926) 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.537+0000 7f23e2ffd640 1 --2- 172.21.15.1:0/681634938 >> v2:172.21.15.1:3300/0 conn(0x7f23e41569c0 0x7f23e40716a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.538+0000 7f23e3fff640 1 -- 172.21.15.1:0/681634938 >> v2:172.21.15.1:3300/0 conn(0x7f23e41569c0 msgr2=0x7f23e40716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.538+0000 7f23e3fff640 1 --2- 172.21.15.1:0/681634938 >> v2:172.21.15.1:3300/0 conn(0x7f23e41569c0 0x7f23e40716a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.538+0000 7f23e3fff640 1 -- 172.21.15.1:0/681634938 >> v1:172.21.15.1:6789/0 conn(0x7f23e415d130 legacy=0x7f23e406c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.538+0000 7f23e3fff640 1 -- 172.21.15.1:0/681634938 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e40c3fc0 con 0x7f23e4071be0 2024-05-09T05:17:49.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.539+0000 7f23e3fff640 1 --2- 172.21.15.1:0/681634938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e4071be0 0x7f23e40c48c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009e50 tx=0x7f23dc02a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=997b390debd31106 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.539+0000 7f23e27fc640 1 -- 172.21.15.1:0/681634938 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc036070 con 0x7f23e4071be0 2024-05-09T05:17:49.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.539+0000 7f23e27fc640 1 -- 172.21.15.1:0/681634938 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23dc031600 con 0x7f23e4071be0 2024-05-09T05:17:49.537 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.539+0000 7f23e27fc640 1 -- 172.21.15.1:0/681634938 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23dc0319d0 con 0x7f23e4071be0 2024-05-09T05:17:49.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.540+0000 7f23e9c29640 1 -- 172.21.15.1:0/681634938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e4071be0 msgr2=0x7f23e40c48c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.540+0000 7f23e9c29640 1 --2- 172.21.15.1:0/681634938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e4071be0 0x7f23e40c48c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f23dc009e50 tx=0x7f23dc02a710 comp rx=0 tx=0).stop 2024-05-09T05:17:49.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.540+0000 7f23e9c29640 1 -- 172.21.15.1:0/681634938 shutdown_connections 2024-05-09T05:17:49.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.540+0000 7f23e9c29640 1 --2- 172.21.15.1:0/681634938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e4071be0 0x7f23e40c48c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.540+0000 7f23e9c29640 1 --2- 172.21.15.1:0/681634938 >> v2:172.21.15.1:3300/0 conn(0x7f23e41569c0 0x7f23e40716a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.540+0000 7f23e9c29640 1 -- 172.21.15.1:0/681634938 >> 172.21.15.1:0/681634938 conn(0x7f23e40747b0 msgr2=0x7f23e40ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:49.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.540+0000 7f23e9c29640 1 -- 172.21.15.1:0/681634938 shutdown_connections 2024-05-09T05:17:49.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.540+0000 7f23e9c29640 1 -- 172.21.15.1:0/681634938 wait complete. 2024-05-09T05:17:49.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.543+0000 7f23e9c29640 1 Processor -- start 2024-05-09T05:17:49.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.543+0000 7f23e9c29640 1 -- start start 2024-05-09T05:17:49.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e9c29640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e4071be0 0x7f23e41f2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e9c29640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23e41569c0 0x7f23e40cdef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e9c29640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e415d130 0x7f23e414bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e9c29640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f23e406e1a0 con 0x7f23e41569c0 2024-05-09T05:17:49.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e9c29640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23e406e780 con 0x7f23e4071be0 2024-05-09T05:17:49.543 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e9c29640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23e40cac90 con 0x7f23e415d130 2024-05-09T05:17:49.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23e41569c0 0x7f23e40cdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23e41569c0 0x7f23e40cdef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:56396/0 (socket says 172.21.15.1:56396) 2024-05-09T05:17:49.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.545+0000 7f23e2ffd640 1 -- 172.21.15.1:0/2413798059 learned_addr learned my addr 172.21.15.1:0/2413798059 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:49.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.546+0000 7f23e37fe640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e4071be0 0x7f23e41f2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.546+0000 7f23e3fff640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e415d130 0x7f23e414bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.546+0000 7f23e2ffd640 1 -- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e415d130 msgr2=0x7f23e414bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.546+0000 7f23e2ffd640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e415d130 0x7f23e414bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.546+0000 7f23e2ffd640 1 -- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e4071be0 msgr2=0x7f23e41f2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.546+0000 7f23e2ffd640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23e4071be0 0x7f23e41f2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.546+0000 7f23e2ffd640 1 -- 172.21.15.1:0/2413798059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e40ce430 con 0x7f23e41569c0 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.546+0000 7f23e2ffd640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23e41569c0 0x7f23e40cdef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f23d800d5f0 tx=0x7f23d800dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.547+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d80125f0 con 0x7f23e41569c0 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.547+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f23d8019070 con 0x7f23e41569c0 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.547+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d8012990 con 0x7f23e41569c0 2024-05-09T05:17:49.545 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.547+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e40c8290 con 0x7f23e41569c0 2024-05-09T05:17:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.547+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e40c8850 con 0x7f23e41569c0 2024-05-09T05:17:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.549+0000 7f23e9c29640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e40c5660 0x7f23e40c5c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.549+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23e40cd830 con 0x7f23e40c5660 2024-05-09T05:17:49.546 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.549+0000 7f23e37fe640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e40c5660 0x7f23e40c5c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.549+0000 7f23e37fe640 1 -- 172.21.15.1:0/2413798059 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f23e40c8000 con 0x7f23e40c5660 2024-05-09T05:17:49.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.549+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f23d8012b30 con 0x7f23e41569c0 2024-05-09T05:17:49.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.549+0000 7f23e37fe640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e40c5660 0x7f23e40c5c50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f23d4001310 tx=0x7f23d40017d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23cbfff640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23d0076320 0x7f23d0078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f23d8097a40 con 0x7f23e41569c0 2024-05-09T05:17:49.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23e3fff640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23d0076320 0x7f23d0078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4019070 con 0x7f23e40c5660 2024-05-09T05:17:49.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f23d40115b0 con 0x7f23e40c5660 2024-05-09T05:17:49.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e40c5660 msgr2=0x7f23e40c5c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23cbfff640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e40c5660 0x7f23e40c5c50 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f23d4001310 tx=0x7f23d40017d0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e40c5660 msgr2=0x7f23e40c5c50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:49.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.550+0000 7f23cbfff640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e40c5660 0x7f23e40c5c50 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.549 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.551+0000 7f23e3fff640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23d0076320 0x7f23d0078830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f23dc033000 tx=0x7f23dc009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.562+0000 7f23e9c29640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e414c270 0x7f23e40c6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.562+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23e4050870 con 0x7f23e414c270 2024-05-09T05:17:49.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.563+0000 7f23e37fe640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e414c270 0x7f23e40c6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.564+0000 7f23e37fe640 1 -- 172.21.15.1:0/2413798059 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f23e415a120 con 0x7f23e414c270 2024-05-09T05:17:49.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.565+0000 7f23e37fe640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e414c270 0x7f23e40c6190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f23d4016000 tx=0x7f23d4000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.565+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23d4019040 con 0x7f23e414c270 2024-05-09T05:17:49.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.565+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f23d401c8c0 con 0x7f23e414c270 2024-05-09T05:17:49.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.565+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e414c270 msgr2=0x7f23e40c6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.565+0000 7f23cbfff640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e414c270 0x7f23e40c6190 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f23d4016000 tx=0x7f23d4000ed0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.563 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.565+0000 7f23cbfff640 1 -- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e414c270 msgr2=0x7f23e40c6190 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.565+0000 7f23cbfff640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23e414c270 0x7f23e40c6190 secure :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f23d4016000 tx=0x7f23d4000ed0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 88, 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 90860, 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:17:49.564 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 14, 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.565 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.566 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:17:49.567 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:17:49.568 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.569 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:17:49.570 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.571 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:17:49.572 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.573 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:17:49.574 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:17:49.575 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:17:49.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23d0076320 msgr2=0x7f23d0078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e9c29640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f23d0076320 0x7f23d0078830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f23dc033000 tx=0x7f23dc009000 comp rx=0 tx=0).stop 2024-05-09T05:17:49.576 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23e41569c0 msgr2=0x7f23e40cdef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e9c29640 1 --2- 172.21.15.1:0/2413798059 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23e41569c0 0x7f23e40cdef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f23d800d5f0 tx=0x7f23d800dab0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e3fff640 1 -- 172.21.15.1:0/2413798059 reap_dead start 2024-05-09T05:17:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e3fff640 1 -- 172.21.15.1:0/2413798059 reap_dead start 2024-05-09T05:17:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 shutdown_connections 2024-05-09T05:17:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 >> 172.21.15.1:0/2413798059 conn(0x7f23e40747b0 msgr2=0x7f23e415a340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.566+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 shutdown_connections 2024-05-09T05:17:49.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.567+0000 7f23e9c29640 1 -- 172.21.15.1:0/2413798059 wait complete. 2024-05-09T05:17:49.583 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:17:49.583 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-05-09T05:17:49.583 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:17:49.583 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-05-09T05:17:49.583 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:17:49.583 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:17:49.583 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:17:49.596 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:17:49.596 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:17:49.596 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:17:49.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.780+0000 7f8f26c96640 1 Processor -- start 2024-05-09T05:17:49.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.780+0000 7f8f26c96640 1 -- start start 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f26c96640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8f200c81d0 0x7f8f2015e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f26c96640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f200c85e0 0x7f8f2015e780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f26c96640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8f20154cc0 con 0x7f8f201548b0 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f26c96640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8f200c9680 con 0x7f8f200c81d0 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f26c96640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8f200c98d0 con 0x7f8f200c85e0 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f17fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8f200c81d0 0x7f8f2015e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f25233640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f200c85e0 msgr2=0x7f8f2015e780 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f25233640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f200c85e0 0x7f8f2015e780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:49.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f17fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8f200c81d0 0x7f8f2015e240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33638/0 (socket says 172.21.15.1:33638) 2024-05-09T05:17:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f17fff640 1 -- 172.21.15.1:0/352789838 learned_addr learned my addr 172.21.15.1:0/352789838 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.782+0000 7f8f24a32640 1 --1- 172.21.15.1:0/352789838 >> v1:172.21.15.196:6789/0 conn(0x7f8f201548b0 0x7f8f200c2be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:42298/0 (socket says 172.21.15.1:42298) 2024-05-09T05:17:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.783+0000 7f8f17fff640 1 -- 172.21.15.1:0/352789838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f200c85e0 msgr2=0x7f8f2015e780 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.783+0000 7f8f17fff640 1 --2- 172.21.15.1:0/352789838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f200c85e0 0x7f8f2015e780 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.783+0000 7f8f17fff640 1 -- 172.21.15.1:0/352789838 >> v1:172.21.15.196:6789/0 conn(0x7f8f201548b0 legacy=0x7f8f200c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.783+0000 7f8f177fe640 1 -- 172.21.15.1:0/352789838 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3795929282 0 0) 0x7f8f20154cc0 con 0x7f8f201548b0 2024-05-09T05:17:49.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.783+0000 7f8f17fff640 1 -- 172.21.15.1:0/352789838 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f200c9bb0 con 0x7f8f200c81d0 2024-05-09T05:17:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.784+0000 7f8f17fff640 1 --2- 172.21.15.1:0/352789838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c81d0 0x7f8f2015e240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009870 tx=0x7f8f1c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67e163cf5d3b79a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.784+0000 7f8f177fe640 1 -- 172.21.15.1:0/352789838 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c037070 con 0x7f8f200c81d0 2024-05-09T05:17:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.784+0000 7f8f177fe640 1 -- 172.21.15.1:0/352789838 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f1c032440 con 0x7f8f200c81d0 2024-05-09T05:17:49.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.784+0000 7f8f177fe640 1 -- 172.21.15.1:0/352789838 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c032810 con 0x7f8f200c81d0 2024-05-09T05:17:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.785+0000 7f8f26c96640 1 -- 172.21.15.1:0/352789838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c81d0 msgr2=0x7f8f2015e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.785+0000 7f8f26c96640 1 --2- 172.21.15.1:0/352789838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c81d0 0x7f8f2015e240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c009870 tx=0x7f8f1c02a270 comp rx=0 tx=0).stop 2024-05-09T05:17:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.785+0000 7f8f26c96640 1 -- 172.21.15.1:0/352789838 shutdown_connections 2024-05-09T05:17:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.785+0000 7f8f26c96640 1 --2- 172.21.15.1:0/352789838 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f200c85e0 0x7f8f2015e780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.785+0000 7f8f26c96640 1 --2- 172.21.15.1:0/352789838 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c81d0 0x7f8f2015e240 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.785+0000 7f8f26c96640 1 -- 172.21.15.1:0/352789838 >> 172.21.15.1:0/352789838 conn(0x7f8f20070dd0 msgr2=0x7f8f200711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.785+0000 7f8f26c96640 1 -- 172.21.15.1:0/352789838 shutdown_connections 2024-05-09T05:17:49.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.785+0000 7f8f26c96640 1 -- 172.21.15.1:0/352789838 wait complete. 2024-05-09T05:17:49.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.788+0000 7f8f26c96640 1 Processor -- start 2024-05-09T05:17:49.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.788+0000 7f8f26c96640 1 -- start start 2024-05-09T05:17:49.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.789+0000 7f8f26c96640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f200c81d0 0x7f8f200ccc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.789+0000 7f8f26c96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c85e0 0x7f8f201562e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.789+0000 7f8f26c96640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f201548b0 0x7f8f20157040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.789+0000 7f8f26c96640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8f200ca6f0 con 0x7f8f200c85e0 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.789+0000 7f8f26c96640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8f2015dd20 con 0x7f8f200c81d0 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.789+0000 7f8f17fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c85e0 0x7f8f201562e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f24a32640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f200c81d0 0x7f8f200ccc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f17fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c85e0 0x7f8f201562e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33646/0 (socket says 172.21.15.1:33646) 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f24a32640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f200c81d0 0x7f8f200ccc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:53798/0 (socket says 172.21.15.1:53798) 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8f200ca870 con 0x7f8f201548b0 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f17fff640 1 -- 172.21.15.1:0/538616896 learned_addr learned my addr 172.21.15.1:0/538616896 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f25233640 1 -- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f201548b0 msgr2=0x7f8f20157040 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:17:49.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f25233640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f201548b0 0x7f8f20157040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:49.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f17fff640 1 -- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f201548b0 msgr2=0x7f8f20157040 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:49.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f17fff640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f201548b0 0x7f8f20157040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f17fff640 1 -- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f200c81d0 msgr2=0x7f8f200ccc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f17fff640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f200c81d0 0x7f8f200ccc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f17fff640 1 -- 172.21.15.1:0/538616896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f200cd1c0 con 0x7f8f200c85e0 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f24a32640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f200c81d0 0x7f8f200ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.790+0000 7f8f17fff640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c85e0 0x7f8f201562e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c02a860 tx=0x7f8f1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.791+0000 7f8f157fa640 1 -- 172.21.15.1:0/538616896 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c041070 con 0x7f8f200c85e0 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.791+0000 7f8f157fa640 1 -- 172.21.15.1:0/538616896 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8f1c03c930 con 0x7f8f200c85e0 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.791+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f201f75b0 con 0x7f8f200c85e0 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.791+0000 7f8f157fa640 1 -- 172.21.15.1:0/538616896 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f1c037040 con 0x7f8f200c85e0 2024-05-09T05:17:49.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.791+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f200cc480 con 0x7f8f200c85e0 2024-05-09T05:17:49.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.792+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8f200696d0 con 0x7f8f200c85e0 2024-05-09T05:17:49.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.793+0000 7f8f157fa640 1 -- 172.21.15.1:0/538616896 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8f1c044070 con 0x7f8f200c85e0 2024-05-09T05:17:49.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.794+0000 7f8f157fa640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8f0c076300 0x7f8f0c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:49.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.794+0000 7f8f157fa640 1 -- 172.21.15.1:0/538616896 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8f1c0b7c10 con 0x7f8f200c85e0 2024-05-09T05:17:49.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.794+0000 7f8f24a32640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8f0c076300 0x7f8f0c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:49.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.796+0000 7f8f24a32640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8f0c076300 0x7f8f0c078810 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8f10000f60 tx=0x7f8f10007e60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:49.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:49.801+0000 7f8f157fa640 1 -- 172.21.15.1:0/538616896 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8f1c04b070 con 0x7f8f200c85e0 2024-05-09T05:17:50.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.271+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8f200c7190 con 0x7f8f200c85e0 2024-05-09T05:17:50.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.272+0000 7f8f157fa640 1 -- 172.21.15.1:0/538616896 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8f200c7190 con 0x7f8f200c85e0 2024-05-09T05:17:50.271 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:17:50.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8f0c076300 msgr2=0x7f8f0c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:50.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8f0c076300 0x7f8f0c078810 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f8f10000f60 tx=0x7f8f10007e60 comp rx=0 tx=0).stop 2024-05-09T05:17:50.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c85e0 msgr2=0x7f8f201562e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c85e0 0x7f8f201562e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8f1c02a860 tx=0x7f8f1c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 shutdown_connections 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8f0c076300 0x7f8f0c078810 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8f201548b0 0x7f8f20157040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8f200c85e0 0x7f8f201562e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 --2- 172.21.15.1:0/538616896 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8f200c81d0 0x7f8f200ccc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 >> 172.21.15.1:0/538616896 conn(0x7f8f20070dd0 msgr2=0x7f8f20150060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 shutdown_connections 2024-05-09T05:17:50.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:50.275+0000 7f8f26c96640 1 -- 172.21.15.1:0/538616896 wait complete. 2024-05-09T05:17:53.293 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:17:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.483+0000 7fb1bce49640 1 Processor -- start 2024-05-09T05:17:53.482 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.483+0000 7fb1bce49640 1 -- start start 2024-05-09T05:17:53.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1bce49640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 0x7fb1b8073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:53.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1bce49640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb1b8073ba0 0x7fb1b80c78b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:53.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1bce49640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1b80c8f30 con 0x7fb1b80c8a60 2024-05-09T05:17:53.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1bce49640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb1b80c5710 con 0x7fb1b8073ba0 2024-05-09T05:17:53.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1b5d9b640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 msgr2=0x7fb1b8073660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:17:53.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1bce49640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b80c59f0 con 0x7fb1b806da90 2024-05-09T05:17:53.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1b659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb1b80c8a60 0x7fb1b814f690 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38466/0 (socket says 172.21.15.1:38466) 2024-05-09T05:17:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1b659c640 1 -- 172.21.15.1:0/416569314 learned_addr learned my addr 172.21.15.1:0/416569314 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:17:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1b5d9b640 1 --2- 172.21.15.1:0/416569314 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 0x7fb1b8073660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.485+0000 7fb1b6d9d640 1 --2- 172.21.15.1:0/416569314 >> v2:172.21.15.1:3300/0 conn(0x7fb1b8073ba0 0x7fb1b80c78b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.486+0000 7fb1b559a640 1 -- 172.21.15.1:0/416569314 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 34452746 0 0) 0x7fb1b80c8f30 con 0x7fb1b80c8a60 2024-05-09T05:17:53.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.486+0000 7fb1b559a640 1 -- 172.21.15.1:0/416569314 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb1a0003520 con 0x7fb1b80c8a60 2024-05-09T05:17:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.486+0000 7fb1b6d9d640 1 -- 172.21.15.1:0/416569314 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 msgr2=0x7fb1b8073660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:17:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.486+0000 7fb1b6d9d640 1 --2- 172.21.15.1:0/416569314 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 0x7fb1b8073660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.486+0000 7fb1b6d9d640 1 -- 172.21.15.1:0/416569314 >> v1:172.21.15.1:6789/0 conn(0x7fb1b80c8a60 legacy=0x7fb1b814f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.486+0000 7fb1b6d9d640 1 -- 172.21.15.1:0/416569314 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b80c5cd0 con 0x7fb1b8073ba0 2024-05-09T05:17:53.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.487+0000 7fb1b6d9d640 1 --2- 172.21.15.1:0/416569314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b8073ba0 0x7fb1b80c78b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009d90 tx=0x7fb1b002a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3706762436538fcb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:53.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.488+0000 7fb1b559a640 1 -- 172.21.15.1:0/416569314 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0007740 con 0x7fb1b8073ba0 2024-05-09T05:17:53.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.488+0000 7fb1b559a640 1 -- 172.21.15.1:0/416569314 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b00078e0 con 0x7fb1b8073ba0 2024-05-09T05:17:53.487 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.488+0000 7fb1b559a640 1 -- 172.21.15.1:0/416569314 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0007cb0 con 0x7fb1b8073ba0 2024-05-09T05:17:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.489+0000 7fb1bce49640 1 -- 172.21.15.1:0/416569314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b8073ba0 msgr2=0x7fb1b80c78b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.489+0000 7fb1bce49640 1 --2- 172.21.15.1:0/416569314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b8073ba0 0x7fb1b80c78b0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009d90 tx=0x7fb1b002a680 comp rx=0 tx=0).stop 2024-05-09T05:17:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.489+0000 7fb1bce49640 1 -- 172.21.15.1:0/416569314 shutdown_connections 2024-05-09T05:17:53.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.489+0000 7fb1bce49640 1 --2- 172.21.15.1:0/416569314 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 0x7fb1b8073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.489+0000 7fb1bce49640 1 --2- 172.21.15.1:0/416569314 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b8073ba0 0x7fb1b80c78b0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.490+0000 7fb1bce49640 1 -- 172.21.15.1:0/416569314 >> 172.21.15.1:0/416569314 conn(0x7fb1b80ce0e0 msgr2=0x7fb1b80ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:53.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.490+0000 7fb1bce49640 1 -- 172.21.15.1:0/416569314 shutdown_connections 2024-05-09T05:17:53.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.490+0000 7fb1bce49640 1 -- 172.21.15.1:0/416569314 wait complete. 2024-05-09T05:17:53.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.492+0000 7fb1bce49640 1 Processor -- start 2024-05-09T05:17:53.491 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.492+0000 7fb1bce49640 1 -- start start 2024-05-09T05:17:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.493+0000 7fb1bce49640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 0x7fb1b814d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.493+0000 7fb1bce49640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1b8073ba0 0x7fb1b8139ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1bce49640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b80c8a60 0x7fb1b812d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:53.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b659c640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 msgr2=0x7fb1b814d660 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b659c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 0x7fb1b814d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1bce49640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b80ca930 con 0x7fb1b80c8a60 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1bce49640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb1b80ca7b0 con 0x7fb1b8073ba0 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1bce49640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb1b806dea0 con 0x7fb1b806da90 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b80c8a60 0x7fb1b812d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b5d9b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1b8073ba0 0x7fb1b8139ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b80c8a60 0x7fb1b812d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33666/0 (socket says 172.21.15.1:33666) 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b6d9d640 1 -- 172.21.15.1:0/3027684424 learned_addr learned my addr 172.21.15.1:0/3027684424 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b6d9d640 1 -- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 msgr2=0x7fb1b814d660 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b6d9d640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 0x7fb1b814d660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b6d9d640 1 -- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1b8073ba0 msgr2=0x7fb1b8139ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b6d9d640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1b8073ba0 0x7fb1b8139ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b6d9d640 1 -- 172.21.15.1:0/3027684424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1b813a1e0 con 0x7fb1b80c8a60 2024-05-09T05:17:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.494+0000 7fb1b5d9b640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1b8073ba0 0x7fb1b8139ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:17:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.495+0000 7fb1b6d9d640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b80c8a60 0x7fb1b812d690 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009eb0 tx=0x7fb1b0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.495+0000 7fb1a6ffd640 1 -- 172.21.15.1:0/3027684424 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b003f070 con 0x7fb1b80c8a60 2024-05-09T05:17:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.495+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1b813ae70 con 0x7fb1b80c8a60 2024-05-09T05:17:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.495+0000 7fb1a6ffd640 1 -- 172.21.15.1:0/3027684424 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1b0031b40 con 0x7fb1b80c8a60 2024-05-09T05:17:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.495+0000 7fb1a6ffd640 1 -- 172.21.15.1:0/3027684424 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1b0043410 con 0x7fb1b80c8a60 2024-05-09T05:17:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.495+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1b813b3b0 con 0x7fb1b80c8a60 2024-05-09T05:17:53.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.497+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb1b8050870 con 0x7fb1b80c8a60 2024-05-09T05:17:53.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.497+0000 7fb1a6ffd640 1 -- 172.21.15.1:0/3027684424 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb1b00435b0 con 0x7fb1b80c8a60 2024-05-09T05:17:53.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.498+0000 7fb1a6ffd640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb198076280 0x7fb198078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:53.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.498+0000 7fb1a6ffd640 1 -- 172.21.15.1:0/3027684424 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb1b00b88f0 con 0x7fb1b80c8a60 2024-05-09T05:17:53.497 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.498+0000 7fb1b659c640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb198076280 0x7fb198078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:53.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.500+0000 7fb1b659c640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb198076280 0x7fb198078790 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8004810 tx=0x7fb1a8007a50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:53.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.506+0000 7fb1a6ffd640 1 -- 172.21.15.1:0/3027684424 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb1b00821d0 con 0x7fb1b80c8a60 2024-05-09T05:17:53.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.982+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fb1b81653f0 con 0x7fb1b80c8a60 2024-05-09T05:17:53.981 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.983+0000 7fb1a6ffd640 1 -- 172.21.15.1:0/3027684424 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fb1b0081ad0 con 0x7fb1b80c8a60 2024-05-09T05:17:53.982 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":18,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:17:53.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb198076280 msgr2=0x7fb198078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:53.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb198076280 0x7fb198078790 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fb1a8004810 tx=0x7fb1a8007a50 comp rx=0 tx=0).stop 2024-05-09T05:17:53.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b80c8a60 msgr2=0x7fb1b812d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:53.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b80c8a60 0x7fb1b812d690 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb1b0009eb0 tx=0x7fb1b0004a40 comp rx=0 tx=0).stop 2024-05-09T05:17:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 shutdown_connections 2024-05-09T05:17:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb198076280 0x7fb198078790 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1b8073ba0 0x7fb1b8139ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1b80c8a60 0x7fb1b812d690 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 --2- 172.21.15.1:0/3027684424 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1b806da90 0x7fb1b814d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 >> 172.21.15.1:0/3027684424 conn(0x7fb1b80ce0e0 msgr2=0x7fb1b806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 shutdown_connections 2024-05-09T05:17:53.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:53.986+0000 7fb1bce49640 1 -- 172.21.15.1:0/3027684424 wait complete. 2024-05-09T05:17:57.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:17:57.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.195+0000 7f5688d0d640 1 Processor -- start 2024-05-09T05:17:57.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.195+0000 7f5688d0d640 1 -- start start 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.197+0000 7f5688d0d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f568415ae40 0x7f56840c9140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.197+0000 7f568259c640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f56840713a0 0x7f56840c6e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:42314/0 (socket says 172.21.15.1:42314) 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.197+0000 7f5688d0d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56840c7f50 con 0x7f56840c9680 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.197+0000 7f5688d0d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f56840c56a0 con 0x7f56840713a0 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.197+0000 7f5688d0d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f56840ca540 con 0x7f568415ae40 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.197+0000 7f568259c640 1 -- 172.21.15.1:0/949440562 learned_addr learned my addr 172.21.15.1:0/949440562 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.197+0000 7f5681d9b640 1 --2- 172.21.15.1:0/949440562 >> v2:172.21.15.196:3300/0 conn(0x7f568415ae40 0x7f56840c9140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.198+0000 7f5681d9b640 1 -- 172.21.15.1:0/949440562 >> v1:172.21.15.196:6789/0 conn(0x7f56840713a0 legacy=0x7f56840c6e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:57.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.198+0000 7f5681d9b640 1 -- 172.21.15.1:0/949440562 >> v1:172.21.15.1:6789/0 conn(0x7f56840c9680 legacy=0x7f568416b5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:57.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.198+0000 7f5681d9b640 1 -- 172.21.15.1:0/949440562 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56840c8210 con 0x7f568415ae40 2024-05-09T05:17:57.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.199+0000 7f5681d9b640 1 --2- 172.21.15.1:0/949440562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f56840c9140 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56780097a0 tx=0x7f567802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3651ae818024adc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:57.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.199+0000 7f568159a640 1 -- 172.21.15.1:0/949440562 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678037070 con 0x7f568415ae40 2024-05-09T05:17:57.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.199+0000 7f568159a640 1 -- 172.21.15.1:0/949440562 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5678032440 con 0x7f568415ae40 2024-05-09T05:17:57.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.200+0000 7f568159a640 1 -- 172.21.15.1:0/949440562 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5678037070 con 0x7f568415ae40 2024-05-09T05:17:57.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.200+0000 7f5688d0d640 1 -- 172.21.15.1:0/949440562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 msgr2=0x7f56840c9140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:57.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.201+0000 7f5688d0d640 1 --2- 172.21.15.1:0/949440562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f56840c9140 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f56780097a0 tx=0x7f567802a1a0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.201+0000 7f5688d0d640 1 -- 172.21.15.1:0/949440562 shutdown_connections 2024-05-09T05:17:57.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.201+0000 7f5688d0d640 1 --2- 172.21.15.1:0/949440562 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f56840c9140 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.201+0000 7f5688d0d640 1 -- 172.21.15.1:0/949440562 >> 172.21.15.1:0/949440562 conn(0x7f56840caec0 msgr2=0x7f56840cb2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:57.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.201+0000 7f5688d0d640 1 -- 172.21.15.1:0/949440562 shutdown_connections 2024-05-09T05:17:57.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.201+0000 7f5688d0d640 1 -- 172.21.15.1:0/949440562 wait complete. 2024-05-09T05:17:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.203+0000 7f5688d0d640 1 Processor -- start 2024-05-09T05:17:57.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.203+0000 7f5688d0d640 1 -- start start 2024-05-09T05:17:57.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5688d0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56840713a0 0x7f568415eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:57.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5688d0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f56840c9680 0x7f5684160a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:57.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5688d0d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f5684160f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:57.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5688d0d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5684071df0 con 0x7f56840713a0 2024-05-09T05:17:57.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5688d0d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f56840c9a90 con 0x7f568415ae40 2024-05-09T05:17:57.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5688d0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f56840718c0 con 0x7f56840c9680 2024-05-09T05:17:57.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5682d9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f5684160f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f568259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56840713a0 0x7f568415eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5681d9b640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f56840c9680 msgr2=0x7f5684160a40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5681d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f56840c9680 0x7f5684160a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5682d9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f5684160f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:53828/0 (socket says 172.21.15.1:53828) 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f568259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56840713a0 0x7f568415eb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:33668/0 (socket says 172.21.15.1:33668) 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.205+0000 7f5682d9d640 1 -- 172.21.15.1:0/2049978281 learned_addr learned my addr 172.21.15.1:0/2049978281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.206+0000 7f5682d9d640 1 -- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f56840c9680 msgr2=0x7f5684160a40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.206+0000 7f5682d9d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f56840c9680 0x7f5684160a40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.206+0000 7f5682d9d640 1 -- 172.21.15.1:0/2049978281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56840713a0 msgr2=0x7f568415eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.206+0000 7f5682d9d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56840713a0 0x7f568415eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.206+0000 7f5682d9d640 1 -- 172.21.15.1:0/2049978281 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568415f070 con 0x7f568415ae40 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.206+0000 7f5682d9d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f5684160f80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c730 tx=0x7f567c00cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.207+0000 7f5672ffd640 1 -- 172.21.15.1:0/2049978281 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c016020 con 0x7f568415ae40 2024-05-09T05:17:57.208 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.207+0000 7f5672ffd640 1 -- 172.21.15.1:0/2049978281 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f567c006d70 con 0x7f568415ae40 2024-05-09T05:17:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.207+0000 7f5672ffd640 1 -- 172.21.15.1:0/2049978281 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567c003bd0 con 0x7f568415ae40 2024-05-09T05:17:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.207+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56841ea3b0 con 0x7f568415ae40 2024-05-09T05:17:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.207+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5684154940 con 0x7f568415ae40 2024-05-09T05:17:57.209 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.208+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f56841fb590 con 0x7f568415ae40 2024-05-09T05:17:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.209+0000 7f5672ffd640 1 -- 172.21.15.1:0/2049978281 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f567c0050d0 con 0x7f568415ae40 2024-05-09T05:17:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.210+0000 7f5672ffd640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56640762d0 0x7f56640787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:17:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.210+0000 7f5672ffd640 1 -- 172.21.15.1:0/2049978281 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f567c096ff0 con 0x7f568415ae40 2024-05-09T05:17:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.210+0000 7f568259c640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56640762d0 0x7f56640787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:17:57.210 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.211+0000 7f568259c640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56640762d0 0x7f56640787e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5674000f60 tx=0x7f5674009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:17:57.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.217+0000 7f5672ffd640 1 -- 172.21.15.1:0/2049978281 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f567c020020 con 0x7f568415ae40 2024-05-09T05:17:57.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.685+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5684050870 con 0x7f568415ae40 2024-05-09T05:17:57.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.685+0000 7f5672ffd640 1 -- 172.21.15.1:0/2049978281 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f567c060a40 con 0x7f568415ae40 2024-05-09T05:17:57.684 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":88,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56640762d0 msgr2=0x7f56640787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56640762d0 0x7f56640787e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5674000f60 tx=0x7f5674009040 comp rx=0 tx=0).stop 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 msgr2=0x7f5684160f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f5684160f80 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f567c00c730 tx=0x7f567c00cbf0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 shutdown_connections 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56640762d0 0x7f56640787e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f568415ae40 0x7f5684160f80 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f56840c9680 0x7f5684160a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 --2- 172.21.15.1:0/2049978281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f56840713a0 0x7f568415eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:17:57.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 >> 172.21.15.1:0/2049978281 conn(0x7f56840caec0 msgr2=0x7f568406d220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:17:57.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 shutdown_connections 2024-05-09T05:17:57.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:17:57.688+0000 7f5688d0d640 1 -- 172.21.15.1:0/2049978281 wait complete. 2024-05-09T05:18:00.712 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:18:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.894+0000 7fa171c43640 1 Processor -- start 2024-05-09T05:18:00.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.894+0000 7fa171c43640 1 -- start start 2024-05-09T05:18:00.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.896+0000 7fa171c43640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa16c0d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:00.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.896+0000 7fa16b7fe640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 msgr2=0x7fa16c0d03d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:00.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.896+0000 7fa171c43640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16c06f6d0 con 0x7fa16c06f1b0 2024-05-09T05:18:00.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.896+0000 7fa171c43640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa16c0cc4d0 con 0x7fa16c15ca90 2024-05-09T05:18:00.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.896+0000 7fa171c43640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa16c0cee50 con 0x7fa16c0d16b0 2024-05-09T05:18:00.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.896+0000 7fa16b7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa16c0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:00.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.896+0000 7fa16affd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fa16c06f1b0 0x7fa16c0ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:47244/0 (socket says 172.21.15.1:47244) 2024-05-09T05:18:00.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:00.896+0000 7fa16affd640 1 -- 172.21.15.1:0/2039017967 learned_addr learned my addr 172.21.15.1:0/2039017967 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:01.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:01.097+0000 7fa16b7fe640 1 -- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 msgr2=0x7fa16c0d03d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:01.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:01.097+0000 7fa16b7fe640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa16c0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:18:01.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:01.498+0000 7fa16b7fe640 1 -- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 msgr2=0x7fa16c0d03d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:01.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:01.498+0000 7fa16b7fe640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa16c0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:18:02.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:02.299+0000 7fa16b7fe640 1 -- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 msgr2=0x7fa16c0d03d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:02.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:02.299+0000 7fa16b7fe640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa16c0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:18:03.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.815+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 729264228 0 0) 0x7fa16c0cc4d0 con 0x7fa16c15ca90 2024-05-09T05:18:03.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.815+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa154003520 con 0x7fa16c15ca90 2024-05-09T05:18:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.826+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3269665408 0 0) 0x7fa16c06f6d0 con 0x7fa16c06f1b0 2024-05-09T05:18:03.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.826+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fa16c0cc4d0 con 0x7fa16c06f1b0 2024-05-09T05:18:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 4087764513 0 0) 0x7fa154003520 con 0x7fa16c15ca90 2024-05-09T05:18:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 msgr2=0x7fa16c0d03d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa16c0d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:03.829 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 >> v1:172.21.15.1:6789/0 conn(0x7fa16c06f1b0 legacy=0x7fa16c0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c0cc7b0 con 0x7fa16c15ca90 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fa164001a50 con 0x7fa16c15ca90 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 >> v1:172.21.15.196:6789/0 conn(0x7fa16c15ca90 legacy=0x7fa16c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154003b40 0x7fa154003f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa154004800 0x7fa154004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa15400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c0cee50 con 0x7fa154003b40 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa154005500 con 0x7fa154004800 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.831+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa15400bf00 con 0x7fa16c0d16b0 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16bfff640 1 -- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 msgr2=0x7fa15400b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16bfff640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa15400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16affd640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa154004800 0x7fa154004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:03.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16b7fe640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154003b40 0x7fa154003f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16affd640 1 -- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 msgr2=0x7fa15400b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16affd640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c0d16b0 0x7fa15400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16affd640 1 -- 172.21.15.1:0/2039017967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154003b40 msgr2=0x7fa154003f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16affd640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154003b40 0x7fa154003f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16affd640 1 -- 172.21.15.1:0/2039017967 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c0cc7b0 con 0x7fa154004800 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.832+0000 7fa16b7fe640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154003b40 0x7fa154003f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.833+0000 7fa16affd640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa154004800 0x7fa154004c10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa16c15a370 tx=0x7fa154005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25ec55d4663cf57a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.833+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1600014e0 con 0x7fa154004800 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.833+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa16002e070 con 0x7fa154004800 2024-05-09T05:18:03.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.833+0000 7fa16a7fc640 1 -- 172.21.15.1:0/2039017967 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa16002ad60 con 0x7fa154004800 2024-05-09T05:18:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.835+0000 7fa171c43640 1 -- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa154004800 msgr2=0x7fa154004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.835+0000 7fa171c43640 1 --2- 172.21.15.1:0/2039017967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa154004800 0x7fa154004c10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa16c15a370 tx=0x7fa154005830 comp rx=0 tx=0).stop 2024-05-09T05:18:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.835+0000 7fa16bfff640 1 -- 172.21.15.1:0/2039017967 reap_dead start 2024-05-09T05:18:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.835+0000 7fa171c43640 1 -- 172.21.15.1:0/2039017967 shutdown_connections 2024-05-09T05:18:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.835+0000 7fa171c43640 1 -- 172.21.15.1:0/2039017967 >> 172.21.15.1:0/2039017967 conn(0x7fa16c0b3660 msgr2=0x7fa16c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:03.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.835+0000 7fa171c43640 1 -- 172.21.15.1:0/2039017967 shutdown_connections 2024-05-09T05:18:03.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.835+0000 7fa171c43640 1 -- 172.21.15.1:0/2039017967 wait complete. 2024-05-09T05:18:03.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.838+0000 7fa171c43640 1 Processor -- start 2024-05-09T05:18:03.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.838+0000 7fa171c43640 1 -- start start 2024-05-09T05:18:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa171c43640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154004800 0x7fa16c0c0500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa171c43640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c1f62b0 0x7fa16c154120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa171c43640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa16c151d60 0x7fa16c152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa171c43640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c0ccd70 con 0x7fa154004800 2024-05-09T05:18:03.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa16affd640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c1f62b0 msgr2=0x7fa16c154120 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa16affd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c1f62b0 0x7fa16c154120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa171c43640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa16c0cd0e0 con 0x7fa16c151d60 2024-05-09T05:18:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa171c43640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa16c0d03d0 con 0x7fa16c1f62b0 2024-05-09T05:18:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa16b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154004800 0x7fa16c0c0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa16bfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa16c151d60 0x7fa16c152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa16b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154004800 0x7fa16c0c0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42942/0 (socket says 172.21.15.1:42942) 2024-05-09T05:18:03.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.840+0000 7fa16b7fe640 1 -- 172.21.15.1:0/3398214194 learned_addr learned my addr 172.21.15.1:0/3398214194 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.841+0000 7fa16bfff640 1 -- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c1f62b0 msgr2=0x7fa16c154120 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.841+0000 7fa16bfff640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c1f62b0 0x7fa16c154120 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.841+0000 7fa16bfff640 1 -- 172.21.15.1:0/3398214194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154004800 msgr2=0x7fa16c0c0500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.841+0000 7fa16bfff640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154004800 0x7fa16c0c0500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.841+0000 7fa16bfff640 1 -- 172.21.15.1:0/3398214194 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa16c1f97e0 con 0x7fa16c151d60 2024-05-09T05:18:03.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.841+0000 7fa16b7fe640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154004800 0x7fa16c0c0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.841+0000 7fa16bfff640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa16c151d60 0x7fa16c152170 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0cff30 tx=0x7fa16400cb90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.841+0000 7fa153fff640 1 -- 172.21.15.1:0/3398214194 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa164017070 con 0x7fa16c151d60 2024-05-09T05:18:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.842+0000 7fa153fff640 1 -- 172.21.15.1:0/3398214194 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa164005d60 con 0x7fa16c151d60 2024-05-09T05:18:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.842+0000 7fa153fff640 1 -- 172.21.15.1:0/3398214194 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa15c009a30 con 0x7fa16c151d60 2024-05-09T05:18:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.842+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa16c1f9ac0 con 0x7fa16c151d60 2024-05-09T05:18:03.841 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.842+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa16c1fa080 con 0x7fa16c151d60 2024-05-09T05:18:03.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.844+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa16c050870 con 0x7fa16c151d60 2024-05-09T05:18:03.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.844+0000 7fa153fff640 1 -- 172.21.15.1:0/3398214194 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa16400a850 con 0x7fa16c151d60 2024-05-09T05:18:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.845+0000 7fa153fff640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa158076280 0x7fa158078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.845+0000 7fa153fff640 1 -- 172.21.15.1:0/3398214194 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa164096940 con 0x7fa16c151d60 2024-05-09T05:18:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.846+0000 7fa16b7fe640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa158076280 0x7fa158078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:03.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.846+0000 7fa16b7fe640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa158076280 0x7fa158078790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa15c003c20 tx=0x7fa15c010040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:03.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:03.852+0000 7fa153fff640 1 -- 172.21.15.1:0/3398214194 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fa164060180 con 0x7fa16c151d60 2024-05-09T05:18:04.333 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.335+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fa16c0c43e0 con 0x7fa16c151d60 2024-05-09T05:18:04.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.336+0000 7fa153fff640 1 -- 172.21.15.1:0/3398214194 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7fa164068170 con 0x7fa16c151d60 2024-05-09T05:18:04.334 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":92,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:18:04.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.339+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa158076280 msgr2=0x7fa158078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.339+0000 7fa171c43640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa158076280 0x7fa158078790 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa15c003c20 tx=0x7fa15c010040 comp rx=0 tx=0).stop 2024-05-09T05:18:04.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.339+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa16c151d60 msgr2=0x7fa16c152170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.339+0000 7fa171c43640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa16c151d60 0x7fa16c152170 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa16c0cff30 tx=0x7fa16400cb90 comp rx=0 tx=0).stop 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.340+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 shutdown_connections 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.340+0000 7fa171c43640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa158076280 0x7fa158078790 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.340+0000 7fa171c43640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa16c151d60 0x7fa16c152170 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.340+0000 7fa171c43640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa16c1f62b0 0x7fa16c154120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.340+0000 7fa171c43640 1 --2- 172.21.15.1:0/3398214194 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa154004800 0x7fa16c0c0500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.340+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 >> 172.21.15.1:0/3398214194 conn(0x7fa16c0b3660 msgr2=0x7fa16c15b340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.340+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 shutdown_connections 2024-05-09T05:18:04.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.340+0000 7fa171c43640 1 -- 172.21.15.1:0/3398214194 wait complete. 2024-05-09T05:18:04.356 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:18:04.357 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:18:04.357 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:18:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.538+0000 7f4e1fa19640 1 Processor -- start 2024-05-09T05:18:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.538+0000 7f4e1fa19640 1 -- start start 2024-05-09T05:18:04.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1fa19640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4e180736b0 0x7f4e180c7810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1fa19640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f4e180689e0 0x7f4e180c7d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1fa19640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4e180cce20 con 0x7f4e180cc970 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1fa19640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f4e180cd100 con 0x7f4e180736b0 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1fa19640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f4e180cd3e0 con 0x7f4e180689e0 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1cfb4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4e180736b0 0x7f4e180c7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1cfb4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f4e180736b0 0x7f4e180c7810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:42950/0 (socket says 172.21.15.1:42950) 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1cfb4640 1 -- 172.21.15.1:0/4212789378 learned_addr learned my addr 172.21.15.1:0/4212789378 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.540+0000 7f4e1dfb6640 1 --2- 172.21.15.1:0/4212789378 >> v2:172.21.15.196:3300/0 conn(0x7f4e180689e0 0x7f4e180c7d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.541+0000 7f4e1cfb4640 1 -- 172.21.15.1:0/4212789378 >> v2:172.21.15.196:3300/0 conn(0x7f4e180689e0 msgr2=0x7f4e180c7d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.541+0000 7f4e1cfb4640 1 --2- 172.21.15.1:0/4212789378 >> v2:172.21.15.196:3300/0 conn(0x7f4e180689e0 0x7f4e180c7d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.539 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.541+0000 7f4e1cfb4640 1 -- 172.21.15.1:0/4212789378 >> v1:172.21.15.1:6789/0 conn(0x7f4e180cc970 legacy=0x7f4e1815b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.541+0000 7f4e1cfb4640 1 -- 172.21.15.1:0/4212789378 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e1815a8c0 con 0x7f4e180736b0 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.542+0000 7f4e1cfb4640 1 --2- 172.21.15.1:0/4212789378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180736b0 0x7f4e180c7810 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4e1806edd0 tx=0x7f4e0402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b91170e16be2828 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.542+0000 7f4e17fff640 1 -- 172.21.15.1:0/4212789378 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04037070 con 0x7f4e180736b0 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.542+0000 7f4e17fff640 1 -- 172.21.15.1:0/4212789378 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e04032440 con 0x7f4e180736b0 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.542+0000 7f4e17fff640 1 -- 172.21.15.1:0/4212789378 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04032810 con 0x7f4e180736b0 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.543+0000 7f4e1fa19640 1 -- 172.21.15.1:0/4212789378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180736b0 msgr2=0x7f4e180c7810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.543+0000 7f4e1fa19640 1 --2- 172.21.15.1:0/4212789378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180736b0 0x7f4e180c7810 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4e1806edd0 tx=0x7f4e0402a270 comp rx=0 tx=0).stop 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.543+0000 7f4e1fa19640 1 -- 172.21.15.1:0/4212789378 shutdown_connections 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.543+0000 7f4e1fa19640 1 --2- 172.21.15.1:0/4212789378 >> v2:172.21.15.196:3300/0 conn(0x7f4e180689e0 0x7f4e180c7d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.543+0000 7f4e1fa19640 1 --2- 172.21.15.1:0/4212789378 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180736b0 0x7f4e180c7810 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.543+0000 7f4e1fa19640 1 -- 172.21.15.1:0/4212789378 >> 172.21.15.1:0/4212789378 conn(0x7f4e180ce180 msgr2=0x7f4e180ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:04.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.543+0000 7f4e1fa19640 1 -- 172.21.15.1:0/4212789378 shutdown_connections 2024-05-09T05:18:04.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.543+0000 7f4e1fa19640 1 -- 172.21.15.1:0/4212789378 wait complete. 2024-05-09T05:18:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.546+0000 7f4e1fa19640 1 Processor -- start 2024-05-09T05:18:04.544 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.546+0000 7f4e1fa19640 1 -- start start 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1fa19640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e180689e0 0x7f4e18140d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1fa19640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4e180736b0 0x7f4e180c5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1fa19640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180cc970 0x7f4e180c5e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1cfb4640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4e180736b0 msgr2=0x7f4e180c5480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1cfb4640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4e180736b0 0x7f4e180c5480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1d7b5640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e180689e0 0x7f4e18140d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1d7b5640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e180689e0 0x7f4e18140d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:34972/0 (socket says 172.21.15.1:34972) 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1d7b5640 1 -- 172.21.15.1:0/2654804670 learned_addr learned my addr 172.21.15.1:0/2654804670 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4e1806c9c0 con 0x7f4e180cc970 2024-05-09T05:18:04.547 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4e18154d20 con 0x7f4e180689e0 2024-05-09T05:18:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.548+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4e1806cb40 con 0x7f4e180736b0 2024-05-09T05:18:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.549+0000 7f4e1dfb6640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180cc970 0x7f4e180c5e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.549+0000 7f4e1d7b5640 1 -- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4e180736b0 msgr2=0x7f4e180c5480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.549+0000 7f4e1d7b5640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4e180736b0 0x7f4e180c5480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.549+0000 7f4e1d7b5640 1 -- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180cc970 msgr2=0x7f4e180c5e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.549+0000 7f4e1d7b5640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180cc970 0x7f4e180c5e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.549+0000 7f4e1d7b5640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e180ca100 con 0x7f4e180689e0 2024-05-09T05:18:04.548 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.549+0000 7f4e1d7b5640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e180689e0 0x7f4e18140d50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00c760 tx=0x7f4e0c00cc20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.550+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c003b70 con 0x7f4e180689e0 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.550+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4e0c003d30 con 0x7f4e180689e0 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.550+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e0c00aa90 con 0x7f4e180689e0 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.550+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e18141290 con 0x7f4e180689e0 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.550+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e180ca6c0 con 0x7f4e180689e0 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.551+0000 7f4e1fa19640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180c9970 0x7f4e180c6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.551+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4e18141760 con 0x7f4e180c9970 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.551+0000 7f4e1cfb4640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180c9970 0x7f4e180c6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.552+0000 7f4e1cfb4640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4e18050870 con 0x7f4e180c9970 2024-05-09T05:18:04.550 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.552+0000 7f4e1cfb4640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180c9970 0x7f4e180c6380 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4e180ce150 tx=0x7f4e040046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.552+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04040030 con 0x7f4e180c9970 2024-05-09T05:18:04.551 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.552+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4e0c00ac30 con 0x7f4e180689e0 2024-05-09T05:18:04.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.553+0000 7f4e15ffb640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4dfc0761d0 0x7f4dfc0786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.553+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4e0c0974f0 con 0x7f4e180689e0 2024-05-09T05:18:04.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.554+0000 7f4e1dfb6640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4dfc0761d0 0x7f4dfc0786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.554+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f4e04038040 con 0x7f4e180c9970 2024-05-09T05:18:04.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.554+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180c9970 msgr2=0x7f4e180c6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.554+0000 7f4e15ffb640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180c9970 0x7f4e180c6380 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4e180ce150 tx=0x7f4e040046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.554+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180c9970 msgr2=0x7f4e180c6380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:04.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.554+0000 7f4e15ffb640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e180c9970 0x7f4e180c6380 secure :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f4e180ce150 tx=0x7f4e040046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.553 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.555+0000 7f4e1dfb6640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4dfc0761d0 0x7f4dfc0786e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4e18073660 tx=0x7f4e10009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.567+0000 7f4e1fa19640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e18140470 0x7f4e180c68c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.567+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4e181408c0 con 0x7f4e18140470 2024-05-09T05:18:04.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.567+0000 7f4e1cfb4640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e18140470 0x7f4e180c68c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.567+0000 7f4e1cfb4640 1 -- 172.21.15.1:0/2654804670 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4e1806c6f0 con 0x7f4e18140470 2024-05-09T05:18:04.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.568+0000 7f4e1cfb4640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e18140470 0x7f4e180c68c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000c00 tx=0x7f4e04009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.568+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e04037070 con 0x7f4e18140470 2024-05-09T05:18:04.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.568+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5052 (secure 0 0 0) 0x7f4e04003ae0 con 0x7f4e18140470 2024-05-09T05:18:04.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.568+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e18140470 msgr2=0x7f4e180c68c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.568+0000 7f4e15ffb640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e18140470 0x7f4e180c68c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f4e04000c00 tx=0x7f4e04009240 comp rx=0 tx=0).stop 2024-05-09T05:18:04.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.568+0000 7f4e15ffb640 1 -- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e18140470 msgr2=0x7f4e180c68c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:04.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.568+0000 7f4e15ffb640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4e18140470 0x7f4e180c68c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.567 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:04.567 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 92, 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 286763, 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:04.568 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:04.569 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:04.570 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:04.571 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:04.572 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:04.572 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:04.574 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:04.575 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.576 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.577 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:04.578 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:04.579 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.580 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.581 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:18:04.582 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.583 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:04.584 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:04.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4dfc0761d0 msgr2=0x7f4dfc0786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1fa19640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4dfc0761d0 0x7f4dfc0786e0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f4e18073660 tx=0x7f4e10009190 comp rx=0 tx=0).stop 2024-05-09T05:18:04.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e180689e0 msgr2=0x7f4e18140d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.586 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1fa19640 1 --2- 172.21.15.1:0/2654804670 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4e180689e0 0x7f4e18140d50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f4e0c00c760 tx=0x7f4e0c00cc20 comp rx=0 tx=0).stop 2024-05-09T05:18:04.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1dfb6640 1 -- 172.21.15.1:0/2654804670 reap_dead start 2024-05-09T05:18:04.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1dfb6640 1 -- 172.21.15.1:0/2654804670 reap_dead start 2024-05-09T05:18:04.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 shutdown_connections 2024-05-09T05:18:04.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 >> 172.21.15.1:0/2654804670 conn(0x7f4e180ce180 msgr2=0x7f4e180744d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:04.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.569+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 shutdown_connections 2024-05-09T05:18:04.587 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.570+0000 7f4e1fa19640 1 -- 172.21.15.1:0/2654804670 wait complete. 2024-05-09T05:18:04.587 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:18:04.587 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:18:04.588 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:18:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.767+0000 7fa497cb1640 1 Processor -- start 2024-05-09T05:18:04.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.767+0000 7fa497cb1640 1 -- start start 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.768+0000 7fa497cb1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c89a0 0x7fa4900ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.768+0000 7fa497cb1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa4900c46e0 0x7fa490159650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa497cb1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4900c4af0 0x7fa490159b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa495a4d640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c89a0 msgr2=0x7fa4900ca930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa495a4d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c89a0 0x7fa4900ca930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa497cb1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa4900c9290 con 0x7fa4900c4af0 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa497cb1640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fa4900c3be0 con 0x7fa4900c46e0 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa497cb1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa49006e890 con 0x7fa4900c89a0 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49524c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa4900c46e0 0x7fa490159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49624e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa4900c4af0 0x7fa490159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.767 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49524c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa4900c46e0 0x7fa490159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:34974/0 (socket says 172.21.15.1:34974) 2024-05-09T05:18:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49524c640 1 -- 172.21.15.1:0/1700814527 learned_addr learned my addr 172.21.15.1:0/1700814527 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49624e640 1 -- 172.21.15.1:0/1700814527 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c89a0 msgr2=0x7fa4900ca930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49624e640 1 --2- 172.21.15.1:0/1700814527 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c89a0 0x7fa4900ca930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49624e640 1 -- 172.21.15.1:0/1700814527 >> v2:172.21.15.196:3300/0 conn(0x7fa4900c46e0 msgr2=0x7fa490159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49624e640 1 --2- 172.21.15.1:0/1700814527 >> v2:172.21.15.196:3300/0 conn(0x7fa4900c46e0 0x7fa490159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.769+0000 7fa49624e640 1 -- 172.21.15.1:0/1700814527 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4900c3e60 con 0x7fa4900c4af0 2024-05-09T05:18:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.770+0000 7fa49624e640 1 --2- 172.21.15.1:0/1700814527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4900c4af0 0x7fa490159b90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa490074dd0 tx=0x7fa48802a5f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46665b103a30d16f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.770+0000 7fa494a4b640 1 -- 172.21.15.1:0/1700814527 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488007740 con 0x7fa4900c4af0 2024-05-09T05:18:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.770+0000 7fa494a4b640 1 -- 172.21.15.1:0/1700814527 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa4880078e0 con 0x7fa4900c4af0 2024-05-09T05:18:04.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.770+0000 7fa494a4b640 1 -- 172.21.15.1:0/1700814527 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa488007cb0 con 0x7fa4900c4af0 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.771+0000 7fa497cb1640 1 -- 172.21.15.1:0/1700814527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4900c4af0 msgr2=0x7fa490159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.771+0000 7fa497cb1640 1 --2- 172.21.15.1:0/1700814527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4900c4af0 0x7fa490159b90 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa490074dd0 tx=0x7fa48802a5f0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.772+0000 7fa497cb1640 1 -- 172.21.15.1:0/1700814527 shutdown_connections 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.772+0000 7fa497cb1640 1 --2- 172.21.15.1:0/1700814527 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4900c4af0 0x7fa490159b90 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.772+0000 7fa497cb1640 1 --2- 172.21.15.1:0/1700814527 >> v2:172.21.15.196:3300/0 conn(0x7fa4900c46e0 0x7fa490159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.772+0000 7fa497cb1640 1 --2- 172.21.15.1:0/1700814527 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c89a0 0x7fa4900ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.772+0000 7fa497cb1640 1 -- 172.21.15.1:0/1700814527 >> 172.21.15.1:0/1700814527 conn(0x7fa4900747b0 msgr2=0x7fa4900ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.772+0000 7fa497cb1640 1 -- 172.21.15.1:0/1700814527 shutdown_connections 2024-05-09T05:18:04.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.772+0000 7fa497cb1640 1 -- 172.21.15.1:0/1700814527 wait complete. 2024-05-09T05:18:04.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.774+0000 7fa497cb1640 1 Processor -- start 2024-05-09T05:18:04.772 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.774+0000 7fa497cb1640 1 -- start start 2024-05-09T05:18:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa497cb1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4900c46e0 0x7fa4901f3970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa497cb1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c4af0 0x7fa4901586f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa497cb1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4900c89a0 0x7fa490155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa497cb1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900c97c0 con 0x7fa4900c89a0 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa497cb1640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa4900c94e0 con 0x7fa4900c46e0 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa495a4d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4900c46e0 0x7fa4901f3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa497cb1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa4900cae70 con 0x7fa4900c4af0 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa495a4d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4900c46e0 0x7fa4901f3970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:34988/0 (socket says 172.21.15.1:34988) 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa495a4d640 1 -- 172.21.15.1:0/191885252 learned_addr learned my addr 172.21.15.1:0/191885252 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa49524c640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c4af0 msgr2=0x7fa4901586f0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa49524c640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c4af0 0x7fa4901586f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa495a4d640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c4af0 msgr2=0x7fa4901586f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.776+0000 7fa495a4d640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa4900c4af0 0x7fa4901586f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.777+0000 7fa495a4d640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4900c89a0 msgr2=0x7fa490155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.777+0000 7fa495a4d640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa4900c89a0 0x7fa490155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.777+0000 7fa495a4d640 1 -- 172.21.15.1:0/191885252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa4901f3eb0 con 0x7fa4900c46e0 2024-05-09T05:18:04.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.777+0000 7fa495a4d640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4900c46e0 0x7fa4901f3970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00b580 tx=0x7fa48c00ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.778+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c013020 con 0x7fa4900c46e0 2024-05-09T05:18:04.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.778+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa48c003de0 con 0x7fa4900c46e0 2024-05-09T05:18:04.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.778+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa490154be0 con 0x7fa4900c46e0 2024-05-09T05:18:04.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.778+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48c0052b0 con 0x7fa4900c46e0 2024-05-09T05:18:04.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.778+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4901554e0 con 0x7fa4900c46e0 2024-05-09T05:18:04.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.780+0000 7fa497cb1640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490154510 0x7fa490155d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.780+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa4901f4380 con 0x7fa490154510 2024-05-09T05:18:04.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.780+0000 7fa49524c640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490154510 0x7fa490155d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.780+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa48c016020 con 0x7fa4900c46e0 2024-05-09T05:18:04.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.780+0000 7fa49524c640 1 -- 172.21.15.1:0/191885252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa490165f00 con 0x7fa490154510 2024-05-09T05:18:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa49524c640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490154510 0x7fa490155d90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa49015a780 tx=0x7fa48000e950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa4867fc640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4740762d0 0x7fa4740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa48c097390 con 0x7fa4900c46e0 2024-05-09T05:18:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa480017030 con 0x7fa490154510 2024-05-09T05:18:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa49624e640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4740762d0 0x7fa4740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fa480004020 con 0x7fa490154510 2024-05-09T05:18:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490154510 msgr2=0x7fa490155d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa4867fc640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490154510 0x7fa490155d90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fa49015a780 tx=0x7fa48000e950 comp rx=0 tx=0).stop 2024-05-09T05:18:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490154510 msgr2=0x7fa490155d90 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.781+0000 7fa4867fc640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490154510 0x7fa490155d90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.780 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.782+0000 7fa49624e640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4740762d0 0x7fa4740787e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa488034000 tx=0x7fa48802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.795+0000 7fa497cb1640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490153fc0 0x7fa4901562d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:04.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.795+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa490050870 con 0x7fa490153fc0 2024-05-09T05:18:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.796+0000 7fa49524c640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490153fc0 0x7fa4901562d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.796+0000 7fa49524c640 1 -- 172.21.15.1:0/191885252 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fa4900ca360 con 0x7fa490153fc0 2024-05-09T05:18:04.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.796+0000 7fa49524c640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490153fc0 0x7fa4901562d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa49015a780 tx=0x7fa480004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.797+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa48000f040 con 0x7fa490153fc0 2024-05-09T05:18:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.797+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7fa480003960 con 0x7fa490153fc0 2024-05-09T05:18:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.797+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490153fc0 msgr2=0x7fa4901562d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.797+0000 7fa4867fc640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490153fc0 0x7fa4901562d0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa49015a780 tx=0x7fa480004730 comp rx=0 tx=0).stop 2024-05-09T05:18:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.797+0000 7fa4867fc640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490153fc0 msgr2=0x7fa4901562d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:04.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.797+0000 7fa4867fc640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa490153fc0 0x7fa4901562d0 secure :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fa49015a780 tx=0x7fa480004730 comp rx=0 tx=0).stop 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 92, 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 35201, 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:04.800 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.801 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:04.802 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:04.803 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:04.804 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.805 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:04.806 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:04.807 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4740762d0 msgr2=0x7fa4740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa4740762d0 0x7fa4740787e0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa488034000 tx=0x7fa48802e000 comp rx=0 tx=0).stop 2024-05-09T05:18:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4900c46e0 msgr2=0x7fa4901f3970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:04.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa49624e640 1 -- 172.21.15.1:0/191885252 reap_dead start 2024-05-09T05:18:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4900c46e0 0x7fa4901f3970 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fa48c00b580 tx=0x7fa48c00ba40 comp rx=0 tx=0).stop 2024-05-09T05:18:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 shutdown_connections 2024-05-09T05:18:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 --2- 172.21.15.1:0/191885252 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa4900c46e0 0x7fa4901f3970 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 >> 172.21.15.1:0/191885252 conn(0x7fa4900747b0 msgr2=0x7fa49006d6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 shutdown_connections 2024-05-09T05:18:04.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:04.798+0000 7fa497cb1640 1 -- 172.21.15.1:0/191885252 wait complete. 2024-05-09T05:18:04.813 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:18:04.813 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:18:04.925 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:18:04.928 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:18:24.814 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:18:24.815 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-05-09T05:18:24.815 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T05:18:24.815 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T05:18:24.818 INFO:tasks.ceph.mon.c:Started 2024-05-09T05:18:24.818 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:18:24.818 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:18:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.005+0000 7f3097cc6640 1 Processor -- start 2024-05-09T05:18:25.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.005+0000 7f3097cc6640 1 -- start start 2024-05-09T05:18:25.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3097cc6640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 0x7f30900c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:25.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3097cc6640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f30900c9810 0x7f309015cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:25.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3097cc6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f30901594f0 con 0x7f30900c87d0 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3097cc6640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f309015a380 con 0x7f30900c9810 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3097cc6640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3090159770 con 0x7f30901538b0 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3095261640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 0x7f30900c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3095a62640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f30900c87d0 0x7f30900c5ca0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:52370/0 (socket says 172.21.15.1:52370) 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3095a62640 1 -- 172.21.15.1:0/3133271329 learned_addr learned my addr 172.21.15.1:0/3133271329 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3096263640 1 --2- 172.21.15.1:0/3133271329 >> v2:172.21.15.196:3300/0 conn(0x7f30900c9810 0x7f309015cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3095261640 1 -- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 msgr2=0x7f30900c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3095261640 1 -- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 msgr2=0x7f30900c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:25.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3095261640 1 --2- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 0x7f30900c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.007+0000 7f3095261640 1 --2- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 0x7f30900c92d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.008+0000 7f3094a60640 1 -- 172.21.15.1:0/3133271329 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3332523173 0 0) 0x7f3080001110 con 0x7f30900c87d0 2024-05-09T05:18:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.008+0000 7f3096263640 1 -- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 msgr2=0x7f30900c92d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.008+0000 7f3096263640 1 --2- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 0x7f30900c92d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.008+0000 7f3096263640 1 -- 172.21.15.1:0/3133271329 >> v1:172.21.15.196:6789/0 conn(0x7f30900c87d0 legacy=0x7f30900c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.008+0000 7f3096263640 1 -- 172.21.15.1:0/3133271329 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3090159a50 con 0x7f30900c9810 2024-05-09T05:18:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.009+0000 7f3096263640 1 --2- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30900c9810 0x7f309015cc90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3088009d90 tx=0x7f308802a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=25e3f1e418ac4212 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:25.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.009+0000 7f3094a60640 1 -- 172.21.15.1:0/3133271329 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088031600 con 0x7f30900c9810 2024-05-09T05:18:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.009+0000 7f3094a60640 1 -- 172.21.15.1:0/3133271329 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f30880317a0 con 0x7f30900c9810 2024-05-09T05:18:25.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.009+0000 7f3094a60640 1 -- 172.21.15.1:0/3133271329 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088031b70 con 0x7f30900c9810 2024-05-09T05:18:25.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.010+0000 7f3097cc6640 1 -- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30900c9810 msgr2=0x7f309015cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.010+0000 7f3097cc6640 1 --2- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30900c9810 0x7f309015cc90 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f3088009d90 tx=0x7f308802a5d0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.010+0000 7f3097cc6640 1 -- 172.21.15.1:0/3133271329 shutdown_connections 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.010+0000 7f3097cc6640 1 --2- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30900c9810 0x7f309015cc90 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.010+0000 7f3097cc6640 1 --2- 172.21.15.1:0/3133271329 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30901538b0 0x7f30900c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.010+0000 7f3097cc6640 1 -- 172.21.15.1:0/3133271329 >> 172.21.15.1:0/3133271329 conn(0x7f30900745a0 msgr2=0x7f3090071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.010+0000 7f3097cc6640 1 -- 172.21.15.1:0/3133271329 shutdown_connections 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.010+0000 7f3097cc6640 1 -- 172.21.15.1:0/3133271329 wait complete. 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.013+0000 7f3097cc6640 1 Processor -- start 2024-05-09T05:18:25.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.013+0000 7f3097cc6640 1 -- start start 2024-05-09T05:18:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.015+0000 7f3097cc6640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 0x7f3090145b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.015+0000 7f3097cc6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30900c9810 0x7f309013c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.015+0000 7f3097cc6640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30901538b0 0x7f309013c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.015+0000 7f3097cc6640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f30900c70c0 con 0x7f30900c9810 2024-05-09T05:18:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.015+0000 7f3097cc6640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f309015a010 con 0x7f30901538b0 2024-05-09T05:18:25.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.015+0000 7f3097cc6640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f30900c7240 con 0x7f30900c87d0 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095a62640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 0x7f3090145b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095261640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30900c9810 0x7f309013c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095a62640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 0x7f3090145b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:39438/0 (socket says 172.21.15.1:39438) 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095a62640 1 -- 172.21.15.1:0/1587903377 learned_addr learned my addr 172.21.15.1:0/1587903377 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3096263640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30901538b0 0x7f309013c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095261640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30900c9810 0x7f309013c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45990/0 (socket says 172.21.15.1:45990) 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095a62640 1 -- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 msgr2=0x7f3090145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095a62640 1 -- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 msgr2=0x7f3090145b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095a62640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 0x7f3090145b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3095a62640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 0x7f3090145b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:25.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3096263640 1 -- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 msgr2=0x7f3090145b30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3096263640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 0x7f3090145b30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3096263640 1 -- 172.21.15.1:0/1587903377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30900c9810 msgr2=0x7f309013c2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3096263640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30900c9810 0x7f309013c2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3096263640 1 -- 172.21.15.1:0/1587903377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3090146770 con 0x7f30901538b0 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.016+0000 7f3096263640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30901538b0 0x7f309013c7f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3088031000 tx=0x7f3088004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.017+0000 7f30867fc640 1 -- 172.21.15.1:0/1587903377 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088040070 con 0x7f30901538b0 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.017+0000 7f30867fc640 1 -- 172.21.15.1:0/1587903377 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3088030b60 con 0x7f30901538b0 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.017+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30900765b0 con 0x7f30901538b0 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.017+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3090146d10 con 0x7f30901538b0 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.017+0000 7f30867fc640 1 -- 172.21.15.1:0/1587903377 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3088009450 con 0x7f30901538b0 2024-05-09T05:18:25.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.018+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3090050870 con 0x7f30901538b0 2024-05-09T05:18:25.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.022+0000 7f30867fc640 1 -- 172.21.15.1:0/1587903377 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3088039040 con 0x7f30901538b0 2024-05-09T05:18:25.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.023+0000 7f30867fc640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3074076300 0x7f3074078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:25.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.023+0000 7f30867fc640 1 -- 172.21.15.1:0/1587903377 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f30880b8c70 con 0x7f30901538b0 2024-05-09T05:18:25.020 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.023+0000 7f3095a62640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3074076300 0x7f3074078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:25.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.024+0000 7f3095a62640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3074076300 0x7f3074078810 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3080000f60 tx=0x7f3080007cc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:25.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.029+0000 7f30867fc640 1 -- 172.21.15.1:0/1587903377 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f30880824b0 con 0x7f30901538b0 2024-05-09T05:18:25.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.506+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f30900c55a0 con 0x7f30901538b0 2024-05-09T05:18:25.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.507+0000 7f30867fc640 1 -- 172.21.15.1:0/1587903377 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f3088081db0 con 0x7f30901538b0 2024-05-09T05:18:25.505 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":92,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:18:25.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3074076300 msgr2=0x7f3074078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:25.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3074076300 0x7f3074078810 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3080000f60 tx=0x7f3080007cc0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30901538b0 msgr2=0x7f309013c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:25.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30901538b0 0x7f309013c7f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f3088031000 tx=0x7f3088004a40 comp rx=0 tx=0).stop 2024-05-09T05:18:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 shutdown_connections 2024-05-09T05:18:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3074076300 0x7f3074078810 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f30901538b0 0x7f309013c7f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f30900c9810 0x7f309013c2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 --2- 172.21.15.1:0/1587903377 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f30900c87d0 0x7f3090145b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 >> 172.21.15.1:0/1587903377 conn(0x7f30900745a0 msgr2=0x7f3090150c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 shutdown_connections 2024-05-09T05:18:25.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:25.511+0000 7f3097cc6640 1 -- 172.21.15.1:0/1587903377 wait complete. 2024-05-09T05:18:28.529 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:18:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.720+0000 7fd8a1e61640 1 Processor -- start 2024-05-09T05:18:28.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.720+0000 7fd8a1e61640 1 -- start start 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd8a1e61640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd89c06f1b0 0x7fd89c0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd8a1e61640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd89c15ca90 0x7fd89c15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd8a1e61640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd89c06f6d0 con 0x7fd89c0d16b0 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd8a1e61640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd89c0cd0e0 con 0x7fd89c15ca90 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd8a1e61640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd89c0cc4d0 con 0x7fd89c06f1b0 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd89b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd89c0d16b0 0x7fd89c0d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:52384/0 (socket says 172.21.15.1:52384) 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd89b7fe640 1 -- 172.21.15.1:0/775206212 learned_addr learned my addr 172.21.15.1:0/775206212 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd893fff640 1 --2- 172.21.15.1:0/775206212 >> v2:172.21.15.196:3300/0 conn(0x7fd89c06f1b0 0x7fd89c0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:28.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.723+0000 7fd89bfff640 1 --2- 172.21.15.1:0/775206212 >> v2:172.21.15.1:3300/0 conn(0x7fd89c15ca90 0x7fd89c15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.724+0000 7fd89affd640 1 -- 172.21.15.1:0/775206212 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2143592335 0 0) 0x7fd89c06f6d0 con 0x7fd89c0d16b0 2024-05-09T05:18:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.724+0000 7fd89affd640 1 -- 172.21.15.1:0/775206212 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd888003520 con 0x7fd89c0d16b0 2024-05-09T05:18:28.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.724+0000 7fd893fff640 1 -- 172.21.15.1:0/775206212 >> v2:172.21.15.1:3300/0 conn(0x7fd89c15ca90 msgr2=0x7fd89c15cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.724+0000 7fd893fff640 1 --2- 172.21.15.1:0/775206212 >> v2:172.21.15.1:3300/0 conn(0x7fd89c15ca90 0x7fd89c15cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.724+0000 7fd893fff640 1 -- 172.21.15.1:0/775206212 >> v1:172.21.15.196:6789/0 conn(0x7fd89c0d16b0 legacy=0x7fd89c0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.724+0000 7fd893fff640 1 -- 172.21.15.1:0/775206212 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c0cc7b0 con 0x7fd89c06f1b0 2024-05-09T05:18:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.724+0000 7fd89bfff640 1 --2- 172.21.15.1:0/775206212 >> v2:172.21.15.1:3300/0 conn(0x7fd89c15ca90 0x7fd89c15cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.725+0000 7fd893fff640 1 --2- 172.21.15.1:0/775206212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c06f1b0 0x7fd89c0ce740 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd89c06d0d0 tx=0x7fd88402a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a75d1c0e0647fee9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.725+0000 7fd89affd640 1 -- 172.21.15.1:0/775206212 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884037070 con 0x7fd89c06f1b0 2024-05-09T05:18:28.723 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.725+0000 7fd89affd640 1 -- 172.21.15.1:0/775206212 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd884032440 con 0x7fd89c06f1b0 2024-05-09T05:18:28.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.725+0000 7fd89affd640 1 -- 172.21.15.1:0/775206212 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884032810 con 0x7fd89c06f1b0 2024-05-09T05:18:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.727+0000 7fd8a1e61640 1 -- 172.21.15.1:0/775206212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c06f1b0 msgr2=0x7fd89c0ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.727+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/775206212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c06f1b0 0x7fd89c0ce740 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fd89c06d0d0 tx=0x7fd88402a230 comp rx=0 tx=0).stop 2024-05-09T05:18:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.727+0000 7fd8a1e61640 1 -- 172.21.15.1:0/775206212 shutdown_connections 2024-05-09T05:18:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.727+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/775206212 >> v2:172.21.15.1:3300/0 conn(0x7fd89c15ca90 0x7fd89c15cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.727+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/775206212 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c06f1b0 0x7fd89c0ce740 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.727+0000 7fd8a1e61640 1 -- 172.21.15.1:0/775206212 >> 172.21.15.1:0/775206212 conn(0x7fd89c0b3660 msgr2=0x7fd89c0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.727+0000 7fd8a1e61640 1 -- 172.21.15.1:0/775206212 shutdown_connections 2024-05-09T05:18:28.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.727+0000 7fd8a1e61640 1 -- 172.21.15.1:0/775206212 wait complete. 2024-05-09T05:18:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.730+0000 7fd8a1e61640 1 Processor -- start 2024-05-09T05:18:28.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.730+0000 7fd8a1e61640 1 -- start start 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd8a1e61640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 0x7fd89c158980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd8a1e61640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd89c0d16b0 0x7fd89c1f2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd8a1e61640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c15ca90 0x7fd89c0caaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd8a1e61640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c15d3e0 con 0x7fd89c0d16b0 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd8a1e61640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c0ccd70 con 0x7fd89c15ca90 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd8a1e61640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd89c0cec80 con 0x7fd89c06f1b0 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd893fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd89c0d16b0 0x7fd89c1f2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd89b7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 0x7fd89c158980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd893fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd89c0d16b0 0x7fd89c1f2f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46010/0 (socket says 172.21.15.1:46010) 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd893fff640 1 -- 172.21.15.1:0/2163325052 learned_addr learned my addr 172.21.15.1:0/2163325052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd89bfff640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c15ca90 0x7fd89c0caaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:28.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd89b7fe640 1 -- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 msgr2=0x7fd89c158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd89b7fe640 1 -- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 msgr2=0x7fd89c158980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd89b7fe640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 0x7fd89c158980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.732+0000 7fd89b7fe640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 0x7fd89c158980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd893fff640 1 -- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 msgr2=0x7fd89c158980 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd893fff640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 0x7fd89c158980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd893fff640 1 -- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c15ca90 msgr2=0x7fd89c0caaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd893fff640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c15ca90 0x7fd89c0caaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd893fff640 1 -- 172.21.15.1:0/2163325052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c0cb030 con 0x7fd89c0d16b0 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd893fff640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd89c0d16b0 0x7fd89c1f2f80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd884009890 tx=0x7fd8840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd898ff9640 1 -- 172.21.15.1:0/2163325052 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd884037050 con 0x7fd89c0d16b0 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd898ff9640 1 -- 172.21.15.1:0/2163325052 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd884043070 con 0x7fd89c0d16b0 2024-05-09T05:18:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.733+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89c1575e0 con 0x7fd89c0d16b0 2024-05-09T05:18:28.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.735+0000 7fd898ff9640 1 -- 172.21.15.1:0/2163325052 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd88403ca50 con 0x7fd89c0d16b0 2024-05-09T05:18:28.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.735+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd89c0ca2f0 con 0x7fd89c0d16b0 2024-05-09T05:18:28.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.737+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd89c050870 con 0x7fd89c0d16b0 2024-05-09T05:18:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.738+0000 7fd898ff9640 1 -- 172.21.15.1:0/2163325052 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd88403cbf0 con 0x7fd89c0d16b0 2024-05-09T05:18:28.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.739+0000 7fd898ff9640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd87c0762d0 0x7fd87c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:28.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.739+0000 7fd898ff9640 1 -- 172.21.15.1:0/2163325052 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd8840b7a10 con 0x7fd89c0d16b0 2024-05-09T05:18:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.739+0000 7fd89b7fe640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd87c0762d0 0x7fd87c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.740+0000 7fd89b7fe640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd87c0762d0 0x7fd87c0787e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd88c007a60 tx=0x7fd88c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:28.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:28.747+0000 7fd898ff9640 1 -- 172.21.15.1:0/2163325052 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd8840c0020 con 0x7fd89c0d16b0 2024-05-09T05:18:29.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.233+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd89c157300 con 0x7fd89c0d16b0 2024-05-09T05:18:29.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.234+0000 7fd898ff9640 1 -- 172.21.15.1:0/2163325052 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1560 (secure 0 0 0) 0x7fd884081380 con 0x7fd89c0d16b0 2024-05-09T05:18:29.232 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd87c0762d0 msgr2=0x7fd87c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd87c0762d0 0x7fd87c0787e0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fd88c007a60 tx=0x7fd88c009040 comp rx=0 tx=0).stop 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd89c0d16b0 msgr2=0x7fd89c1f2f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd89c0d16b0 0x7fd89c1f2f80 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fd884009890 tx=0x7fd8840046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 shutdown_connections 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd87c0762d0 0x7fd87c0787e0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd89c15ca90 0x7fd89c0caaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd89c0d16b0 0x7fd89c1f2f80 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 --2- 172.21.15.1:0/2163325052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd89c06f1b0 0x7fd89c158980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 >> 172.21.15.1:0/2163325052 conn(0x7fd89c0b3660 msgr2=0x7fd89c15bf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:29.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 shutdown_connections 2024-05-09T05:18:29.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.238+0000 7fd8a1e61640 1 -- 172.21.15.1:0/2163325052 wait complete. 2024-05-09T05:18:29.257 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:18:29.257 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:18:29.257 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:18:29.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.454+0000 7fb1a97f3640 1 Processor -- start 2024-05-09T05:18:29.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.454+0000 7fb1a97f3640 1 -- start start 2024-05-09T05:18:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.456+0000 7fb1a97f3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb1a40b4ec0 0x7fb1a40cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.456+0000 7fb1a97f3640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a415c7b0 con 0x7fb1a40ce010 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.456+0000 7fb1a97f3640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb1a40cca50 con 0x7fb1a4160460 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.456+0000 7fb1a97f3640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb1a415d0c0 con 0x7fb1a40b4ec0 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.456+0000 7fb1a27fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb1a40ce010 0x7fb1a415fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:60858/0 (socket says 172.21.15.1:60858) 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.456+0000 7fb1a27fc640 1 -- 172.21.15.1:0/2550186531 learned_addr learned my addr 172.21.15.1:0/2550186531 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.456+0000 7fb1a2ffd640 1 --2- 172.21.15.1:0/2550186531 >> v2:172.21.15.196:3300/0 conn(0x7fb1a40b4ec0 0x7fb1a40cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.457+0000 7fb1a1ffb640 1 -- 172.21.15.1:0/2550186531 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3439066038 0 0) 0x7fb1a40cca50 con 0x7fb1a4160460 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.457+0000 7fb1a1ffb640 1 -- 172.21.15.1:0/2550186531 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb18c003520 con 0x7fb1a4160460 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.457+0000 7fb1a1ffb640 1 -- 172.21.15.1:0/2550186531 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1590450783 0 0) 0x7fb1a415c7b0 con 0x7fb1a40ce010 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.457+0000 7fb1a1ffb640 1 -- 172.21.15.1:0/2550186531 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb18c003880 con 0x7fb1a40ce010 2024-05-09T05:18:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.457+0000 7fb1a2ffd640 1 -- 172.21.15.1:0/2550186531 >> v1:172.21.15.196:6789/0 conn(0x7fb1a4160460 legacy=0x7fb1a4167130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.457+0000 7fb1a2ffd640 1 -- 172.21.15.1:0/2550186531 >> v1:172.21.15.1:6789/0 conn(0x7fb1a40ce010 legacy=0x7fb1a415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.457+0000 7fb1a2ffd640 1 -- 172.21.15.1:0/2550186531 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a40ccd30 con 0x7fb1a40b4ec0 2024-05-09T05:18:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.459+0000 7fb1a2ffd640 1 --2- 172.21.15.1:0/2550186531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40b4ec0 0x7fb1a40cf9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1940097a0 tx=0x7fb19402a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4e8f47be0759b23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.459+0000 7fb1a1ffb640 1 -- 172.21.15.1:0/2550186531 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194037070 con 0x7fb1a40b4ec0 2024-05-09T05:18:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.459+0000 7fb1a1ffb640 1 -- 172.21.15.1:0/2550186531 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb194032640 con 0x7fb1a40b4ec0 2024-05-09T05:18:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.459+0000 7fb1a1ffb640 1 -- 172.21.15.1:0/2550186531 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194032a10 con 0x7fb1a40b4ec0 2024-05-09T05:18:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.460+0000 7fb1a97f3640 1 -- 172.21.15.1:0/2550186531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40b4ec0 msgr2=0x7fb1a40cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.460+0000 7fb1a97f3640 1 --2- 172.21.15.1:0/2550186531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40b4ec0 0x7fb1a40cf9d0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb1940097a0 tx=0x7fb19402a2c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.460+0000 7fb1a97f3640 1 -- 172.21.15.1:0/2550186531 shutdown_connections 2024-05-09T05:18:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.460+0000 7fb1a97f3640 1 --2- 172.21.15.1:0/2550186531 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40b4ec0 0x7fb1a40cf9d0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.460+0000 7fb1a97f3640 1 -- 172.21.15.1:0/2550186531 >> 172.21.15.1:0/2550186531 conn(0x7fb1a40d2360 msgr2=0x7fb1a40ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.460+0000 7fb1a97f3640 1 -- 172.21.15.1:0/2550186531 shutdown_connections 2024-05-09T05:18:29.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.460+0000 7fb1a97f3640 1 -- 172.21.15.1:0/2550186531 wait complete. 2024-05-09T05:18:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.464+0000 7fb1a97f3640 1 Processor -- start 2024-05-09T05:18:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.464+0000 7fb1a97f3640 1 -- start start 2024-05-09T05:18:29.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.466+0000 7fb1a97f3640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a40b4ec0 0x7fb1a4147150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.466+0000 7fb1a97f3640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40ce010 0x7fb1a415bc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.466+0000 7fb1a97f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1a4160460 0x7fb1a415c150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.467+0000 7fb1a97f3640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a40cc750 con 0x7fb1a40b4ec0 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.467+0000 7fb1a27fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40ce010 0x7fb1a415bc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.467+0000 7fb1a27fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40ce010 0x7fb1a415bc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:49662/0 (socket says 172.21.15.1:49662) 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.467+0000 7fb1a27fc640 1 -- 172.21.15.1:0/32394111 learned_addr learned my addr 172.21.15.1:0/32394111 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.467+0000 7fb1a2ffd640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a40b4ec0 0x7fb1a4147150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.467+0000 7fb1a37fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1a4160460 0x7fb1a415c150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.467+0000 7fb1a97f3640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a415cd50 con 0x7fb1a40ce010 2024-05-09T05:18:29.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.467+0000 7fb1a97f3640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb1a40cff10 con 0x7fb1a4160460 2024-05-09T05:18:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a37fe640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1a4160460 msgr2=0x7fb1a415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:18:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a37fe640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1a4160460 msgr2=0x7fb1a415c150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:18:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a37fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1a4160460 0x7fb1a415c150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a37fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1a4160460 0x7fb1a415c150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a27fc640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1a4160460 msgr2=0x7fb1a415c150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a27fc640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb1a4160460 0x7fb1a415c150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a27fc640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a40b4ec0 msgr2=0x7fb1a4147150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a27fc640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a40b4ec0 0x7fb1a4147150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a27fc640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1a4154920 con 0x7fb1a40ce010 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.468+0000 7fb1a27fc640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40ce010 0x7fb1a415bc10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb19800c400 tx=0x7fb19800c8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.469+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1980074e0 con 0x7fb1a40ce010 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.469+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb1980076c0 con 0x7fb1a40ce010 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.469+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb198004a20 con 0x7fb1a40ce010 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.469+0000 7fb1a97f3640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1a415d970 con 0x7fb1a40ce010 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.469+0000 7fb1a97f3640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1a41f6800 con 0x7fb1a40ce010 2024-05-09T05:18:29.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.471+0000 7fb1a97f3640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f6a00 0x7fb1a406d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.471+0000 7fb1a97f3640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a41f3750 con 0x7fb1a41f6a00 2024-05-09T05:18:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.471+0000 7fb1a2ffd640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f6a00 0x7fb1a406d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.471+0000 7fb1a2ffd640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb1a41543d0 con 0x7fb1a41f6a00 2024-05-09T05:18:29.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.472+0000 7fb1a2ffd640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f6a00 0x7fb1a406d410 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40c8b40 tx=0x7fb1940046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.470 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.472+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb198004bc0 con 0x7fb1a40ce010 2024-05-09T05:18:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.473+0000 7fb18b7fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb190076300 0x7fb190078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.473+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb1980970e0 con 0x7fb1a40ce010 2024-05-09T05:18:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.474+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194042070 con 0x7fb1a41f6a00 2024-05-09T05:18:29.471 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.474+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb19403d470 con 0x7fb1a41f6a00 2024-05-09T05:18:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.474+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f6a00 msgr2=0x7fb1a406d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.474+0000 7fb1a37fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb190076300 0x7fb190078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.475+0000 7fb18b7fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f6a00 0x7fb1a406d410 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40c8b40 tx=0x7fb1940046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.475+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f6a00 msgr2=0x7fb1a406d410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:29.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.475+0000 7fb18b7fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f6a00 0x7fb1a406d410 secure :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40c8b40 tx=0x7fb1940046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.476+0000 7fb1a37fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb190076300 0x7fb190078810 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb1a415a3a0 tx=0x7fb19c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.486+0000 7fb1a97f3640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f72e0 0x7fb1a406d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.483 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.486+0000 7fb1a97f3640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb1a4050870 con 0x7fb1a41f72e0 2024-05-09T05:18:29.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.486+0000 7fb1a2ffd640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f72e0 0x7fb1a406d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.484 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.487+0000 7fb1a2ffd640 1 -- 172.21.15.1:0/32394111 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb1a4150550 con 0x7fb1a41f72e0 2024-05-09T05:18:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.487+0000 7fb1a2ffd640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f72e0 0x7fb1a406d950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40b44d0 tx=0x7fb194008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.488+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb194042070 con 0x7fb1a41f72e0 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.488+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7fb1940093c0 con 0x7fb1a41f72e0 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.488+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f72e0 msgr2=0x7fb1a406d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.488+0000 7fb18b7fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f72e0 0x7fb1a406d950 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40b44d0 tx=0x7fb194008030 comp rx=0 tx=0).stop 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.488+0000 7fb18b7fe640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f72e0 msgr2=0x7fb1a406d950 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.488+0000 7fb18b7fe640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb1a41f72e0 0x7fb1a406d950 secure :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fb1a40b44d0 tx=0x7fb194008030 comp rx=0 tx=0).stop 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:29.486 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 311683, 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.487 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.488 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:29.489 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:29.490 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.491 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:29.492 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.493 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:29.494 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.495 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:29.496 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:18:29.497 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:18:29.498 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.499 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.499 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.499 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:18:29.499 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.499 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.499 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.499 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:29.499 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:29.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb190076300 msgr2=0x7fb190078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb190076300 0x7fb190078810 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fb1a415a3a0 tx=0x7fb19c009040 comp rx=0 tx=0).stop 2024-05-09T05:18:29.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1a37fe640 1 -- 172.21.15.1:0/32394111 reap_dead start 2024-05-09T05:18:29.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 -- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40ce010 msgr2=0x7fb1a415bc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40ce010 0x7fb1a415bc10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fb19800c400 tx=0x7fb19800c8c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 -- 172.21.15.1:0/32394111 shutdown_connections 2024-05-09T05:18:29.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 --2- 172.21.15.1:0/32394111 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb1a40ce010 0x7fb1a415bc10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 -- 172.21.15.1:0/32394111 >> 172.21.15.1:0/32394111 conn(0x7fb1a40d2360 msgr2=0x7fb1a414ca40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:29.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 -- 172.21.15.1:0/32394111 shutdown_connections 2024-05-09T05:18:29.501 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.489+0000 7fb1897fa640 1 -- 172.21.15.1:0/32394111 wait complete. 2024-05-09T05:18:29.505 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:18:29.505 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:18:29.505 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:18:29.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.691+0000 7f505350a640 1 Processor -- start 2024-05-09T05:18:29.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.691+0000 7f505350a640 1 -- start start 2024-05-09T05:18:29.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.694+0000 7f505350a640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5044009360 0x7f50440e3820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.694+0000 7f505350a640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f5044046db0 0x7f50440471c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.694+0000 7f505350a640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5044047b30 con 0x7f504404be40 2024-05-09T05:18:29.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.694+0000 7f505350a640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f504404d2e0 con 0x7f5044046db0 2024-05-09T05:18:29.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.694+0000 7f505350a640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f504404c3f0 con 0x7f5044009360 2024-05-09T05:18:29.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.694+0000 7f5051d07640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5044009360 0x7f50440e3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.694+0000 7f5051d07640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5044009360 0x7f50440e3820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44628/0 (socket says 172.21.15.1:44628) 2024-05-09T05:18:29.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.694+0000 7f5051d07640 1 -- 172.21.15.1:0/1834427952 learned_addr learned my addr 172.21.15.1:0/1834427952 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:29.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.695+0000 7f5052d09640 1 --2- 172.21.15.1:0/1834427952 >> v2:172.21.15.1:3300/0 conn(0x7f5044046db0 0x7f50440471c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.695+0000 7f5051d07640 1 -- 172.21.15.1:0/1834427952 >> v2:172.21.15.1:3300/0 conn(0x7f5044046db0 msgr2=0x7f50440471c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.695+0000 7f5051d07640 1 --2- 172.21.15.1:0/1834427952 >> v2:172.21.15.1:3300/0 conn(0x7f5044046db0 0x7f50440471c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.695+0000 7f5051d07640 1 -- 172.21.15.1:0/1834427952 >> v1:172.21.15.1:6789/0 conn(0x7f504404be40 legacy=0x7f5044006d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.695+0000 7f5051d07640 1 -- 172.21.15.1:0/1834427952 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f504404c6d0 con 0x7f5044009360 2024-05-09T05:18:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.696+0000 7f5051d07640 1 --2- 172.21.15.1:0/1834427952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044009360 0x7f50440e3820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f50440342a0 tx=0x7f503c02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b8a43451e63fe04a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.696+0000 7f5051506640 1 -- 172.21.15.1:0/1834427952 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c004030 con 0x7f5044009360 2024-05-09T05:18:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.696+0000 7f5051506640 1 -- 172.21.15.1:0/1834427952 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c0041d0 con 0x7f5044009360 2024-05-09T05:18:29.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.696+0000 7f5051506640 1 -- 172.21.15.1:0/1834427952 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c003610 con 0x7f5044009360 2024-05-09T05:18:29.695 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.697+0000 7f505350a640 1 -- 172.21.15.1:0/1834427952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044009360 msgr2=0x7f50440e3820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.697+0000 7f505350a640 1 --2- 172.21.15.1:0/1834427952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044009360 0x7f50440e3820 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f50440342a0 tx=0x7f503c02a1c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.698+0000 7f505350a640 1 -- 172.21.15.1:0/1834427952 shutdown_connections 2024-05-09T05:18:29.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.698+0000 7f505350a640 1 --2- 172.21.15.1:0/1834427952 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044009360 0x7f50440e3820 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.698+0000 7f505350a640 1 --2- 172.21.15.1:0/1834427952 >> v2:172.21.15.1:3300/0 conn(0x7f5044046db0 0x7f50440471c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.698+0000 7f505350a640 1 -- 172.21.15.1:0/1834427952 >> 172.21.15.1:0/1834427952 conn(0x7f50440514c0 msgr2=0x7f50440518d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:29.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.698+0000 7f505350a640 1 -- 172.21.15.1:0/1834427952 shutdown_connections 2024-05-09T05:18:29.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.698+0000 7f505350a640 1 -- 172.21.15.1:0/1834427952 wait complete. 2024-05-09T05:18:29.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.701+0000 7f505350a640 1 Processor -- start 2024-05-09T05:18:29.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.701+0000 7f505350a640 1 -- start start 2024-05-09T05:18:29.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.702+0000 7f505350a640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5044009360 0x7f50440dd630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.702+0000 7f505350a640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044046db0 0x7f50440df550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f505350a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504404be40 0x7f50440d23c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f505350a640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f504404cf70 con 0x7f504404be40 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f505350a640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f504404cc90 con 0x7f5044046db0 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f505350a640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f50440da8f0 con 0x7f5044009360 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044046db0 0x7f50440df550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5052d09640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504404be40 0x7f50440d23c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044046db0 0x7f50440df550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44638/0 (socket says 172.21.15.1:44638) 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 -- 172.21.15.1:0/1332302411 learned_addr learned my addr 172.21.15.1:0/1332302411 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5052508640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5044009360 0x7f50440dd630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5052508640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5044009360 msgr2=0x7f50440dd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5052508640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5044009360 msgr2=0x7f50440dd630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5052508640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5044009360 0x7f50440dd630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5052508640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5044009360 0x7f50440dd630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5044009360 msgr2=0x7f50440dd630 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5044009360 0x7f50440dd630 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504404be40 msgr2=0x7f50440d23c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504404be40 0x7f50440d23c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 -- 172.21.15.1:0/1332302411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50440dfa90 con 0x7f5044046db0 2024-05-09T05:18:29.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5052d09640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f504404be40 0x7f50440d23c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:29.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.703+0000 7f5051d07640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044046db0 0x7f50440df550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f503c009940 tx=0x7f503c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.704+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c041020 con 0x7f5044046db0 2024-05-09T05:18:29.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.704+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f503c006040 con 0x7f5044046db0 2024-05-09T05:18:29.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.704+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f503c004030 con 0x7f5044046db0 2024-05-09T05:18:29.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.704+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f50440dc330 con 0x7f5044046db0 2024-05-09T05:18:29.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.705+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f50440dc5f0 con 0x7f5044046db0 2024-05-09T05:18:29.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.706+0000 7f505350a640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440dd0e0 0x7f50440dc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.706+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f50440058f0 con 0x7f50440dd0e0 2024-05-09T05:18:29.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.706+0000 7f5052508640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440dd0e0 0x7f50440dc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.707+0000 7f5052508640 1 -- 172.21.15.1:0/1332302411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5044051ce0 con 0x7f50440dd0e0 2024-05-09T05:18:29.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.707+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f503c044020 con 0x7f5044046db0 2024-05-09T05:18:29.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.707+0000 7f5052508640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440dd0e0 0x7f50440dc8d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f504800e890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.708+0000 7f5042ffd640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50300762d0 0x7f50300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.708+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f503c0b84b0 con 0x7f5044046db0 2024-05-09T05:18:29.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.708+0000 7f5052d09640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50300762d0 0x7f50300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.709+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048019070 con 0x7f50440dd0e0 2024-05-09T05:18:29.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.709+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f50480052e0 con 0x7f50440dd0e0 2024-05-09T05:18:29.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.709+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440dd0e0 msgr2=0x7f50440dc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.709+0000 7f5042ffd640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440dd0e0 0x7f50440dc8d0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f504800e890 comp rx=0 tx=0).stop 2024-05-09T05:18:29.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.709+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440dd0e0 msgr2=0x7f50440dc8d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:29.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.709+0000 7f5042ffd640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440dd0e0 0x7f50440dc8d0 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f504800e890 comp rx=0 tx=0).stop 2024-05-09T05:18:29.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.709+0000 7f5052d09640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50300762d0 0x7f50300787e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f504c076dc0 tx=0x7f504c08f040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.721+0000 7f505350a640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440d03d0 0x7f50440d07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.721+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5044009010 con 0x7f50440d03d0 2024-05-09T05:18:29.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.721+0000 7f5052508640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440d03d0 0x7f50440d07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.722+0000 7f5052508640 1 -- 172.21.15.1:0/1332302411 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5044047910 con 0x7f50440d03d0 2024-05-09T05:18:29.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.723+0000 7f5052508640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440d03d0 0x7f50440d07e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048001080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.723+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5048019040 con 0x7f50440d03d0 2024-05-09T05:18:29.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4550 (secure 0 0 0) 0x7f50480037f0 con 0x7f50440d03d0 2024-05-09T05:18:29.721 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440d03d0 msgr2=0x7f50440d07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f5042ffd640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440d03d0 0x7f50440d07e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048001080 comp rx=0 tx=0).stop 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f5042ffd640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440d03d0 msgr2=0x7f50440d07e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 60127, 2024-05-09T05:18:29.722 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.723 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.724 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:29.725 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.726 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:29.727 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.728 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.729 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.730 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.731 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.732 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.733 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.734 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.734 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:18:29.734 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.734 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.734 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.734 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:29.734 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f5042ffd640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50440d03d0 0x7f50440d07e0 secure :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f5048007fd0 tx=0x7f5048001080 comp rx=0 tx=0).stop 2024-05-09T05:18:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50300762d0 msgr2=0x7f50300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f505350a640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50300762d0 0x7f50300787e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f504c076dc0 tx=0x7f504c08f040 comp rx=0 tx=0).stop 2024-05-09T05:18:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f5052d09640 1 -- 172.21.15.1:0/1332302411 reap_dead start 2024-05-09T05:18:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044046db0 msgr2=0x7f50440df550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f505350a640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044046db0 0x7f50440df550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f503c009940 tx=0x7f503c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 shutdown_connections 2024-05-09T05:18:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f505350a640 1 --2- 172.21.15.1:0/1332302411 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5044046db0 0x7f50440df550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.724+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 >> 172.21.15.1:0/1332302411 conn(0x7f50440514c0 msgr2=0x7f50441700c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.725+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 shutdown_connections 2024-05-09T05:18:29.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.725+0000 7f505350a640 1 -- 172.21.15.1:0/1332302411 wait complete. 2024-05-09T05:18:29.741 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:18:29.741 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:18:29.742 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:18:29.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.928+0000 7fe051ad1640 1 Processor -- start 2024-05-09T05:18:29.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.928+0000 7fe051ad1640 1 -- start start 2024-05-09T05:18:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.930+0000 7fe051ad1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe04c06c6f0 0x7fe04c06e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe051ad1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fe04c15b090 0x7fe04c160f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe051ad1640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe04c06db10 con 0x7fe04c161460 2024-05-09T05:18:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe051ad1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fe04c165080 con 0x7fe04c15b090 2024-05-09T05:18:29.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe051ad1640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe04c1641b0 con 0x7fe04c06c6f0 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe04b7fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe04c06c6f0 0x7fe04c06e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe04b7fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe04c06c6f0 0x7fe04c06e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44668/0 (socket says 172.21.15.1:44668) 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe04b7fe640 1 -- 172.21.15.1:0/930483330 learned_addr learned my addr 172.21.15.1:0/930483330 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe04affd640 1 --2- 172.21.15.1:0/930483330 >> v2:172.21.15.1:3300/0 conn(0x7fe04c15b090 0x7fe04c160f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe04b7fe640 1 -- 172.21.15.1:0/930483330 >> v2:172.21.15.1:3300/0 conn(0x7fe04c15b090 msgr2=0x7fe04c160f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.931+0000 7fe04b7fe640 1 --2- 172.21.15.1:0/930483330 >> v2:172.21.15.1:3300/0 conn(0x7fe04c15b090 0x7fe04c160f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.932+0000 7fe04a7fc640 1 -- 172.21.15.1:0/930483330 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 892578331 0 0) 0x7fe040001110 con 0x7fe04c161460 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.932+0000 7fe04b7fe640 1 -- 172.21.15.1:0/930483330 >> v1:172.21.15.1:6789/0 conn(0x7fe04c161460 legacy=0x7fe04c0716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.932+0000 7fe04b7fe640 1 -- 172.21.15.1:0/930483330 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04c164470 con 0x7fe04c06c6f0 2024-05-09T05:18:29.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.932+0000 7fe04affd640 1 --2- 172.21.15.1:0/930483330 >> v2:172.21.15.1:3300/0 conn(0x7fe04c15b090 0x7fe04c160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:29.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.934+0000 7fe04b7fe640 1 --2- 172.21.15.1:0/930483330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c06c6f0 0x7fe04c06e1a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0cf300 tx=0x7fe03c02a310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7cfdd19903a1a9ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.934+0000 7fe04a7fc640 1 -- 172.21.15.1:0/930483330 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c037070 con 0x7fe04c06c6f0 2024-05-09T05:18:29.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.934+0000 7fe04a7fc640 1 -- 172.21.15.1:0/930483330 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe03c032640 con 0x7fe04c06c6f0 2024-05-09T05:18:29.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.934+0000 7fe04a7fc640 1 -- 172.21.15.1:0/930483330 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c02ae60 con 0x7fe04c06c6f0 2024-05-09T05:18:29.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.935+0000 7fe051ad1640 1 -- 172.21.15.1:0/930483330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c06c6f0 msgr2=0x7fe04c06e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.935+0000 7fe051ad1640 1 --2- 172.21.15.1:0/930483330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c06c6f0 0x7fe04c06e1a0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe04c0cf300 tx=0x7fe03c02a310 comp rx=0 tx=0).stop 2024-05-09T05:18:29.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.936+0000 7fe051ad1640 1 -- 172.21.15.1:0/930483330 shutdown_connections 2024-05-09T05:18:29.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.936+0000 7fe051ad1640 1 --2- 172.21.15.1:0/930483330 >> v2:172.21.15.1:3300/0 conn(0x7fe04c15b090 0x7fe04c160f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.936+0000 7fe051ad1640 1 --2- 172.21.15.1:0/930483330 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c06c6f0 0x7fe04c06e1a0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.936+0000 7fe051ad1640 1 -- 172.21.15.1:0/930483330 >> 172.21.15.1:0/930483330 conn(0x7fe04c0747b0 msgr2=0x7fe04c0ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:29.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.936+0000 7fe051ad1640 1 -- 172.21.15.1:0/930483330 shutdown_connections 2024-05-09T05:18:29.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.936+0000 7fe051ad1640 1 -- 172.21.15.1:0/930483330 wait complete. 2024-05-09T05:18:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.938+0000 7fe051ad1640 1 Processor -- start 2024-05-09T05:18:29.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.938+0000 7fe051ad1640 1 -- start start 2024-05-09T05:18:29.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.941+0000 7fe051ad1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe04c06c6f0 0x7fe04c1553c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.941+0000 7fe051ad1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c15b090 0x7fe04c1faac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.941+0000 7fe051ad1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c161460 0x7fe04c159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.941+0000 7fe051ad1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c164d10 con 0x7fe04c06c6f0 2024-05-09T05:18:29.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.941+0000 7fe051ad1640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe04c164a30 con 0x7fe04c161460 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.941+0000 7fe051ad1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe04c161870 con 0x7fe04c15b090 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04bfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c161460 0x7fe04c159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04bfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c161460 0x7fe04c159810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44680/0 (socket says 172.21.15.1:44680) 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04bfff640 1 -- 172.21.15.1:0/3449185391 learned_addr learned my addr 172.21.15.1:0/3449185391 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04b7fe640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe04c06c6f0 0x7fe04c1553c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04affd640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c15b090 0x7fe04c1faac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04bfff640 1 -- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c15b090 msgr2=0x7fe04c1faac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04bfff640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c15b090 0x7fe04c1faac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04bfff640 1 -- 172.21.15.1:0/3449185391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe04c06c6f0 msgr2=0x7fe04c1553c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04bfff640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe04c06c6f0 0x7fe04c1553c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04bfff640 1 -- 172.21.15.1:0/3449185391 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe04c1fb000 con 0x7fe04c161460 2024-05-09T05:18:29.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.942+0000 7fe04affd640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c15b090 0x7fe04c1faac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:18:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.943+0000 7fe04bfff640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c161460 0x7fe04c159810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c390 tx=0x7fe04000c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.943+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe040017070 con 0x7fe04c161460 2024-05-09T05:18:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.943+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe0400035f0 con 0x7fe04c161460 2024-05-09T05:18:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.943+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0400039e0 con 0x7fe04c161460 2024-05-09T05:18:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.943+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe04c163bb0 con 0x7fe04c161460 2024-05-09T05:18:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.944+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe04c1638d0 con 0x7fe04c161460 2024-05-09T05:18:29.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.945+0000 7fe051ad1640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c162dd0 0x7fe04c156e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.945+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe04c050870 con 0x7fe04c162dd0 2024-05-09T05:18:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.946+0000 7fe04b7fe640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c162dd0 0x7fe04c156e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.946+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe04001a070 con 0x7fe04c161460 2024-05-09T05:18:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.946+0000 7fe04b7fe640 1 -- 172.21.15.1:0/3449185391 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe04c06d6e0 con 0x7fe04c162dd0 2024-05-09T05:18:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.947+0000 7fe04b7fe640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c162dd0 0x7fe04c156e70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe04c071300 tx=0x7fe03c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.947+0000 7fe033fff640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0380763d0 0x7fe0380788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.947+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe040096920 con 0x7fe04c161460 2024-05-09T05:18:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.947+0000 7fe04affd640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0380763d0 0x7fe0380788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.947+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c042070 con 0x7fe04c162dd0 2024-05-09T05:18:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.948+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fe03c03e660 con 0x7fe04c162dd0 2024-05-09T05:18:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.948+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c162dd0 msgr2=0x7fe04c156e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.948+0000 7fe033fff640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c162dd0 0x7fe04c156e70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe04c071300 tx=0x7fe03c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.948+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c162dd0 msgr2=0x7fe04c156e70 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.948+0000 7fe033fff640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c162dd0 0x7fe04c156e70 secure :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fe04c071300 tx=0x7fe03c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.948+0000 7fe04affd640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0380763d0 0x7fe0380788e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe04c06d650 tx=0x7fe04400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.958+0000 7fe051ad1640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c1573b0 0x7fe04c0c4b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.958+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe04c1577c0 con 0x7fe04c1573b0 2024-05-09T05:18:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.958+0000 7fe04b7fe640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c1573b0 0x7fe04c0c4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.959+0000 7fe04b7fe640 1 -- 172.21.15.1:0/3449185391 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fe04c0b49c0 con 0x7fe04c1573b0 2024-05-09T05:18:29.956 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.959+0000 7fe04b7fe640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c1573b0 0x7fe04c0c4b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe03c000c00 tx=0x7fe03c008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.959+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe03c037040 con 0x7fe04c1573b0 2024-05-09T05:18:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4549 (secure 0 0 0) 0x7fe03c003820 con 0x7fe04c1573b0 2024-05-09T05:18:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c1573b0 msgr2=0x7fe04c0c4b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe033fff640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c1573b0 0x7fe04c0c4b90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe03c000c00 tx=0x7fe03c008030 comp rx=0 tx=0).stop 2024-05-09T05:18:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe033fff640 1 -- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c1573b0 msgr2=0x7fe04c0c4b90 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:29.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe033fff640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe04c1573b0 0x7fe04c0c4b90 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fe03c000c00 tx=0x7fe03c008030 comp rx=0 tx=0).stop 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 5030, 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:18:29.958 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.959 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.960 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:29.961 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:29.962 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.963 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:29.964 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.964 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.964 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.964 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.964 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:29.964 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.964 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.964 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.965 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.966 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:29.967 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:29.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:29.969 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:29.970 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:29.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0380763d0 msgr2=0x7fe0380788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe051ad1640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe0380763d0 0x7fe0380788e0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fe04c06d650 tx=0x7fe04400b040 comp rx=0 tx=0).stop 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe04bfff640 1 -- 172.21.15.1:0/3449185391 reap_dead start 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c161460 msgr2=0x7fe04c159810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe051ad1640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c161460 0x7fe04c159810 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fe04000c390 tx=0x7fe04000c850 comp rx=0 tx=0).stop 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 shutdown_connections 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe051ad1640 1 --2- 172.21.15.1:0/3449185391 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe04c161460 0x7fe04c159810 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.960+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 >> 172.21.15.1:0/3449185391 conn(0x7fe04c0747b0 msgr2=0x7fe04c0c8f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.961+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 shutdown_connections 2024-05-09T05:18:29.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:29.961+0000 7fe051ad1640 1 -- 172.21.15.1:0/3449185391 wait complete. 2024-05-09T05:18:29.977 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:18:29.977 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:18:29.977 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:18:30.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.161+0000 7f7d3543b640 1 Processor -- start 2024-05-09T05:18:30.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.161+0000 7f7d3543b640 1 -- start start 2024-05-09T05:18:30.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d3543b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d30159440 0x7f7d30070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:30.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d3543b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 0x7f7d300cb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:30.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d3543b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d300c4430 con 0x7f7d30070530 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d3543b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7d300c3b90 con 0x7f7d30159440 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d3543b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d300c3e70 con 0x7f7d30153830 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d2effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d30159440 0x7f7d30070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d2e7fc640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f7d30070530 0x7f7d3015d3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:51550/0 (socket says 172.21.15.1:51550) 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d2effd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d30159440 0x7f7d30070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41720/0 (socket says 172.21.15.1:41720) 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d2f7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 0x7f7d300cb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.163+0000 7f7d2effd640 1 -- 172.21.15.1:0/1150938447 learned_addr learned my addr 172.21.15.1:0/1150938447 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.164+0000 7f7d2effd640 1 -- 172.21.15.1:0/1150938447 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 msgr2=0x7f7d300cb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.164+0000 7f7d2effd640 1 --2- 172.21.15.1:0/1150938447 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 0x7f7d300cb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.164+0000 7f7d2effd640 1 -- 172.21.15.1:0/1150938447 >> v1:172.21.15.196:6789/0 conn(0x7f7d30070530 legacy=0x7f7d3015d3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:30.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.164+0000 7f7d2effd640 1 -- 172.21.15.1:0/1150938447 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d300c4150 con 0x7f7d30159440 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.165+0000 7f7d2effd640 1 --2- 172.21.15.1:0/1150938447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30159440 0x7f7d30070ae0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009870 tx=0x7f7d2402a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9365f1cbc65ce7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.165+0000 7f7d2dffb640 1 -- 172.21.15.1:0/1150938447 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24037070 con 0x7f7d30159440 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.165+0000 7f7d2dffb640 1 -- 172.21.15.1:0/1150938447 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2402ae50 con 0x7f7d30159440 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.165+0000 7f7d2dffb640 1 -- 172.21.15.1:0/1150938447 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24032860 con 0x7f7d30159440 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.166+0000 7f7d3543b640 1 -- 172.21.15.1:0/1150938447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30159440 msgr2=0x7f7d30070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.166+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1150938447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30159440 0x7f7d30070ae0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009870 tx=0x7f7d2402a300 comp rx=0 tx=0).stop 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.166+0000 7f7d3543b640 1 -- 172.21.15.1:0/1150938447 shutdown_connections 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.166+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1150938447 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 0x7f7d300cb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.166+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1150938447 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30159440 0x7f7d30070ae0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.166+0000 7f7d3543b640 1 -- 172.21.15.1:0/1150938447 >> 172.21.15.1:0/1150938447 conn(0x7f7d30071b40 msgr2=0x7f7d30071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:30.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.166+0000 7f7d3543b640 1 -- 172.21.15.1:0/1150938447 shutdown_connections 2024-05-09T05:18:30.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.166+0000 7f7d3543b640 1 -- 172.21.15.1:0/1150938447 wait complete. 2024-05-09T05:18:30.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.169+0000 7f7d3543b640 1 Processor -- start 2024-05-09T05:18:30.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.170+0000 7f7d3543b640 1 -- start start 2024-05-09T05:18:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d3543b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30070530 0x7f7d301549c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d3543b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 0x7f7d3013b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d3543b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d30159440 0x7f7d301fa510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d3543b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d300c3870 con 0x7f7d30070530 2024-05-09T05:18:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d3543b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7d3015ceb0 con 0x7f7d30159440 2024-05-09T05:18:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d3543b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d3006fad0 con 0x7f7d30153830 2024-05-09T05:18:30.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30070530 0x7f7d301549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d2effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30070530 0x7f7d301549c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:41726/0 (socket says 172.21.15.1:41726) 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d2effd640 1 -- 172.21.15.1:0/1729304611 learned_addr learned my addr 172.21.15.1:0/1729304611 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d2f7fe640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d30159440 0x7f7d301fa510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.172+0000 7f7d2e7fc640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 0x7f7d3013b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d2effd640 1 -- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 msgr2=0x7f7d3013b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d2effd640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 0x7f7d3013b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d2effd640 1 -- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d30159440 msgr2=0x7f7d301fa510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d2effd640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d30159440 0x7f7d301fa510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d2effd640 1 -- 172.21.15.1:0/1729304611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d301fae60 con 0x7f7d30070530 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d2f7fe640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d30159440 0x7f7d301fa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d2effd640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30070530 0x7f7d301549c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009870 tx=0x7f7d240046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d177fe640 1 -- 172.21.15.1:0/1729304611 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24041070 con 0x7f7d30070530 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d30153d30 con 0x7f7d30070530 2024-05-09T05:18:30.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d177fe640 1 -- 172.21.15.1:0/1729304611 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d2403c660 con 0x7f7d30070530 2024-05-09T05:18:30.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d177fe640 1 -- 172.21.15.1:0/1729304611 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d24037040 con 0x7f7d30070530 2024-05-09T05:18:30.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.173+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d30153ff0 con 0x7f7d30070530 2024-05-09T05:18:30.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.175+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d30156610 con 0x7f7d30070530 2024-05-09T05:18:30.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.176+0000 7f7d177fe640 1 -- 172.21.15.1:0/1729304611 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7d24044070 con 0x7f7d30070530 2024-05-09T05:18:30.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.177+0000 7f7d177fe640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d1c0762d0 0x7f7d1c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:30.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.178+0000 7f7d177fe640 1 -- 172.21.15.1:0/1729304611 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7d240b7bf0 con 0x7f7d30070530 2024-05-09T05:18:30.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.178+0000 7f7d2e7fc640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d1c0762d0 0x7f7d1c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:30.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.178+0000 7f7d2e7fc640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d1c0762d0 0x7f7d1c0787e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7d3013db70 tx=0x7f7d20000bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:30.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.184+0000 7f7d177fe640 1 -- 172.21.15.1:0/1729304611 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7d24080e80 con 0x7f7d30070530 2024-05-09T05:18:30.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.671+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f7d30050870 con 0x7f7d30070530 2024-05-09T05:18:30.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.673+0000 7f7d177fe640 1 -- 172.21.15.1:0/1729304611 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f7d24085180 con 0x7f7d30070530 2024-05-09T05:18:30.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d1c0762d0 msgr2=0x7f7d1c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d1c0762d0 0x7f7d1c0787e0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f7d3013db70 tx=0x7f7d20000bc0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30070530 msgr2=0x7f7d301549c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30070530 0x7f7d301549c0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009870 tx=0x7f7d240046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 shutdown_connections 2024-05-09T05:18:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d1c0762d0 0x7f7d1c0787e0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d30159440 0x7f7d301fa510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d30153830 0x7f7d3013b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 --2- 172.21.15.1:0/1729304611 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d30070530 0x7f7d301549c0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 >> 172.21.15.1:0/1729304611 conn(0x7f7d30071b40 msgr2=0x7f7d30158790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.677+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 shutdown_connections 2024-05-09T05:18:30.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:30.678+0000 7f7d3543b640 1 -- 172.21.15.1:0/1729304611 wait complete. 2024-05-09T05:18:30.694 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:18:34.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:18:34.996 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:18:40.694 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:18:40.694 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:18:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.847+0000 7f8a7cb84640 1 Processor -- start 2024-05-09T05:18:40.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.847+0000 7f8a7cb84640 1 -- start start 2024-05-09T05:18:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7cb84640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a78073f60 0x7f8a78153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7cb84640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8a78072fe0 0x7f8a781594d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7cb84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 0x7f8a780ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:40.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7cb84640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8a780737c0 con 0x7f8a78073f60 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7cb84640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8a780c4260 con 0x7f8a78072fe0 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7cb84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8a780c4540 con 0x7f8a780c2b30 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a78073f60 0x7f8a78153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7659c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8a78073f60 0x7f8a78153910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43218/0 (socket says 172.21.15.1:43218) 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a7659c640 1 -- 172.21.15.1:0/1355574203 learned_addr learned my addr 172.21.15.1:0/1355574203 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a6ffff640 1 --2- 172.21.15.1:0/1355574203 >> v2:172.21.15.196:3300/0 conn(0x7f8a78072fe0 0x7f8a781594d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.849+0000 7f8a76d9d640 1 --2- 172.21.15.1:0/1355574203 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 0x7f8a780ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.850+0000 7f8a7659c640 1 -- 172.21.15.1:0/1355574203 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 msgr2=0x7f8a780ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.850+0000 7f8a7659c640 1 --2- 172.21.15.1:0/1355574203 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 0x7f8a780ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.850+0000 7f8a7659c640 1 -- 172.21.15.1:0/1355574203 >> v2:172.21.15.196:3300/0 conn(0x7f8a78072fe0 msgr2=0x7f8a781594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.850+0000 7f8a7659c640 1 --2- 172.21.15.1:0/1355574203 >> v2:172.21.15.196:3300/0 conn(0x7f8a78072fe0 0x7f8a781594d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:40.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.850+0000 7f8a7659c640 1 -- 172.21.15.1:0/1355574203 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a780c4820 con 0x7f8a78073f60 2024-05-09T05:18:40.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.851+0000 7f8a7659c640 1 --2- 172.21.15.1:0/1355574203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78073f60 0x7f8a78153910 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8a781570e0 tx=0x7f8a6802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28404a4688983877 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.851+0000 7f8a75d9b640 1 -- 172.21.15.1:0/1355574203 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68036070 con 0x7f8a78073f60 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.851+0000 7f8a75d9b640 1 -- 172.21.15.1:0/1355574203 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68031600 con 0x7f8a78073f60 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.851+0000 7f8a75d9b640 1 -- 172.21.15.1:0/1355574203 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a680319d0 con 0x7f8a78073f60 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 -- 172.21.15.1:0/1355574203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78073f60 msgr2=0x7f8a78153910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/1355574203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78073f60 0x7f8a78153910 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8a781570e0 tx=0x7f8a6802a730 comp rx=0 tx=0).stop 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 -- 172.21.15.1:0/1355574203 shutdown_connections 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/1355574203 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 0x7f8a780ca6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/1355574203 >> v2:172.21.15.196:3300/0 conn(0x7f8a78072fe0 0x7f8a781594d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/1355574203 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78073f60 0x7f8a78153910 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 -- 172.21.15.1:0/1355574203 >> 172.21.15.1:0/1355574203 conn(0x7f8a78070dd0 msgr2=0x7f8a780711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 -- 172.21.15.1:0/1355574203 shutdown_connections 2024-05-09T05:18:40.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.852+0000 7f8a7cb84640 1 -- 172.21.15.1:0/1355574203 wait complete. 2024-05-09T05:18:40.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.854+0000 7f8a7cb84640 1 Processor -- start 2024-05-09T05:18:40.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.854+0000 7f8a7cb84640 1 -- start start 2024-05-09T05:18:40.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7cb84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78072fe0 0x7f8a781eed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7cb84640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a78073f60 0x7f8a781478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7cb84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 0x7f8a78145100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7cb84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8a780733f0 con 0x7f8a78072fe0 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7cb84640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8a78159a10 con 0x7f8a78073f60 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7cb84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8a780c2f40 con 0x7f8a780c2b30 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78072fe0 0x7f8a781eed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78072fe0 0x7f8a781eed70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43232/0 (socket says 172.21.15.1:43232) 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7659c640 1 -- 172.21.15.1:0/3568012001 learned_addr learned my addr 172.21.15.1:0/3568012001 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:40.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a76d9d640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 0x7f8a78145100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a6ffff640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a78073f60 0x7f8a781478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7659c640 1 -- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 msgr2=0x7f8a78145100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7659c640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 0x7f8a78145100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7659c640 1 -- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a78073f60 msgr2=0x7f8a781478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7659c640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a78073f60 0x7f8a781478a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.855+0000 7f8a7659c640 1 -- 172.21.15.1:0/3568012001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a781ef2b0 con 0x7f8a78072fe0 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.856+0000 7f8a7659c640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78072fe0 0x7f8a781eed70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a68000f80 tx=0x7f8a68004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.856+0000 7f8a6effd640 1 -- 172.21.15.1:0/3568012001 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68036050 con 0x7f8a78072fe0 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.856+0000 7f8a6effd640 1 -- 172.21.15.1:0/3568012001 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8a68042070 con 0x7f8a78072fe0 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.856+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a7814de90 con 0x7f8a78072fe0 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.856+0000 7f8a6effd640 1 -- 172.21.15.1:0/3568012001 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a68009510 con 0x7f8a78072fe0 2024-05-09T05:18:40.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.856+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a7814e450 con 0x7f8a78072fe0 2024-05-09T05:18:40.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.857+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a780696d0 con 0x7f8a78072fe0 2024-05-09T05:18:40.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.858+0000 7f8a6effd640 1 -- 172.21.15.1:0/3568012001 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8a68007c70 con 0x7f8a78072fe0 2024-05-09T05:18:40.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.858+0000 7f8a6effd640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a58076300 0x7f8a58078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:40.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.859+0000 7f8a6effd640 1 -- 172.21.15.1:0/3568012001 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8a680b8380 con 0x7f8a78072fe0 2024-05-09T05:18:40.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.859+0000 7f8a6ffff640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a58076300 0x7f8a58078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:40.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.859+0000 7f8a6ffff640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a58076300 0x7f8a58078810 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a781598c0 tx=0x7f8a60009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:40.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:40.869+0000 7f8a6effd640 1 -- 172.21.15.1:0/3568012001 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8a680814e0 con 0x7f8a78072fe0 2024-05-09T05:18:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.356+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8a78050870 con 0x7f8a78072fe0 2024-05-09T05:18:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.356+0000 7f8a6effd640 1 -- 172.21.15.1:0/3568012001 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f8a680857e0 con 0x7f8a78072fe0 2024-05-09T05:18:41.354 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:18:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a58076300 msgr2=0x7f8a58078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a58076300 0x7f8a58078810 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f8a781598c0 tx=0x7f8a60009190 comp rx=0 tx=0).stop 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78072fe0 msgr2=0x7f8a781eed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78072fe0 0x7f8a781eed70 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8a68000f80 tx=0x7f8a68004a40 comp rx=0 tx=0).stop 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 shutdown_connections 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8a58076300 0x7f8a58078810 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8a78073f60 0x7f8a781478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8a780c2b30 0x7f8a78145100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 --2- 172.21.15.1:0/3568012001 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8a78072fe0 0x7f8a781eed70 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 >> 172.21.15.1:0/3568012001 conn(0x7f8a78070dd0 msgr2=0x7f8a78148560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 shutdown_connections 2024-05-09T05:18:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.360+0000 7f8a7cb84640 1 -- 172.21.15.1:0/3568012001 wait complete. 2024-05-09T05:18:41.376 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:18:41.377 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:18:41.377 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:18:41.377 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:18:41.561 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.564+0000 7f7f8e57e640 1 Processor -- start 2024-05-09T05:18:41.562 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.564+0000 7f7f8e57e640 1 -- start start 2024-05-09T05:18:41.564 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.566+0000 7f7f8e57e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7f8815de60 0x7f7f8815e270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.566+0000 7f7f8e57e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f880ccd20 con 0x7f7f88154770 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.566+0000 7f7f8d57c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7f88154770 0x7f7f880ca350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54912/0 (socket says 172.21.15.1:54912) 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.566+0000 7f7f8d57c640 1 -- 172.21.15.1:0/948913190 learned_addr learned my addr 172.21.15.1:0/948913190 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.566+0000 7f7f8e57e640 1 -- 172.21.15.1:0/948913190 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7f8815d960 con 0x7f7f880cc200 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.567+0000 7f7f8cd7b640 1 --2- 172.21.15.1:0/948913190 >> v2:172.21.15.196:3300/0 conn(0x7f7f8815de60 0x7f7f8815e270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.567+0000 7f7f8e57e640 1 -- 172.21.15.1:0/948913190 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f7f88154b80 con 0x7f7f8815de60 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.567+0000 7f7f7ffff640 1 -- 172.21.15.1:0/948913190 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050118329 0 0) 0x7f7f880ccd20 con 0x7f7f88154770 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.567+0000 7f7f7ffff640 1 -- 172.21.15.1:0/948913190 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7f74003520 con 0x7f7f88154770 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.567+0000 7f7f7ffff640 1 -- 172.21.15.1:0/948913190 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1314507453 0 0) 0x7f7f8815d960 con 0x7f7f880cc200 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.567+0000 7f7f8cd7b640 1 -- 172.21.15.1:0/948913190 >> v1:172.21.15.196:6789/0 conn(0x7f7f880cc200 legacy=0x7f7f880cc610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.567+0000 7f7f8cd7b640 1 -- 172.21.15.1:0/948913190 >> v1:172.21.15.1:6789/0 conn(0x7f7f88154770 legacy=0x7f7f880ca350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.565 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.567+0000 7f7f8cd7b640 1 -- 172.21.15.1:0/948913190 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f880cb280 con 0x7f7f8815de60 2024-05-09T05:18:41.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.568+0000 7f7f8cd7b640 1 --2- 172.21.15.1:0/948913190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f8815de60 0x7f7f8815e270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f780097a0 tx=0x7f7f7802a2c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62b5ae8f80dca55 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.568+0000 7f7f7ffff640 1 -- 172.21.15.1:0/948913190 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78037070 con 0x7f7f8815de60 2024-05-09T05:18:41.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.568+0000 7f7f7ffff640 1 -- 172.21.15.1:0/948913190 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f78032640 con 0x7f7f8815de60 2024-05-09T05:18:41.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.568+0000 7f7f7ffff640 1 -- 172.21.15.1:0/948913190 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78032a10 con 0x7f7f8815de60 2024-05-09T05:18:41.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.569+0000 7f7f8e57e640 1 -- 172.21.15.1:0/948913190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f8815de60 msgr2=0x7f7f8815e270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.569+0000 7f7f8e57e640 1 --2- 172.21.15.1:0/948913190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f8815de60 0x7f7f8815e270 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f7f780097a0 tx=0x7f7f7802a2c0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.570+0000 7f7f8e57e640 1 -- 172.21.15.1:0/948913190 shutdown_connections 2024-05-09T05:18:41.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.570+0000 7f7f8e57e640 1 --2- 172.21.15.1:0/948913190 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f8815de60 0x7f7f8815e270 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.570+0000 7f7f8e57e640 1 -- 172.21.15.1:0/948913190 >> 172.21.15.1:0/948913190 conn(0x7f7f88072a70 msgr2=0x7f7f88072e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:41.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.570+0000 7f7f8e57e640 1 -- 172.21.15.1:0/948913190 shutdown_connections 2024-05-09T05:18:41.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.570+0000 7f7f8e57e640 1 -- 172.21.15.1:0/948913190 wait complete. 2024-05-09T05:18:41.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.572+0000 7f7f8e57e640 1 Processor -- start 2024-05-09T05:18:41.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.572+0000 7f7f8e57e640 1 -- start start 2024-05-09T05:18:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.574+0000 7f7f8e57e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880cc200 0x7f7f880b1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.574+0000 7f7f8e57e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f88154770 0x7f7f881f3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8e57e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7f8815de60 0x7f7f881e5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8e57e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7f880cbb20 con 0x7f7f880cc200 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8e57e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7f880cb840 con 0x7f7f88154770 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8e57e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7f880cbe00 con 0x7f7f8815de60 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f88154770 0x7f7f881f3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8d57c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880cc200 0x7f7f880b1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f88154770 0x7f7f881f3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:36768/0 (socket says 172.21.15.1:36768) 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 -- 172.21.15.1:0/1953053622 learned_addr learned my addr 172.21.15.1:0/1953053622 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8dd7d640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7f8815de60 0x7f7f881e5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 -- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7f8815de60 msgr2=0x7f7f881e5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7f8815de60 0x7f7f881e5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 -- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880cc200 msgr2=0x7f7f880b1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880cc200 0x7f7f880b1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 -- 172.21.15.1:0/1953053622 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f881f3370 con 0x7f7f88154770 2024-05-09T05:18:41.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.575+0000 7f7f8cd7b640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f88154770 0x7f7f881f3830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7f7802a8b0 tx=0x7f7f780046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.576+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78041020 con 0x7f7f88154770 2024-05-09T05:18:41.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.576+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7f780306f0 con 0x7f7f88154770 2024-05-09T05:18:41.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.576+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78037040 con 0x7f7f88154770 2024-05-09T05:18:41.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.576+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f881f3d70 con 0x7f7f88154770 2024-05-09T05:18:41.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.576+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f8815d350 con 0x7f7f88154770 2024-05-09T05:18:41.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.579+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7f78044020 con 0x7f7f88154770 2024-05-09T05:18:41.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.579+0000 7f7f8e57e640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880b1f60 0x7f7f8815d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.579+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7f880c6740 con 0x7f7f880b1f60 2024-05-09T05:18:41.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.579+0000 7f7f8d57c640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880b1f60 0x7f7f8815d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.580+0000 7f7f7dffb640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7f6c0762d0 0x7f7f6c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.580+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7f780b7d90 con 0x7f7f88154770 2024-05-09T05:18:41.577 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.580+0000 7f7f8dd7d640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7f6c0762d0 0x7f7f6c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.580+0000 7f7f8d57c640 1 -- 172.21.15.1:0/1953053622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7f881e51c0 con 0x7f7f880b1f60 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.580+0000 7f7f8d57c640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880b1f60 0x7f7f8815d520 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7f84001480 tx=0x7f7f84001940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.580+0000 7f7f8dd7d640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7f6c0762d0 0x7f7f6c0787e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7f880d05c0 tx=0x7f7f80009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.581+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f84019070 con 0x7f7f880b1f60 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.581+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7f84005ac0 con 0x7f7f880b1f60 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.581+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880b1f60 msgr2=0x7f7f8815d520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.581+0000 7f7f7dffb640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880b1f60 0x7f7f8815d520 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f7f84001480 tx=0x7f7f84001940 comp rx=0 tx=0).stop 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.581+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880b1f60 msgr2=0x7f7f8815d520 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:41.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.581+0000 7f7f7dffb640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f880b1f60 0x7f7f8815d520 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.597+0000 7f7f8e57e640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f8814c050 0x7f7f8814c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.597+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7f88076540 con 0x7f7f8814c050 2024-05-09T05:18:41.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.597+0000 7f7f8d57c640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f8814c050 0x7f7f8814c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.598+0000 7f7f8d57c640 1 -- 172.21.15.1:0/1953053622 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7f880b4bf0 con 0x7f7f8814c050 2024-05-09T05:18:41.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.598+0000 7f7f8d57c640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f8814c050 0x7f7f8814c460 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7f84001480 tx=0x7f7f840057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.599+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f84019070 con 0x7f7f8814c050 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.599+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7f7f8401a040 con 0x7f7f8814c050 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.599+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f8814c050 msgr2=0x7f7f8814c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.599+0000 7f7f7dffb640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f8814c050 0x7f7f8814c460 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7f84001480 tx=0x7f7f840057f0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.599+0000 7f7f7dffb640 1 -- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f8814c050 msgr2=0x7f7f8814c460 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.599+0000 7f7f7dffb640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7f8814c050 0x7f7f8814c460 secure :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f7f84001480 tx=0x7f7f840057f0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:41.597 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 323794, 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:41.598 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:41.599 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.600 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:41.601 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:41.602 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.603 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.604 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:41.605 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.606 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:41.607 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:41.608 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:18:41.609 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.610 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.611 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:41.612 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7f6c0762d0 msgr2=0x7f7f6c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8e57e640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7f6c0762d0 0x7f7f6c0787e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f7f880d05c0 tx=0x7f7f80009040 comp rx=0 tx=0).stop 2024-05-09T05:18:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f88154770 msgr2=0x7f7f881f3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8e57e640 1 --2- 172.21.15.1:0/1953053622 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7f88154770 0x7f7f881f3830 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7f7802a8b0 tx=0x7f7f780046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8dd7d640 1 -- 172.21.15.1:0/1953053622 reap_dead start 2024-05-09T05:18:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8dd7d640 1 -- 172.21.15.1:0/1953053622 reap_dead start 2024-05-09T05:18:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 shutdown_connections 2024-05-09T05:18:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 >> 172.21.15.1:0/1953053622 conn(0x7f7f88072a70 msgr2=0x7f7f88075e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 shutdown_connections 2024-05-09T05:18:41.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.600+0000 7f7f8e57e640 1 -- 172.21.15.1:0/1953053622 wait complete. 2024-05-09T05:18:41.614 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:18:41.614 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:18:41.615 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:18:41.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.797+0000 7f5f82b7e640 1 Processor -- start 2024-05-09T05:18:41.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.797+0000 7f5f82b7e640 1 -- start start 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.799+0000 7f5f82b7e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c1533f0 0x7f5f7c15a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.799+0000 7f5f82b7e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5f7c0c3870 0x7f5f7c0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.799+0000 7f5f82b7e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5f7c15d790 con 0x7f5f7c15c890 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.799+0000 7f5f82b7e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f5f7c159ce0 con 0x7f5f7c0c3870 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.799+0000 7f5f82b7e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5f7c15d4b0 con 0x7f5f7c1533f0 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.799+0000 7f5f81b7c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c1533f0 0x7f5f7c15a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.799+0000 7f5f8137b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5f7c0c3870 0x7f5f7c0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.799+0000 7f5f81b7c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c1533f0 0x7f5f7c15a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:53752/0 (socket says 172.21.15.1:53752) 2024-05-09T05:18:41.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.800+0000 7f5f81b7c640 1 -- 172.21.15.1:0/3522049208 learned_addr learned my addr 172.21.15.1:0/3522049208 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.800+0000 7f5f81b7c640 1 -- 172.21.15.1:0/3522049208 >> v2:172.21.15.196:3300/0 conn(0x7f5f7c0c3870 msgr2=0x7f5f7c0c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.800+0000 7f5f81b7c640 1 --2- 172.21.15.1:0/3522049208 >> v2:172.21.15.196:3300/0 conn(0x7f5f7c0c3870 0x7f5f7c0c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.800+0000 7f5f81b7c640 1 -- 172.21.15.1:0/3522049208 >> v1:172.21.15.196:6789/0 conn(0x7f5f7c15c890 legacy=0x7f5f7c15cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.800+0000 7f5f81b7c640 1 -- 172.21.15.1:0/3522049208 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c1593b0 con 0x7f5f7c1533f0 2024-05-09T05:18:41.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.800+0000 7f5f8137b640 1 --2- 172.21.15.1:0/3522049208 >> v2:172.21.15.196:3300/0 conn(0x7f5f7c0c3870 0x7f5f7c0c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.800+0000 7f5f80b7a640 1 -- 172.21.15.1:0/3522049208 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 367608661 0 0) 0x7f5f7c15d790 con 0x7f5f7c15c890 2024-05-09T05:18:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.801+0000 7f5f81b7c640 1 --2- 172.21.15.1:0/3522049208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c1533f0 0x7f5f7c15a790 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009e50 tx=0x7f5f7802a7c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9d7b54d9a2524f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.801+0000 7f5f80b7a640 1 -- 172.21.15.1:0/3522049208 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78036070 con 0x7f5f7c1533f0 2024-05-09T05:18:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.801+0000 7f5f80b7a640 1 -- 172.21.15.1:0/3522049208 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f78031600 con 0x7f5f7c1533f0 2024-05-09T05:18:41.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.802+0000 7f5f80b7a640 1 -- 172.21.15.1:0/3522049208 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f780319f0 con 0x7f5f7c1533f0 2024-05-09T05:18:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.802+0000 7f5f82b7e640 1 -- 172.21.15.1:0/3522049208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c1533f0 msgr2=0x7f5f7c15a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.802+0000 7f5f82b7e640 1 --2- 172.21.15.1:0/3522049208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c1533f0 0x7f5f7c15a790 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009e50 tx=0x7f5f7802a7c0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.803+0000 7f5f82b7e640 1 -- 172.21.15.1:0/3522049208 shutdown_connections 2024-05-09T05:18:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.803+0000 7f5f82b7e640 1 --2- 172.21.15.1:0/3522049208 >> v2:172.21.15.196:3300/0 conn(0x7f5f7c0c3870 0x7f5f7c0c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.803+0000 7f5f82b7e640 1 --2- 172.21.15.1:0/3522049208 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c1533f0 0x7f5f7c15a790 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.803+0000 7f5f82b7e640 1 -- 172.21.15.1:0/3522049208 >> 172.21.15.1:0/3522049208 conn(0x7f5f7c071930 msgr2=0x7f5f7c071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:41.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.803+0000 7f5f82b7e640 1 -- 172.21.15.1:0/3522049208 shutdown_connections 2024-05-09T05:18:41.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.803+0000 7f5f82b7e640 1 -- 172.21.15.1:0/3522049208 wait complete. 2024-05-09T05:18:41.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.805+0000 7f5f82b7e640 1 Processor -- start 2024-05-09T05:18:41.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.806+0000 7f5f82b7e640 1 -- start start 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f82b7e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f7c0c3870 0x7f5f7c0c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f82b7e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c1533f0 0x7f5f7c1461c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f81b7c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f7c0c3870 0x7f5f7c0c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f81b7c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f7c0c3870 0x7f5f7c0c6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43288/0 (socket says 172.21.15.1:43288) 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f82b7e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c15c890 0x7f5f7c146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f81b7c640 1 -- 172.21.15.1:0/614058041 learned_addr learned my addr 172.21.15.1:0/614058041 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c15a170 con 0x7f5f7c0c3870 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c159970 con 0x7f5f7c1533f0 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f8137b640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c1533f0 0x7f5f7c1461c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f8237d640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c15c890 0x7f5f7c146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.808+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5f7c153800 con 0x7f5f7c15c890 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.809+0000 7f5f8137b640 1 -- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c15c890 msgr2=0x7f5f7c146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.809+0000 7f5f8137b640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c15c890 0x7f5f7c146700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.809+0000 7f5f8137b640 1 -- 172.21.15.1:0/614058041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f7c0c3870 msgr2=0x7f5f7c0c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.809+0000 7f5f8137b640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f7c0c3870 0x7f5f7c0c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.809+0000 7f5f8137b640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f7c0ccef0 con 0x7f5f7c1533f0 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.809+0000 7f5f8237d640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5f7c15c890 0x7f5f7c146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.809+0000 7f5f81b7c640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5f7c0c3870 0x7f5f7c0c6840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.809+0000 7f5f8137b640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c1533f0 0x7f5f7c1461c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000e840 tx=0x7f5f7000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.810+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70017070 con 0x7f5f7c1533f0 2024-05-09T05:18:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.810+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5f7000c940 con 0x7f5f7c1533f0 2024-05-09T05:18:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.810+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000cd30 con 0x7f5f7c1533f0 2024-05-09T05:18:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.810+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f7c0cf660 con 0x7f5f7c1533f0 2024-05-09T05:18:41.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.810+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f7c0c7320 con 0x7f5f7c1533f0 2024-05-09T05:18:41.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.811+0000 7f5f82b7e640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0cd280 0x7f5f7c0c7b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.811+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c0cd980 con 0x7f5f7c0cd280 2024-05-09T05:18:41.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.811+0000 7f5f81b7c640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0cd280 0x7f5f7c0c7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.812+0000 7f5f81b7c640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5f7c156aa0 con 0x7f5f7c0cd280 2024-05-09T05:18:41.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.812+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5f7001a070 con 0x7f5f7c1533f0 2024-05-09T05:18:41.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.812+0000 7f5f81b7c640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0cd280 0x7f5f7c0c7b20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009e50 tx=0x7f5f78004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.813+0000 7f5f6e7fc640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5f600762d0 0x7f5f600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.814+0000 7f5f8237d640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5f600762d0 0x7f5f600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.814+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5f700973c0 con 0x7f5f7c1533f0 2024-05-09T05:18:41.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.814+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78043070 con 0x7f5f7c0cd280 2024-05-09T05:18:41.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.814+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f5f780095a0 con 0x7f5f7c0cd280 2024-05-09T05:18:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.814+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0cd280 msgr2=0x7f5f7c0c7b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.814+0000 7f5f6e7fc640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0cd280 0x7f5f7c0c7b20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f5f78009e50 tx=0x7f5f78004a40 comp rx=0 tx=0).stop 2024-05-09T05:18:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.814+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0cd280 msgr2=0x7f5f7c0c7b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.815+0000 7f5f6e7fc640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0cd280 0x7f5f7c0c7b20 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.815+0000 7f5f8237d640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5f600762d0 0x7f5f600787e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5f74000f60 tx=0x7f5f74009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.826+0000 7f5f82b7e640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0ceff0 0x7f5f7c0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.826+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5f7c0ccca0 con 0x7f5f7c0ceff0 2024-05-09T05:18:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.827+0000 7f5f81b7c640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0ceff0 0x7f5f7c0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:41.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.827+0000 7f5f81b7c640 1 -- 172.21.15.1:0/614058041 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f5f7c050870 con 0x7f5f7c0ceff0 2024-05-09T05:18:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.828+0000 7f5f81b7c640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0ceff0 0x7f5f7c0ca050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c0ccc20 tx=0x7f5f7802f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.828+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f78036040 con 0x7f5f7c0ceff0 2024-05-09T05:18:41.825 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.828+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f5f780044d0 con 0x7f5f7c0ceff0 2024-05-09T05:18:41.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.828+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0ceff0 msgr2=0x7f5f7c0ca050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.828+0000 7f5f6e7fc640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0ceff0 0x7f5f7c0ca050 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f5f7c0ccc20 tx=0x7f5f7802f000 comp rx=0 tx=0).stop 2024-05-09T05:18:41.827 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:41.827 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:41.827 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:41.827 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 72232, 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:41.828 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:41.829 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:41.830 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:41.831 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:41.832 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:41.833 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.833 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:41.833 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.833 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:41.833 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.834 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:41.834 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:41.834 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:41.834 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:41.834 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.834 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:41.835 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:41.835 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.835 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.835 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.835 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:41.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:41.836 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.836 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.836 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.836 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.836 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:41.836 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:41.836 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.837 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:41.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:41.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.838 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.838 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.838 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:41.838 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:41.838 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.838 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:41.839 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:41.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:41.840 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.840 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:41.840 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:41.840 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.840 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:41.841 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:41.842 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:41.842 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.842 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.842 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.842 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:41.842 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:41.843 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.843 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:41.843 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:41.843 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:41.843 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.843 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:41.844 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.828+0000 7f5f6e7fc640 1 -- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0ceff0 msgr2=0x7f5f7c0ca050 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.828+0000 7f5f6e7fc640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c0ceff0 0x7f5f7c0ca050 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5f600762d0 msgr2=0x7f5f600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5f600762d0 0x7f5f600787e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f5f74000f60 tx=0x7f5f74009040 comp rx=0 tx=0).stop 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f8237d640 1 -- 172.21.15.1:0/614058041 reap_dead start 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c1533f0 msgr2=0x7f5f7c1461c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c1533f0 0x7f5f7c1461c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5f7000e840 tx=0x7f5f7000ed00 comp rx=0 tx=0).stop 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 shutdown_connections 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 --2- 172.21.15.1:0/614058041 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5f7c1533f0 0x7f5f7c1461c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:41.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 >> 172.21.15.1:0/614058041 conn(0x7f5f7c071930 msgr2=0x7f5f7c0c3c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 shutdown_connections 2024-05-09T05:18:41.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:41.829+0000 7f5f82b7e640 1 -- 172.21.15.1:0/614058041 wait complete. 2024-05-09T05:18:41.848 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:18:41.848 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:18:41.848 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:18:42.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.031+0000 7f8407a3b640 1 Processor -- start 2024-05-09T05:18:42.028 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.031+0000 7f8407a3b640 1 -- start start 2024-05-09T05:18:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.032+0000 7f8407a3b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f840015add0 0x7f84000c3b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8407a3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8400164310 0x7f8400164720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8407a3b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f840015b1e0 con 0x7f840006c6f0 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8407a3b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8400161960 con 0x7f840015add0 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8407a3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8400164c60 con 0x7f8400164310 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8404fd6640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f840006c6f0 0x7f8400070230 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:34816/0 (socket says 172.21.15.1:34816) 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8405fd8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8400164310 0x7f8400164720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8404fd6640 1 -- 172.21.15.1:0/1190342114 learned_addr learned my addr 172.21.15.1:0/1190342114 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f84057d7640 1 --2- 172.21.15.1:0/1190342114 >> v2:172.21.15.1:3300/0 conn(0x7f840015add0 0x7f84000c3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f83f7fff640 1 -- 172.21.15.1:0/1190342114 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3097252655 0 0) 0x7f840015b1e0 con 0x7f840006c6f0 2024-05-09T05:18:42.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f83f7fff640 1 -- 172.21.15.1:0/1190342114 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f83ec003520 con 0x7f840006c6f0 2024-05-09T05:18:42.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8405fd8640 1 -- 172.21.15.1:0/1190342114 >> v2:172.21.15.1:3300/0 conn(0x7f840015add0 msgr2=0x7f84000c3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:42.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8405fd8640 1 --2- 172.21.15.1:0/1190342114 >> v2:172.21.15.1:3300/0 conn(0x7f840015add0 0x7f84000c3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8405fd8640 1 -- 172.21.15.1:0/1190342114 >> v1:172.21.15.196:6789/0 conn(0x7f840006c6f0 legacy=0x7f8400070230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.033+0000 7f8405fd8640 1 -- 172.21.15.1:0/1190342114 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f840006cf40 con 0x7f8400164310 2024-05-09T05:18:42.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.034+0000 7f8405fd8640 1 --2- 172.21.15.1:0/1190342114 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8400164310 0x7f8400164720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009d90 tx=0x7f83f802a5d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=399e96064e9d6fc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:42.032 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.035+0000 7f83f7fff640 1 -- 172.21.15.1:0/1190342114 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8031600 con 0x7f8400164310 2024-05-09T05:18:42.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.035+0000 7f83f7fff640 1 -- 172.21.15.1:0/1190342114 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83f80317a0 con 0x7f8400164310 2024-05-09T05:18:42.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.035+0000 7f83f7fff640 1 -- 172.21.15.1:0/1190342114 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f8031b70 con 0x7f8400164310 2024-05-09T05:18:42.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.036+0000 7f8407a3b640 1 -- 172.21.15.1:0/1190342114 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8400164310 msgr2=0x7f8400164720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.033 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.036+0000 7f8407a3b640 1 --2- 172.21.15.1:0/1190342114 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8400164310 0x7f8400164720 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f83f8009d90 tx=0x7f83f802a5d0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.036+0000 7f8407a3b640 1 -- 172.21.15.1:0/1190342114 shutdown_connections 2024-05-09T05:18:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.036+0000 7f8407a3b640 1 --2- 172.21.15.1:0/1190342114 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8400164310 0x7f8400164720 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.036+0000 7f8407a3b640 1 --2- 172.21.15.1:0/1190342114 >> v2:172.21.15.1:3300/0 conn(0x7f840015add0 0x7f84000c3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.036+0000 7f8407a3b640 1 -- 172.21.15.1:0/1190342114 >> 172.21.15.1:0/1190342114 conn(0x7f840006d800 msgr2=0x7f840006dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.036+0000 7f8407a3b640 1 -- 172.21.15.1:0/1190342114 shutdown_connections 2024-05-09T05:18:42.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.036+0000 7f8407a3b640 1 -- 172.21.15.1:0/1190342114 wait complete. 2024-05-09T05:18:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.039+0000 7f8407a3b640 1 Processor -- start 2024-05-09T05:18:42.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.039+0000 7f8407a3b640 1 -- start start 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f8407a3b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f840006c6f0 0x7f84000c7f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f8407a3b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840015add0 0x7f84001546c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f8407a3b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8400164310 0x7f840015d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f8407a3b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f84001615f0 con 0x7f8400164310 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f8407a3b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8400161310 con 0x7f840015add0 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f8407a3b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8400160d20 con 0x7f840006c6f0 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f84057d7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f840006c6f0 0x7f84000c7f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f84057d7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f840006c6f0 0x7f84000c7f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:53786/0 (socket says 172.21.15.1:53786) 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.041+0000 7f84057d7640 1 -- 172.21.15.1:0/509398458 learned_addr learned my addr 172.21.15.1:0/509398458 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:42.042 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.042+0000 7f8404fd6640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840015add0 0x7f84001546c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.042+0000 7f8405fd8640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8400164310 0x7f840015d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.042+0000 7f84057d7640 1 -- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840015add0 msgr2=0x7f84001546c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.042+0000 7f84057d7640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f840015add0 0x7f84001546c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.042+0000 7f84057d7640 1 -- 172.21.15.1:0/509398458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8400164310 msgr2=0x7f840015d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.042+0000 7f84057d7640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8400164310 0x7f840015d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.042+0000 7f84057d7640 1 -- 172.21.15.1:0/509398458 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84000c5620 con 0x7f840006c6f0 2024-05-09T05:18:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.042+0000 7f84057d7640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f840006c6f0 0x7f84000c7f90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e870 tx=0x7f83fc00ed30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:42.043 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.043+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc017070 con 0x7f840006c6f0 2024-05-09T05:18:42.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.043+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f83fc00c8e0 con 0x7f840006c6f0 2024-05-09T05:18:42.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.043+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83fc00ccd0 con 0x7f840006c6f0 2024-05-09T05:18:42.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.043+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84000c5900 con 0x7f840006c6f0 2024-05-09T05:18:42.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.043+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f84000c7360 con 0x7f840006c6f0 2024-05-09T05:18:42.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.044+0000 7f8407a3b640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84001f82f0 0x7f84001f6e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.044 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.044+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f840015dc00 con 0x7f84001f82f0 2024-05-09T05:18:42.045 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.044+0000 7f8404fd6640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84001f82f0 0x7f84001f6e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.045+0000 7f8404fd6640 1 -- 172.21.15.1:0/509398458 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f84001f7dc0 con 0x7f84001f82f0 2024-05-09T05:18:42.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.045+0000 7f8404fd6640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84001f82f0 0x7f84001f6e40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8400071070 tx=0x7f83f0001930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:42.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.045+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f83fc01a070 con 0x7f840006c6f0 2024-05-09T05:18:42.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f83f5ffb640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f83e4076320 0x7f83e4078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f83fc097670 con 0x7f840006c6f0 2024-05-09T05:18:42.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f8405fd8640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f83e4076320 0x7f83e4078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0019070 con 0x7f84001f82f0 2024-05-09T05:18:42.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f83f0005ac0 con 0x7f84001f82f0 2024-05-09T05:18:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84001f82f0 msgr2=0x7f84001f6e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f83f5ffb640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84001f82f0 0x7f84001f6e40 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8400071070 tx=0x7f83f0001930 comp rx=0 tx=0).stop 2024-05-09T05:18:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84001f82f0 msgr2=0x7f84001f6e40 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.046+0000 7f83f5ffb640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84001f82f0 0x7f84001f6e40 secure :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8400071070 tx=0x7f83f0001930 comp rx=0 tx=0).stop 2024-05-09T05:18:42.047 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.047+0000 7f8405fd8640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f83e4076320 0x7f83e4078830 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f83f8034000 tx=0x7f83f802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.062+0000 7f8407a3b640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84000c5c90 0x7f84001f7380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.059 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.062+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8400050870 con 0x7f84000c5c90 2024-05-09T05:18:42.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.062+0000 7f8404fd6640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84000c5c90 0x7f84001f7380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.063+0000 7f8404fd6640 1 -- 172.21.15.1:0/509398458 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f840015e420 con 0x7f84000c5c90 2024-05-09T05:18:42.060 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.063+0000 7f8404fd6640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84000c5c90 0x7f84001f7380 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8400071070 tx=0x7f83f0007cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.063+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83f0019030 con 0x7f84000c5c90 2024-05-09T05:18:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f83f0005e10 con 0x7f84000c5c90 2024-05-09T05:18:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84000c5c90 msgr2=0x7f84001f7380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f83f5ffb640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84000c5c90 0x7f84001f7380 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8400071070 tx=0x7f83f0007cf0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f83f5ffb640 1 -- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84000c5c90 msgr2=0x7f84001f7380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:42.061 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f83f5ffb640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f84000c5c90 0x7f84001f7380 secure :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f8400071070 tx=0x7f83f0007cf0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.062 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:42.062 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:42.062 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:42.062 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:18:42.062 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 94, 2024-05-09T05:18:42.062 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 17135, 2024-05-09T05:18:42.062 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:42.062 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:42.063 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:42.064 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:42.065 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:42.066 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:42.067 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:42.067 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:42.067 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:42.067 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:42.067 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:42.067 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:42.068 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:42.068 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:42.068 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:42.068 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:42.068 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.068 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:42.068 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:42.069 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:42.070 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:42.071 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.071 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:42.071 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:42.071 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:42.071 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.072 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.072 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:42.072 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:42.073 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:42.074 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.075 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:42.076 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:18:42.077 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:42.077 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:42.077 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:42.077 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:42.077 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f83e4076320 msgr2=0x7f83e4078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f8407a3b640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f83e4076320 0x7f83e4078830 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f83f8034000 tx=0x7f83f802f000 comp rx=0 tx=0).stop 2024-05-09T05:18:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f840006c6f0 msgr2=0x7f84000c7f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f8407a3b640 1 --2- 172.21.15.1:0/509398458 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f840006c6f0 0x7f84000c7f90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f83fc00e870 tx=0x7f83fc00ed30 comp rx=0 tx=0).stop 2024-05-09T05:18:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f8405fd8640 1 -- 172.21.15.1:0/509398458 reap_dead start 2024-05-09T05:18:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f8405fd8640 1 -- 172.21.15.1:0/509398458 reap_dead start 2024-05-09T05:18:42.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 shutdown_connections 2024-05-09T05:18:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.064+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 >> 172.21.15.1:0/509398458 conn(0x7f840006d800 msgr2=0x7f840015a110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.065+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 shutdown_connections 2024-05-09T05:18:42.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.065+0000 7f8407a3b640 1 -- 172.21.15.1:0/509398458 wait complete. 2024-05-09T05:18:42.080 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:18:42.080 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-05-09T05:18:42.080 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:18:42.080 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-05-09T05:18:42.081 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:18:42.081 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:18:42.081 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:18:42.093 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:18:42.093 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:18:42.093 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:18:42.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.272+0000 7f502a6e0640 1 Processor -- start 2024-05-09T05:18:42.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.272+0000 7f502a6e0640 1 -- start start 2024-05-09T05:18:42.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f502a6e0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f50240c85e0 0x7f50240ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.272 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f502a6e0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5024154cc0 con 0x7f50241548b0 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f5023fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f50241548b0 0x7f50240c2be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54916/0 (socket says 172.21.15.1:54916) 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f5023fff640 1 -- 172.21.15.1:0/113066287 learned_addr learned my addr 172.21.15.1:0/113066287 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f5028c7d640 1 --2- 172.21.15.1:0/113066287 >> v2:172.21.15.196:3300/0 conn(0x7f50240c85e0 0x7f50240ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f502a6e0640 1 -- 172.21.15.1:0/113066287 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f50240c98d0 con 0x7f50240c81d0 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f502a6e0640 1 -- 172.21.15.1:0/113066287 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f50240c9680 con 0x7f50240c85e0 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f50237fe640 1 -- 172.21.15.1:0/113066287 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 866914761 0 0) 0x7f50240c98d0 con 0x7f50240c81d0 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.275+0000 7f50237fe640 1 -- 172.21.15.1:0/113066287 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f5010003520 con 0x7f50240c81d0 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.276+0000 7f50237fe640 1 -- 172.21.15.1:0/113066287 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2279970185 0 0) 0x7f5024154cc0 con 0x7f50241548b0 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.276+0000 7f50237fe640 1 -- 172.21.15.1:0/113066287 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f50240c98d0 con 0x7f50241548b0 2024-05-09T05:18:42.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.276+0000 7f5028c7d640 1 -- 172.21.15.1:0/113066287 >> v1:172.21.15.196:6789/0 conn(0x7f50240c81d0 legacy=0x7f502415e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.276+0000 7f5028c7d640 1 -- 172.21.15.1:0/113066287 >> v1:172.21.15.1:6789/0 conn(0x7f50241548b0 legacy=0x7f50240c2be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.276+0000 7f5028c7d640 1 -- 172.21.15.1:0/113066287 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f50240c9bb0 con 0x7f50240c85e0 2024-05-09T05:18:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.277+0000 7f5028c7d640 1 --2- 172.21.15.1:0/113066287 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c85e0 0x7f50240ca6f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f501c009d90 tx=0x7f501c02a6f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=598546dc93cab3a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.277+0000 7f50237fe640 1 -- 172.21.15.1:0/113066287 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c007740 con 0x7f50240c85e0 2024-05-09T05:18:42.275 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.277+0000 7f50237fe640 1 -- 172.21.15.1:0/113066287 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f501c0078e0 con 0x7f50240c85e0 2024-05-09T05:18:42.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.277+0000 7f50237fe640 1 -- 172.21.15.1:0/113066287 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f501c007cb0 con 0x7f50240c85e0 2024-05-09T05:18:42.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.278+0000 7f502a6e0640 1 -- 172.21.15.1:0/113066287 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c85e0 msgr2=0x7f50240ca6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.278+0000 7f502a6e0640 1 --2- 172.21.15.1:0/113066287 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c85e0 0x7f50240ca6f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f501c009d90 tx=0x7f501c02a6f0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.279+0000 7f502a6e0640 1 -- 172.21.15.1:0/113066287 shutdown_connections 2024-05-09T05:18:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.279+0000 7f502a6e0640 1 --2- 172.21.15.1:0/113066287 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c85e0 0x7f50240ca6f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.279+0000 7f502a6e0640 1 -- 172.21.15.1:0/113066287 >> 172.21.15.1:0/113066287 conn(0x7f5024070dd0 msgr2=0x7f50240711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.279+0000 7f502a6e0640 1 -- 172.21.15.1:0/113066287 shutdown_connections 2024-05-09T05:18:42.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.279+0000 7f502a6e0640 1 -- 172.21.15.1:0/113066287 wait complete. 2024-05-09T05:18:42.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.282+0000 7f502a6e0640 1 Processor -- start 2024-05-09T05:18:42.279 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.282+0000 7f502a6e0640 1 -- start start 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f502a6e0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c81d0 0x7f502414d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f502a6e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240c85e0 0x7f50241513f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f502a6e0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50241548b0 0x7f5024143030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f5023fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c81d0 0x7f502414d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f5023fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c81d0 0x7f502414d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:36832/0 (socket says 172.21.15.1:36832) 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f5023fff640 1 -- 172.21.15.1:0/3520920968 learned_addr learned my addr 172.21.15.1:0/3520920968 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f502415e950 con 0x7f50240c85e0 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f502415dd20 con 0x7f50240c81d0 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f502415ead0 con 0x7f50241548b0 2024-05-09T05:18:42.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.284+0000 7f501bfff640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240c85e0 0x7f50241513f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.285+0000 7f5028c7d640 1 -- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50241548b0 msgr2=0x7f5024143030 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.285+0000 7f5028c7d640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50241548b0 0x7f5024143030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.285+0000 7f501bfff640 1 -- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50241548b0 msgr2=0x7f5024143030 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.285+0000 7f501bfff640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50241548b0 0x7f5024143030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.285+0000 7f501bfff640 1 -- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c81d0 msgr2=0x7f502414d970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.285+0000 7f501bfff640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c81d0 0x7f502414d970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.285+0000 7f501bfff640 1 -- 172.21.15.1:0/3520920968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f502414aa80 con 0x7f50240c85e0 2024-05-09T05:18:42.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.285+0000 7f5023fff640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c81d0 0x7f502414d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.286+0000 7f501bfff640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240c85e0 0x7f50241513f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f500c001eb0 tx=0x7f500c00c820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.286+0000 7f50217fa640 1 -- 172.21.15.1:0/3520920968 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c017070 con 0x7f50240c85e0 2024-05-09T05:18:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.286+0000 7f50217fa640 1 -- 172.21.15.1:0/3520920968 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f500c0052d0 con 0x7f50240c85e0 2024-05-09T05:18:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.286+0000 7f50217fa640 1 -- 172.21.15.1:0/3520920968 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f500c003830 con 0x7f50240c85e0 2024-05-09T05:18:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.286+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5024145460 con 0x7f50240c85e0 2024-05-09T05:18:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.286+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f502414c820 con 0x7f50240c85e0 2024-05-09T05:18:42.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.287+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f50240696d0 con 0x7f50240c85e0 2024-05-09T05:18:42.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.289+0000 7f50217fa640 1 -- 172.21.15.1:0/3520920968 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f500c01a070 con 0x7f50240c85e0 2024-05-09T05:18:42.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.289+0000 7f50217fa640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50040762b0 0x7f50040787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.291+0000 7f50217fa640 1 -- 172.21.15.1:0/3520920968 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f500c096720 con 0x7f50240c85e0 2024-05-09T05:18:42.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.292+0000 7f5023fff640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50040762b0 0x7f50040787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:42.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.292+0000 7f5023fff640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50040762b0 0x7f50040787c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5014005880 tx=0x7f5014009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:42.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.298+0000 7f50217fa640 1 -- 172.21.15.1:0/3520920968 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f500c05fa40 con 0x7f50240c85e0 2024-05-09T05:18:42.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.784+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f502414ac50 con 0x7f50240c85e0 2024-05-09T05:18:42.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.784+0000 7f50217fa640 1 -- 172.21.15.1:0/3520920968 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f500c068080 con 0x7f50240c85e0 2024-05-09T05:18:42.787 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:18:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50040762b0 msgr2=0x7f50040787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50040762b0 0x7f50040787c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f5014005880 tx=0x7f5014009040 comp rx=0 tx=0).stop 2024-05-09T05:18:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240c85e0 msgr2=0x7f50241513f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:42.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240c85e0 0x7f50241513f0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f500c001eb0 tx=0x7f500c00c820 comp rx=0 tx=0).stop 2024-05-09T05:18:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 shutdown_connections 2024-05-09T05:18:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f50040762b0 0x7f50040787c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f50241548b0 0x7f5024143030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f50240c85e0 0x7f50241513f0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 --2- 172.21.15.1:0/3520920968 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f50240c81d0 0x7f502414d970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 >> 172.21.15.1:0/3520920968 conn(0x7f5024070dd0 msgr2=0x7f5024148810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 shutdown_connections 2024-05-09T05:18:42.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:42.788+0000 7f502a6e0640 1 -- 172.21.15.1:0/3520920968 wait complete. 2024-05-09T05:18:45.803 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:18:45.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.978+0000 7f66ce670640 1 Processor -- start 2024-05-09T05:18:45.975 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.978+0000 7f66ce670640 1 -- start start 2024-05-09T05:18:45.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.979+0000 7f66ce670640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c815db30 0x7f66c815adf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.979+0000 7f66ce670640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f66c815b330 0x7f66c806ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66ce670640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f66c815a610 con 0x7f66c806d800 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66ce670640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f66c815a960 con 0x7f66c815b330 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66ce670640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f66c80c7730 con 0x7f66c815db30 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c7fff640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c815db30 msgr2=0x7f66c815adf0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c7fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c815db30 0x7f66c815adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c77fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f66c815b330 0x7f66c806ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c77fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f66c815b330 0x7f66c806ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:36848/0 (socket says 172.21.15.1:36848) 2024-05-09T05:18:45.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c77fe640 1 -- 172.21.15.1:0/3882491080 learned_addr learned my addr 172.21.15.1:0/3882491080 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c77fe640 1 -- 172.21.15.1:0/3882491080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c815db30 msgr2=0x7f66c815adf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:45.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c77fe640 1 --2- 172.21.15.1:0/3882491080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c815db30 0x7f66c815adf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c77fe640 1 -- 172.21.15.1:0/3882491080 >> v1:172.21.15.196:6789/0 conn(0x7f66c806d800 legacy=0x7f66c806c6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.980+0000 7f66c77fe640 1 -- 172.21.15.1:0/3882491080 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c815e070 con 0x7f66c815b330 2024-05-09T05:18:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.982+0000 7f66c77fe640 1 --2- 172.21.15.1:0/3882491080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815b330 0x7f66c806ded0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f66c80c5e60 tx=0x7f66bc02a1c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7e987b1a9f51ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.983+0000 7f66c6ffd640 1 -- 172.21.15.1:0/3882491080 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc037070 con 0x7f66c815b330 2024-05-09T05:18:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.983+0000 7f66c6ffd640 1 -- 172.21.15.1:0/3882491080 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66bc032440 con 0x7f66c815b330 2024-05-09T05:18:45.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.983+0000 7f66c6ffd640 1 -- 172.21.15.1:0/3882491080 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc032810 con 0x7f66c815b330 2024-05-09T05:18:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.984+0000 7f66ce670640 1 -- 172.21.15.1:0/3882491080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815b330 msgr2=0x7f66c806ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.984+0000 7f66ce670640 1 --2- 172.21.15.1:0/3882491080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815b330 0x7f66c806ded0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f66c80c5e60 tx=0x7f66bc02a1c0 comp rx=0 tx=0).stop 2024-05-09T05:18:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.984+0000 7f66ce670640 1 -- 172.21.15.1:0/3882491080 shutdown_connections 2024-05-09T05:18:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.984+0000 7f66ce670640 1 --2- 172.21.15.1:0/3882491080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815b330 0x7f66c806ded0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.984+0000 7f66ce670640 1 --2- 172.21.15.1:0/3882491080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c815db30 0x7f66c815adf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.984+0000 7f66ce670640 1 -- 172.21.15.1:0/3882491080 >> 172.21.15.1:0/3882491080 conn(0x7f66c80cea90 msgr2=0x7f66c80ceea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.984+0000 7f66ce670640 1 -- 172.21.15.1:0/3882491080 shutdown_connections 2024-05-09T05:18:45.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.984+0000 7f66ce670640 1 -- 172.21.15.1:0/3882491080 wait complete. 2024-05-09T05:18:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.987+0000 7f66ce670640 1 Processor -- start 2024-05-09T05:18:45.984 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.987+0000 7f66ce670640 1 -- start start 2024-05-09T05:18:45.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.989+0000 7f66ce670640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c806d800 0x7f66c81fac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:45.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66c7fff640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c806d800 msgr2=0x7f66c81fac20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:45.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66c7fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c806d800 0x7f66c81fac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:45.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66ce670640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66c815b330 0x7f66c814d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:45.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66ce670640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815db30 0x7f66c814fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:45.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66ce670640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f66c815e910 con 0x7f66c815b330 2024-05-09T05:18:45.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66c77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66c815b330 0x7f66c814d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:45.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66c77fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66c815b330 0x7f66c814d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:43324/0 (socket says 172.21.15.1:43324) 2024-05-09T05:18:45.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66c77fe640 1 -- 172.21.15.1:0/238551034 learned_addr learned my addr 172.21.15.1:0/238551034 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:45.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f66c815e630 con 0x7f66c815db30 2024-05-09T05:18:45.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.990+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f66c80cb250 con 0x7f66c806d800 2024-05-09T05:18:45.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.991+0000 7f66ccc0d640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815db30 0x7f66c814fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:45.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.991+0000 7f66c77fe640 1 -- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c806d800 msgr2=0x7f66c81fac20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:45.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.991+0000 7f66c77fe640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c806d800 0x7f66c81fac20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.991+0000 7f66c77fe640 1 -- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815db30 msgr2=0x7f66c814fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.991+0000 7f66c77fe640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815db30 0x7f66c814fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.991+0000 7f66c77fe640 1 -- 172.21.15.1:0/238551034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66c81f76d0 con 0x7f66c815b330 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.991+0000 7f66ccc0d640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815db30 0x7f66c814fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.991+0000 7f66c77fe640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66c815b330 0x7f66c814d3c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f66bc000c00 tx=0x7f66bc0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.992+0000 7f66c4ff9640 1 -- 172.21.15.1:0/238551034 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc041070 con 0x7f66c815b330 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.992+0000 7f66c4ff9640 1 -- 172.21.15.1:0/238551034 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f66bc030990 con 0x7f66c815b330 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.992+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66c81f79b0 con 0x7f66c815b330 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.992+0000 7f66c4ff9640 1 -- 172.21.15.1:0/238551034 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66bc037040 con 0x7f66c815b330 2024-05-09T05:18:45.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.992+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66c81fa420 con 0x7f66c815b330 2024-05-09T05:18:45.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.994+0000 7f66c4ff9640 1 -- 172.21.15.1:0/238551034 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f66bc044070 con 0x7f66c815b330 2024-05-09T05:18:45.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.995+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f66c8050870 con 0x7f66c815b330 2024-05-09T05:18:45.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.995+0000 7f66c4ff9640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66b4076300 0x7f66b4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:45.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.995+0000 7f66c4ff9640 1 -- 172.21.15.1:0/238551034 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f66bc0b8270 con 0x7f66c815b330 2024-05-09T05:18:45.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.996+0000 7f66c7fff640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66b4076300 0x7f66b4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:45.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:45.996+0000 7f66c7fff640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66b4076300 0x7f66b4078810 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f66c0003cb0 tx=0x7f66c0009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:46.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.004+0000 7f66c4ff9640 1 -- 172.21.15.1:0/238551034 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f66bc081550 con 0x7f66c815b330 2024-05-09T05:18:46.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.491+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f66c80c7e80 con 0x7f66c815b330 2024-05-09T05:18:46.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.492+0000 7f66c4ff9640 1 -- 172.21.15.1:0/238551034 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f66bc085850 con 0x7f66c815b330 2024-05-09T05:18:46.489 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":94,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:18:46.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66b4076300 msgr2=0x7f66b4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:46.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66b4076300 0x7f66b4078810 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f66c0003cb0 tx=0x7f66c0009270 comp rx=0 tx=0).stop 2024-05-09T05:18:46.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66c815b330 msgr2=0x7f66c814d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:46.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66c815b330 0x7f66c814d3c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f66bc000c00 tx=0x7f66bc0046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:46.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 shutdown_connections 2024-05-09T05:18:46.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f66b4076300 0x7f66b4078810 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f66c815db30 0x7f66c814fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f66c815b330 0x7f66c814d3c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 --2- 172.21.15.1:0/238551034 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f66c806d800 0x7f66c81fac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.495+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 >> 172.21.15.1:0/238551034 conn(0x7f66c80cea90 msgr2=0x7f66c80c9770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.496+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 shutdown_connections 2024-05-09T05:18:46.494 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:46.496+0000 7f66ce670640 1 -- 172.21.15.1:0/238551034 wait complete. 2024-05-09T05:18:49.513 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:18:49.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.690+0000 7f9e0569f640 1 Processor -- start 2024-05-09T05:18:49.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.690+0000 7f9e0569f640 1 -- start start 2024-05-09T05:18:49.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.694+0000 7f9e0569f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 0x7f9e0015af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.694+0000 7f9e0569f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e000c4b90 con 0x7f9e000c4200 2024-05-09T05:18:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.694+0000 7f9e0569f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e000c4e70 con 0x7f9e000c9b00 2024-05-09T05:18:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.694+0000 7f9e0569f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9e00164520 con 0x7f9e000c3b40 2024-05-09T05:18:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.694+0000 7f9dfeffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9e000c4200 0x7f9e001649b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54254/0 (socket says 172.21.15.1:54254) 2024-05-09T05:18:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.694+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 learned_addr learned my addr 172.21.15.1:0/4019427172 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.694+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 msgr2=0x7f9e0015af00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:49.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.694+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 0x7f9e0015af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.895+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 msgr2=0x7f9e0015af00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:49.895+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 0x7f9e0015af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:18:50.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:50.296+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 msgr2=0x7f9e0015af00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:50.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:50.296+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 0x7f9e0015af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:18:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:51.097+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 msgr2=0x7f9e0015af00 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:51.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:51.097+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 0x7f9e0015af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:18:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 msgr2=0x7f9e0015af00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:52.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfcff9640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9e000c3b40 0x7f9e0015af00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 >> v1:172.21.15.196:6789/0 conn(0x7f9e000c9b00 legacy=0x7f9e0016b680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 >> v1:172.21.15.1:6789/0 conn(0x7f9e000c4200 legacy=0x7f9e001649b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfcff9640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfcff9640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9e00164520 con 0x7f9dec006280 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f9dec008e40 con 0x7f9dec0036d0 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:18:52.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.694+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.695+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.695+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:18:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.695+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.695+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:52.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.695+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:52.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.896+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:52.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.896+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:52.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.896+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:18:52.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.896+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:52.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.896+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:52.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.897+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:18:52.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.897+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:18:52.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.897+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:52.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.897+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:52.895 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:52.897+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:18:53.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:18:53.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:53.298+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:18:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:18:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:18:54.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:18:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:54.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:54.099+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:18:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 msgr2=0x7f9dec005ba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:55.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfcff9640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.196:3300/0 conn(0x7f9dec0036d0 0x7f9dec005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfcff9640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dff7fe640 1 -- 172.21.15.1:0/4019427172 reap_dead start 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfcff9640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9e000c5150 con 0x7f9e000c4200 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9dec011230 con 0x7f9e000c9b00 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfcff9640 1 -- 172.21.15.1:0/4019427172 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9e00164520 con 0x7f9dec006280 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.694+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.695+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.695+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.695+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:55.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.695+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:55.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.896+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:55.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.896+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:18:55.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.896+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.896+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:55.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:55.896+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:18:56.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.297+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:56.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.297+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:18:56.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.297+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:18:56.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.297+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:18:56.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.297+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:18:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 908379877 0 0) 0x7f9dec011230 con 0x7f9e000c9b00 2024-05-09T05:18:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f9de8003520 con 0x7f9e000c9b00 2024-05-09T05:18:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3615811118 0 0) 0x7f9de8003520 con 0x7f9e000c9b00 2024-05-09T05:18:56.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 msgr2=0x7f9dec008750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 --2- 172.21.15.1:0/4019427172 >> v2:172.21.15.1:3300/0 conn(0x7f9dec006280 0x7f9dec008750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 >> v1:172.21.15.1:6789/0 conn(0x7f9e000c4200 legacy=0x7f9dec00b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dec012520 con 0x7f9e000c9b00 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f9df0001620 con 0x7f9e000c9b00 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 >> v1:172.21.15.196:6789/0 conn(0x7f9e000c9b00 legacy=0x7f9dec00ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80039d0 0x7f9de8003e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de8004800 0x7f9de8004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de80052a0 0x7f9de800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9e00164520 con 0x7f9de80052a0 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9de800c230 con 0x7f9de8004800 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9de800c430 con 0x7f9de80039d0 2024-05-09T05:18:56.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfeffd640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80039d0 msgr2=0x7f9de8003e60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80039d0 0x7f9de8003e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.979+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de8004800 0x7f9de8004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.980+0000 7f9dff7fe640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de80052a0 0x7f9de800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.980+0000 7f9dff7fe640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80039d0 msgr2=0x7f9de8003e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:18:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.980+0000 7f9dff7fe640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80039d0 0x7f9de8003e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.980+0000 7f9dff7fe640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de8004800 msgr2=0x7f9de8004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.980+0000 7f9dff7fe640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9de8004800 0x7f9de8004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:56.979 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.980+0000 7f9dff7fe640 1 -- 172.21.15.1:0/4019427172 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9dec012520 con 0x7f9de80052a0 2024-05-09T05:18:56.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.981+0000 7f9dff7fe640 1 -- 172.21.15.1:0/4019427172 reap_dead start 2024-05-09T05:18:56.980 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.982+0000 7f9dff7fe640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de80052a0 0x7f9de800b9c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9e00072e10 tx=0x7f9de8005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5471f7ed3ad1905f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:56.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.989+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df00079a0 con 0x7f9de80052a0 2024-05-09T05:18:56.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.989+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df002e070 con 0x7f9de80052a0 2024-05-09T05:18:56.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.989+0000 7f9dfdffb640 1 -- 172.21.15.1:0/4019427172 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df0007070 con 0x7f9de80052a0 2024-05-09T05:18:56.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.991+0000 7f9e0569f640 1 -- 172.21.15.1:0/4019427172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de80052a0 msgr2=0x7f9de800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:56.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.991+0000 7f9e0569f640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de80052a0 0x7f9de800b9c0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f9e00072e10 tx=0x7f9de8005830 comp rx=0 tx=0).stop 2024-05-09T05:18:56.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.991+0000 7f9e0569f640 1 -- 172.21.15.1:0/4019427172 shutdown_connections 2024-05-09T05:18:56.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.991+0000 7f9e0569f640 1 --2- 172.21.15.1:0/4019427172 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9de80052a0 0x7f9de800b9c0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:56.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.991+0000 7f9e0569f640 1 -- 172.21.15.1:0/4019427172 >> 172.21.15.1:0/4019427172 conn(0x7f9e000cec70 msgr2=0x7f9e000cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:56.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.991+0000 7f9e0569f640 1 -- 172.21.15.1:0/4019427172 shutdown_connections 2024-05-09T05:18:56.989 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.991+0000 7f9e0569f640 1 -- 172.21.15.1:0/4019427172 wait complete. 2024-05-09T05:18:56.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.994+0000 7f9e0569f640 1 Processor -- start 2024-05-09T05:18:56.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.995+0000 7f9e0569f640 1 -- start start 2024-05-09T05:18:56.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.996+0000 7f9e0569f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80052a0 0x7f9e001600e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:56.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9e0569f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e0015f410 0x7f9e001610d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9e0569f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e00161610 0x7f9e0014d980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9dfeffd640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80052a0 msgr2=0x7f9e001600e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9dfeffd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80052a0 0x7f9e001600e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9dfe7fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e0015f410 0x7f9e001610d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9dff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e00161610 0x7f9e0014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9e0569f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000c4750 con 0x7f9e00161610 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9e0569f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9e000c3f50 con 0x7f9e0015f410 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9e0569f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9e000c48d0 con 0x7f9de80052a0 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9dff7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e00161610 0x7f9e0014d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49140/0 (socket says 172.21.15.1:49140) 2024-05-09T05:18:56.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.997+0000 7f9dff7fe640 1 -- 172.21.15.1:0/3600207521 learned_addr learned my addr 172.21.15.1:0/3600207521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.998+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80052a0 msgr2=0x7f9e001600e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.998+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80052a0 0x7f9e001600e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.998+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/3600207521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e00161610 msgr2=0x7f9e0014d980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.998+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e00161610 0x7f9e0014d980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.998+0000 7f9dfe7fc640 1 -- 172.21.15.1:0/3600207521 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e00160620 con 0x7f9e0015f410 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.998+0000 7f9dff7fe640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e00161610 0x7f9e0014d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.998+0000 7f9dfe7fc640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e0015f410 0x7f9e001610d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9df400cdf0 tx=0x7f9df400a7b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.999+0000 7f9de77fe640 1 -- 172.21.15.1:0/3600207521 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df400ce20 con 0x7f9e0015f410 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.999+0000 7f9de77fe640 1 -- 172.21.15.1:0/3600207521 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9df4024430 con 0x7f9e0015f410 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.999+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e00161a20 con 0x7f9e0015f410 2024-05-09T05:18:56.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.999+0000 7f9de77fe640 1 -- 172.21.15.1:0/3600207521 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9df4024820 con 0x7f9e0015f410 2024-05-09T05:18:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:56.999+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e001f7580 con 0x7f9e0015f410 2024-05-09T05:18:56.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.000+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e00050870 con 0x7f9e0015f410 2024-05-09T05:18:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.002+0000 7f9de77fe640 1 -- 172.21.15.1:0/3600207521 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9df4017050 con 0x7f9e0015f410 2024-05-09T05:18:56.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.002+0000 7f9de77fe640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9dec0772d0 0x7f9dec0797e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.002+0000 7f9de77fe640 1 -- 172.21.15.1:0/3600207521 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9df4099730 con 0x7f9e0015f410 2024-05-09T05:18:57.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.003+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9dec0772d0 0x7f9dec0797e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.004+0000 7f9dfeffd640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9dec0772d0 0x7f9dec0797e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9df8009010 tx=0x7f9df8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.009+0000 7f9de77fe640 1 -- 172.21.15.1:0/3600207521 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9df4062f70 con 0x7f9e0015f410 2024-05-09T05:18:57.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.502+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9e00150c70 con 0x7f9e0015f410 2024-05-09T05:18:57.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.503+0000 7f9de77fe640 1 -- 172.21.15.1:0/3600207521 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1554 (secure 0 0 0) 0x7f9df4062870 con 0x7f9e0015f410 2024-05-09T05:18:57.500 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":98,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9dec0772d0 msgr2=0x7f9dec0797e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9dec0772d0 0x7f9dec0797e0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9df8009010 tx=0x7f9df8009040 comp rx=0 tx=0).stop 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e0015f410 msgr2=0x7f9e001610d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e0015f410 0x7f9e001610d0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9df400cdf0 tx=0x7f9df400a7b0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 shutdown_connections 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9dec0772d0 0x7f9dec0797e0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9e0015f410 0x7f9e001610d0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9e00161610 0x7f9e0014d980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 --2- 172.21.15.1:0/3600207521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9de80052a0 0x7f9e001600e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 >> 172.21.15.1:0/3600207521 conn(0x7f9e000cec70 msgr2=0x7f9e00071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:57.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 shutdown_connections 2024-05-09T05:18:57.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.505+0000 7f9e0569f640 1 -- 172.21.15.1:0/3600207521 wait complete. 2024-05-09T05:18:57.522 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:18:57.522 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:18:57.523 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:18:57.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.700+0000 7efeaa734640 1 Processor -- start 2024-05-09T05:18:57.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.700+0000 7efeaa734640 1 -- start start 2024-05-09T05:18:57.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.702+0000 7efeaa734640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efea40c8870 0x7efea415be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.702+0000 7efeaa734640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efea40c6eb0 0x7efea40c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efeaa734640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efea4158970 con 0x7efea4072d00 2024-05-09T05:18:57.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efeaa734640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7efea40c8c80 con 0x7efea40c8870 2024-05-09T05:18:57.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efea37fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efea40c6eb0 0x7efea40c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efeaa734640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7efea4159230 con 0x7efea40c6eb0 2024-05-09T05:18:57.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efea37fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efea40c6eb0 0x7efea40c72c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33364/0 (socket says 172.21.15.1:33364) 2024-05-09T05:18:57.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efea37fe640 1 -- 172.21.15.1:0/2249245414 learned_addr learned my addr 172.21.15.1:0/2249245414 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:57.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efea3fff640 1 --2- 172.21.15.1:0/2249245414 >> v2:172.21.15.1:3300/0 conn(0x7efea40c8870 0x7efea415be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efea37fe640 1 -- 172.21.15.1:0/2249245414 >> v2:172.21.15.1:3300/0 conn(0x7efea40c8870 msgr2=0x7efea415be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:18:57.701 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.703+0000 7efea37fe640 1 --2- 172.21.15.1:0/2249245414 >> v2:172.21.15.1:3300/0 conn(0x7efea40c8870 0x7efea415be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.704+0000 7efea37fe640 1 -- 172.21.15.1:0/2249245414 >> v1:172.21.15.1:6789/0 conn(0x7efea4072d00 legacy=0x7efea4073110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.704+0000 7efea37fe640 1 -- 172.21.15.1:0/2249245414 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea40c5690 con 0x7efea40c6eb0 2024-05-09T05:18:57.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.704+0000 7efea3fff640 1 --2- 172.21.15.1:0/2249245414 >> v2:172.21.15.1:3300/0 conn(0x7efea40c8870 0x7efea415be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:18:57.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.705+0000 7efea37fe640 1 --2- 172.21.15.1:0/2249245414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c6eb0 0x7efea40c72c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efea40c9140 tx=0x7efe9802a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72390387314ad76e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.705+0000 7efea2ffd640 1 -- 172.21.15.1:0/2249245414 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe98037070 con 0x7efea40c6eb0 2024-05-09T05:18:57.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.705+0000 7efea2ffd640 1 -- 172.21.15.1:0/2249245414 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe98032440 con 0x7efea40c6eb0 2024-05-09T05:18:57.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.705+0000 7efea2ffd640 1 -- 172.21.15.1:0/2249245414 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe98032810 con 0x7efea40c6eb0 2024-05-09T05:18:57.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.706+0000 7efeaa734640 1 -- 172.21.15.1:0/2249245414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c6eb0 msgr2=0x7efea40c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.706+0000 7efeaa734640 1 --2- 172.21.15.1:0/2249245414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c6eb0 0x7efea40c72c0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efea40c9140 tx=0x7efe9802a270 comp rx=0 tx=0).stop 2024-05-09T05:18:57.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.706+0000 7efeaa734640 1 -- 172.21.15.1:0/2249245414 shutdown_connections 2024-05-09T05:18:57.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.706+0000 7efeaa734640 1 --2- 172.21.15.1:0/2249245414 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c6eb0 0x7efea40c72c0 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efea40c9140 tx=0x7efe9802a270 comp rx=0 tx=0).stop 2024-05-09T05:18:57.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.706+0000 7efeaa734640 1 --2- 172.21.15.1:0/2249245414 >> v2:172.21.15.1:3300/0 conn(0x7efea40c8870 0x7efea415be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.706+0000 7efeaa734640 1 -- 172.21.15.1:0/2249245414 >> 172.21.15.1:0/2249245414 conn(0x7efea4070f40 msgr2=0x7efea406c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:57.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.708+0000 7efeaa734640 1 -- 172.21.15.1:0/2249245414 shutdown_connections 2024-05-09T05:18:57.706 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.708+0000 7efeaa734640 1 -- 172.21.15.1:0/2249245414 wait complete. 2024-05-09T05:18:57.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.711+0000 7efeaa734640 1 Processor -- start 2024-05-09T05:18:57.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.711+0000 7efeaa734640 1 -- start start 2024-05-09T05:18:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.713+0000 7efeaa734640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efea4072d00 0x7efea413f9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.713+0000 7efeaa734640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c8870 0x7efea415acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.713+0000 7efea3fff640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efea4072d00 msgr2=0x7efea413f9a0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.713+0000 7efea3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efea4072d00 0x7efea413f9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.713+0000 7efeaa734640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415a060 0x7efea413b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.713+0000 7efeaa734640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efea4158be0 con 0x7efea415a060 2024-05-09T05:18:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.713+0000 7efeaa734640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efea40c7800 con 0x7efea40c8870 2024-05-09T05:18:57.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.713+0000 7efeaa734640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efea4158ec0 con 0x7efea4072d00 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea37fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c8870 0x7efea415acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea37fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c8870 0x7efea415acf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33372/0 (socket says 172.21.15.1:33372) 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea37fe640 1 -- 172.21.15.1:0/2593170186 learned_addr learned my addr 172.21.15.1:0/2593170186 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea8cd1640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415a060 0x7efea413b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea37fe640 1 -- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efea4072d00 msgr2=0x7efea413f9a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea37fe640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efea4072d00 0x7efea413f9a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea37fe640 1 -- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415a060 msgr2=0x7efea413b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea37fe640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415a060 0x7efea413b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea37fe640 1 -- 172.21.15.1:0/2593170186 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efea413bcc0 con 0x7efea40c8870 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.714+0000 7efea8cd1640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415a060 0x7efea413b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.715+0000 7efea37fe640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c8870 0x7efea415acf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe98035040 tx=0x7efe980046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.715+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe9803f070 con 0x7efea40c8870 2024-05-09T05:18:57.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.715+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efe98004220 con 0x7efea40c8870 2024-05-09T05:18:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.715+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efea41ee930 con 0x7efea40c8870 2024-05-09T05:18:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.715+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe98032e30 con 0x7efea40c8870 2024-05-09T05:18:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.715+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efea41eeef0 con 0x7efea40c8870 2024-05-09T05:18:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.717+0000 7efeaa734640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415b8b0 0x7efea415bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.717+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efea4140230 con 0x7efea415b8b0 2024-05-09T05:18:57.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.717+0000 7efea3fff640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415b8b0 0x7efea415bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.717+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efe98039600 con 0x7efea40c8870 2024-05-09T05:18:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.718+0000 7efea3fff640 1 -- 172.21.15.1:0/2593170186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efea413ed00 con 0x7efea415b8b0 2024-05-09T05:18:57.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.718+0000 7efea3fff640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415b8b0 0x7efea415bcc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efe9400c6f0 tx=0x7efe9400cbb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.719+0000 7efea0ff9640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efe900762d0 0x7efe900787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.719+0000 7efea8cd1640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efe900762d0 0x7efe900787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.719+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7efe980b96d0 con 0x7efea40c8870 2024-05-09T05:18:57.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.719+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe94004230 con 0x7efea415b8b0 2024-05-09T05:18:57.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.719+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efe940043d0 con 0x7efea415b8b0 2024-05-09T05:18:57.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.719+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415b8b0 msgr2=0x7efea415bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.719+0000 7efea0ff9640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415b8b0 0x7efea415bcc0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7efe9400c6f0 tx=0x7efe9400cbb0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.720+0000 7efea8cd1640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efe900762d0 0x7efe900787e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efea4157ea0 tx=0x7efe9c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.720+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415b8b0 msgr2=0x7efea415bcc0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:57.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.720+0000 7efea0ff9640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea415b8b0 0x7efea415bcc0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.732+0000 7efeaa734640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea41f1c60 0x7efea41f2070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.732+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efea41ef6f0 con 0x7efea41f1c60 2024-05-09T05:18:57.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.733+0000 7efea3fff640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea41f1c60 0x7efea41f2070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.731 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.733+0000 7efea3fff640 1 -- 172.21.15.1:0/2593170186 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efea4050870 con 0x7efea41f1c60 2024-05-09T05:18:57.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.734+0000 7efea3fff640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea41f1c60 0x7efea41f2070 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe9400c6f0 tx=0x7efe94004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.734+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe94004230 con 0x7efea41f1c60 2024-05-09T05:18:57.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5052 (secure 0 0 0) 0x7efe94003cd0 con 0x7efea41f1c60 2024-05-09T05:18:57.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea41f1c60 msgr2=0x7efea41f2070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efea0ff9640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea41f1c60 0x7efea41f2070 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7efe9400c6f0 tx=0x7efe94004730 comp rx=0 tx=0).stop 2024-05-09T05:18:57.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efea0ff9640 1 -- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea41f1c60 msgr2=0x7efea41f2070 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:57.733 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efea0ff9640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efea41f1c60 0x7efea41f2070 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.733 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:57.733 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:18:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 98, 2024-05-09T05:18:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 339929, 2024-05-09T05:18:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:18:57.734 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:57.735 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:57.736 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:57.737 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:57.738 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.739 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:57.740 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.741 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:57.742 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.743 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:57.744 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.745 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.746 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.747 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.747 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:57.747 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efe900762d0 msgr2=0x7efe900787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efeaa734640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efe900762d0 0x7efe900787e0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7efea4157ea0 tx=0x7efe9c009040 comp rx=0 tx=0).stop 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efea8cd1640 1 -- 172.21.15.1:0/2593170186 reap_dead start 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c8870 msgr2=0x7efea415acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.735+0000 7efeaa734640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c8870 0x7efea415acf0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7efe98035040 tx=0x7efe980046c0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.736+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 shutdown_connections 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.736+0000 7efeaa734640 1 --2- 172.21.15.1:0/2593170186 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efea40c8870 0x7efea415acf0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.736+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 >> 172.21.15.1:0/2593170186 conn(0x7efea4070f40 msgr2=0x7efea40cc0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.736+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 shutdown_connections 2024-05-09T05:18:57.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.736+0000 7efeaa734640 1 -- 172.21.15.1:0/2593170186 wait complete. 2024-05-09T05:18:57.749 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:18:57.749 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:18:57.750 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:18:57.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.932+0000 7f4565727640 1 Processor -- start 2024-05-09T05:18:57.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.932+0000 7f4565727640 1 -- start start 2024-05-09T05:18:57.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.934+0000 7f4565727640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f45600c4af0 0x7f4560153490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.934+0000 7f4565727640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f45600c3be0 con 0x7f45600c46e0 2024-05-09T05:18:57.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.934+0000 7f4565727640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f456006e890 con 0x7f45600c89a0 2024-05-09T05:18:57.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.934+0000 7f4565727640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f45600cb040 con 0x7f45600c4af0 2024-05-09T05:18:57.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.934+0000 7f455e7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f45600c46e0 0x7f4560159650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:54278/0 (socket says 172.21.15.1:54278) 2024-05-09T05:18:57.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.935+0000 7f455e7fc640 1 -- 172.21.15.1:0/2029632487 learned_addr learned my addr 172.21.15.1:0/2029632487 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:18:57.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.935+0000 7f455f7fe640 1 --2- 172.21.15.1:0/2029632487 >> v2:172.21.15.196:3300/0 conn(0x7f45600c4af0 0x7f4560153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.935+0000 7f455dffb640 1 -- 172.21.15.1:0/2029632487 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3145668807 0 0) 0x7f4550001110 con 0x7f45600c46e0 2024-05-09T05:18:57.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.935+0000 7f455dffb640 1 -- 172.21.15.1:0/2029632487 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4548003520 con 0x7f45600c46e0 2024-05-09T05:18:57.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.935+0000 7f455f7fe640 1 -- 172.21.15.1:0/2029632487 >> v1:172.21.15.196:6789/0 conn(0x7f45600c89a0 legacy=0x7f45600ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.935+0000 7f455dffb640 1 -- 172.21.15.1:0/2029632487 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2428813053 0 0) 0x7f456006e890 con 0x7f45600c89a0 2024-05-09T05:18:57.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.935+0000 7f455f7fe640 1 -- 172.21.15.1:0/2029632487 >> v1:172.21.15.1:6789/0 conn(0x7f45600c46e0 legacy=0x7f4560159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.935+0000 7f455f7fe640 1 -- 172.21.15.1:0/2029632487 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f456015fee0 con 0x7f45600c4af0 2024-05-09T05:18:57.934 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.937+0000 7f455f7fe640 1 --2- 172.21.15.1:0/2029632487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c4af0 0x7f4560153490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4558009d90 tx=0x7f455802a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5d0c625abbea6559 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.937+0000 7f455dffb640 1 -- 172.21.15.1:0/2029632487 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558007740 con 0x7f45600c4af0 2024-05-09T05:18:57.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.937+0000 7f455dffb640 1 -- 172.21.15.1:0/2029632487 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45580078e0 con 0x7f45600c4af0 2024-05-09T05:18:57.935 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.937+0000 7f455dffb640 1 -- 172.21.15.1:0/2029632487 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4558007cb0 con 0x7f45600c4af0 2024-05-09T05:18:57.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.939+0000 7f4565727640 1 -- 172.21.15.1:0/2029632487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c4af0 msgr2=0x7f4560153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.939+0000 7f4565727640 1 --2- 172.21.15.1:0/2029632487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c4af0 0x7f4560153490 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f4558009d90 tx=0x7f455802a660 comp rx=0 tx=0).stop 2024-05-09T05:18:57.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.939+0000 7f4565727640 1 -- 172.21.15.1:0/2029632487 shutdown_connections 2024-05-09T05:18:57.936 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.939+0000 7f4565727640 1 --2- 172.21.15.1:0/2029632487 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c4af0 0x7f4560153490 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.939+0000 7f4565727640 1 -- 172.21.15.1:0/2029632487 >> 172.21.15.1:0/2029632487 conn(0x7f45600747b0 msgr2=0x7f45600ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:57.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.939+0000 7f4565727640 1 -- 172.21.15.1:0/2029632487 shutdown_connections 2024-05-09T05:18:57.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.939+0000 7f4565727640 1 -- 172.21.15.1:0/2029632487 wait complete. 2024-05-09T05:18:57.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.942+0000 7f4565727640 1 Processor -- start 2024-05-09T05:18:57.939 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.942+0000 7f4565727640 1 -- start start 2024-05-09T05:18:57.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f4565727640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45600c46e0 0x7f45601efc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f4565727640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c4af0 0x7f4560154b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f4565727640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f45600c89a0 0x7f4560147480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f4565727640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f45600c4140 con 0x7f45600c46e0 2024-05-09T05:18:57.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f4565727640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f45600c9290 con 0x7f45600c4af0 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f4565727640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f45600c94e0 con 0x7f45600c89a0 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f455f7fe640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f45600c89a0 msgr2=0x7f4560147480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f455f7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f45600c89a0 0x7f4560147480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f455effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45600c46e0 0x7f45601efc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f455effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45600c46e0 0x7f45601efc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49196/0 (socket says 172.21.15.1:49196) 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.944+0000 7f455effd640 1 -- 172.21.15.1:0/2578141842 learned_addr learned my addr 172.21.15.1:0/2578141842 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.945+0000 7f455effd640 1 -- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f45600c89a0 msgr2=0x7f4560147480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.945+0000 7f455effd640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f45600c89a0 0x7f4560147480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.945+0000 7f455effd640 1 -- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c4af0 msgr2=0x7f4560154b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.945+0000 7f455effd640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c4af0 0x7f4560154b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.945+0000 7f455effd640 1 -- 172.21.15.1:0/2578141842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45600c6e80 con 0x7f45600c46e0 2024-05-09T05:18:57.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.945+0000 7f455effd640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45600c46e0 0x7f45601efc30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f455400c3a0 tx=0x7f455400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.946+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4554017070 con 0x7f45600c46e0 2024-05-09T05:18:57.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.946+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f45540035f0 con 0x7f45600c46e0 2024-05-09T05:18:57.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.946+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45540039e0 con 0x7f45600c46e0 2024-05-09T05:18:57.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.946+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4560154210 con 0x7f45600c46e0 2024-05-09T05:18:57.945 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.946+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4560155210 con 0x7f45600c46e0 2024-05-09T05:18:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.948+0000 7f4565727640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4560145180 0x7f4560145590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.948+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f45601554d0 con 0x7f4560145180 2024-05-09T05:18:57.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.949+0000 7f455e7fc640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4560145180 0x7f4560145590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.949+0000 7f455e7fc640 1 -- 172.21.15.1:0/2578141842 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4560153fc0 con 0x7f4560145180 2024-05-09T05:18:57.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.950+0000 7f455e7fc640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4560145180 0x7f4560145590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f45600725c0 tx=0x7f455000cb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.950+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f455401a070 con 0x7f45600c46e0 2024-05-09T05:18:57.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.950+0000 7f45477fe640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f454c076300 0x7f454c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.950+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4554096910 con 0x7f45600c46e0 2024-05-09T05:18:57.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.951+0000 7f455f7fe640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f454c076300 0x7f454c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.951+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550017070 con 0x7f4560145180 2024-05-09T05:18:57.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.951+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f455000ce50 con 0x7f4560145180 2024-05-09T05:18:57.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.951+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4560145180 msgr2=0x7f4560145590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.951+0000 7f45477fe640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4560145180 0x7f4560145590 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f45600725c0 tx=0x7f455000cb10 comp rx=0 tx=0).stop 2024-05-09T05:18:57.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.951+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4560145180 msgr2=0x7f4560145590 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:57.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.951+0000 7f45477fe640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4560145180 0x7f4560145590 secure :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f45600725c0 tx=0x7f455000cb10 comp rx=0 tx=0).stop 2024-05-09T05:18:57.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.953+0000 7f455f7fe640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f454c076300 0x7f454c078810 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4558034000 tx=0x7f455802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.962+0000 7f4565727640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c7640 0x7f45600c7a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:18:57.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.962+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f45600b4850 con 0x7f45600c7640 2024-05-09T05:18:57.961 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.963+0000 7f455e7fc640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c7640 0x7f45600c7a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:18:57.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.964+0000 7f455e7fc640 1 -- 172.21.15.1:0/2578141842 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f4560050870 con 0x7f45600c7640 2024-05-09T05:18:57.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.964+0000 7f455e7fc640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c7640 0x7f45600c7a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f45600725c0 tx=0x7f45500012e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:18:57.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4550017040 con 0x7f45600c7640 2024-05-09T05:18:57.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4539 (secure 0 0 0) 0x7f455001a6c0 con 0x7f45600c7640 2024-05-09T05:18:57.962 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c7640 msgr2=0x7f45600c7a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f45477fe640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c7640 0x7f45600c7a50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f45600725c0 tx=0x7f45500012e0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f45477fe640 1 -- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c7640 msgr2=0x7f45600c7a50 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:18:57.963 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f45477fe640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f45600c7640 0x7f45600c7a50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.965 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:18:57.965 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 98, 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 88369, 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:18:57.966 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:57.967 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:18:57.968 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:18:57.969 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:18:57.970 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.971 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.972 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:57.973 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:18:57.974 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:18:57.975 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:18:57.976 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:18:57.977 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:18:57.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f454c076300 msgr2=0x7f454c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f4565727640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f454c076300 0x7f454c078810 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f4558034000 tx=0x7f455802f000 comp rx=0 tx=0).stop 2024-05-09T05:18:57.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45600c46e0 msgr2=0x7f45601efc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:18:57.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f4565727640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45600c46e0 0x7f45601efc30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f455400c3a0 tx=0x7f455400c860 comp rx=0 tx=0).stop 2024-05-09T05:18:57.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.965+0000 7f455f7fe640 1 -- 172.21.15.1:0/2578141842 reap_dead start 2024-05-09T05:18:57.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.966+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 shutdown_connections 2024-05-09T05:18:57.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.966+0000 7f4565727640 1 --2- 172.21.15.1:0/2578141842 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f45600c46e0 0x7f45601efc30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:18:57.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.966+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 >> 172.21.15.1:0/2578141842 conn(0x7f45600747b0 msgr2=0x7f45600ca930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:18:57.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.966+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 shutdown_connections 2024-05-09T05:18:57.978 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:18:57.966+0000 7f4565727640 1 -- 172.21.15.1:0/2578141842 wait complete. 2024-05-09T05:18:57.986 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:18:57.986 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:19:05.060 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:19:05.064 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:19:17.987 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:19:17.987 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-05-09T05:19:17.988 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T05:19:17.988 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T05:19:17.992 INFO:tasks.ceph.mon.c:Started 2024-05-09T05:19:17.993 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:19:17.993 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:19:18.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.174+0000 7f4b9a31f640 1 Processor -- start 2024-05-09T05:19:18.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.175+0000 7f4b9a31f640 1 -- start start 2024-05-09T05:19:18.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.176+0000 7f4b9a31f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f4b9415c8b0 0x7f4b940cc780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:18.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.176+0000 7f4b9a31f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 0x7f4b940cccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:18.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.176+0000 7f4b9a31f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f4b940c8350 con 0x7f4b940cd280 2024-05-09T05:19:18.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.176+0000 7f4b9a31f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f4b940cdf30 con 0x7f4b9415c8b0 2024-05-09T05:19:18.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.176+0000 7f4b9a31f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4b9415ce60 con 0x7f4b94153700 2024-05-09T05:19:18.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.176+0000 7f4b93fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f4b9415c8b0 0x7f4b940cc780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:18.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b988bc640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f4b940cd280 0x7f4b940ce6a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:48800/0 (socket says 172.21.15.1:48800) 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b988bc640 1 -- 172.21.15.1:0/734171561 learned_addr learned my addr 172.21.15.1:0/734171561 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b937fe640 1 --2- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 0x7f4b940cccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b937fe640 1 -- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 msgr2=0x7f4b940cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b937fe640 1 -- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 msgr2=0x7f4b940cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b937fe640 1 --2- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 0x7f4b940cccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b93fff640 1 -- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 msgr2=0x7f4b940cccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b937fe640 1 --2- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 0x7f4b940cccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b93fff640 1 --2- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 0x7f4b940cccc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b93fff640 1 -- 172.21.15.1:0/734171561 >> v1:172.21.15.196:6789/0 conn(0x7f4b940cd280 legacy=0x7f4b940ce6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.177+0000 7f4b93fff640 1 -- 172.21.15.1:0/734171561 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b9415d140 con 0x7f4b9415c8b0 2024-05-09T05:19:18.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.179+0000 7f4b93fff640 1 --2- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b9415c8b0 0x7f4b940cc780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4b840097a0 tx=0x7f4b8402a180 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df936dc191cb75e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.179+0000 7f4b92ffd640 1 -- 172.21.15.1:0/734171561 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84037070 con 0x7f4b9415c8b0 2024-05-09T05:19:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.179+0000 7f4b92ffd640 1 -- 172.21.15.1:0/734171561 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b84032440 con 0x7f4b9415c8b0 2024-05-09T05:19:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.179+0000 7f4b92ffd640 1 -- 172.21.15.1:0/734171561 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b84032810 con 0x7f4b9415c8b0 2024-05-09T05:19:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.181+0000 7f4b9a31f640 1 -- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b9415c8b0 msgr2=0x7f4b940cc780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.181+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b9415c8b0 0x7f4b940cc780 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4b840097a0 tx=0x7f4b8402a180 comp rx=0 tx=0).stop 2024-05-09T05:19:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.181+0000 7f4b9a31f640 1 -- 172.21.15.1:0/734171561 shutdown_connections 2024-05-09T05:19:18.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.181+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b94153700 0x7f4b940cccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.181+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/734171561 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b9415c8b0 0x7f4b940cc780 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.181+0000 7f4b9a31f640 1 -- 172.21.15.1:0/734171561 >> 172.21.15.1:0/734171561 conn(0x7f4b94068650 msgr2=0x7f4b94068a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:18.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.181+0000 7f4b9a31f640 1 -- 172.21.15.1:0/734171561 shutdown_connections 2024-05-09T05:19:18.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.181+0000 7f4b9a31f640 1 -- 172.21.15.1:0/734171561 wait complete. 2024-05-09T05:19:18.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.184+0000 7f4b9a31f640 1 Processor -- start 2024-05-09T05:19:18.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.185+0000 7f4b9a31f640 1 -- start start 2024-05-09T05:19:18.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.187+0000 7f4b9a31f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 0x7f4b941e71c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:18.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.187+0000 7f4b9a31f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b94153700 0x7f4b9414b1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:18.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.187+0000 7f4b9a31f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b9415c8b0 0x7f4b9414f750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.187+0000 7f4b93fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 0x7f4b941e71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.187+0000 7f4b93fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 0x7f4b941e71c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60394/0 (socket says 172.21.15.1:60394) 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b937fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b94153700 0x7f4b9414b1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b937fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b94153700 0x7f4b9414b1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:48932/0 (socket says 172.21.15.1:48932) 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b988bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b9415c8b0 0x7f4b9414f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b988bc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b9415c8b0 0x7f4b9414f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:44234/0 (socket says 172.21.15.1:44234) 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b9a31f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f4b940cedb0 con 0x7f4b9415c8b0 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b9a31f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f4b9415d700 con 0x7f4b94153700 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b9a31f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f4b940cef30 con 0x7f4b940cd280 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b93fff640 1 -- 172.21.15.1:0/1893669459 learned_addr learned my addr 172.21.15.1:0/1893669459 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b93fff640 1 -- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 msgr2=0x7f4b941e71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b93fff640 1 -- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 msgr2=0x7f4b941e71c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:19:18.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b93fff640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 0x7f4b941e71c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b93fff640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 0x7f4b941e71c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b937fe640 1 -- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 msgr2=0x7f4b941e71c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b937fe640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 0x7f4b941e71c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b937fe640 1 -- 172.21.15.1:0/1893669459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b9415c8b0 msgr2=0x7f4b9414f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b937fe640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b9415c8b0 0x7f4b9414f750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b937fe640 1 -- 172.21.15.1:0/1893669459 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b941e7700 con 0x7f4b94153700 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.188+0000 7f4b988bc640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b9415c8b0 0x7f4b9414f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.189+0000 7f4b937fe640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b94153700 0x7f4b9414b1e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00d5d0 tx=0x7f4b7c00da90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:18.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.190+0000 7f4b90ff9640 1 -- 172.21.15.1:0/1893669459 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0125f0 con 0x7f4b94153700 2024-05-09T05:19:18.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.190+0000 7f4b90ff9640 1 -- 172.21.15.1:0/1893669459 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4b7c019070 con 0x7f4b94153700 2024-05-09T05:19:18.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.190+0000 7f4b90ff9640 1 -- 172.21.15.1:0/1893669459 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7c0129d0 con 0x7f4b94153700 2024-05-09T05:19:18.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.190+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b94154070 con 0x7f4b94153700 2024-05-09T05:19:18.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.190+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b941f30b0 con 0x7f4b94153700 2024-05-09T05:19:18.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.192+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b78005290 con 0x7f4b94153700 2024-05-09T05:19:18.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.193+0000 7f4b90ff9640 1 -- 172.21.15.1:0/1893669459 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f4b7c012b70 con 0x7f4b94153700 2024-05-09T05:19:18.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.193+0000 7f4b90ff9640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4b74076350 0x7f4b74078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:18.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.194+0000 7f4b90ff9640 1 -- 172.21.15.1:0/1893669459 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f4b7c097870 con 0x7f4b94153700 2024-05-09T05:19:18.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.194+0000 7f4b93fff640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4b74076350 0x7f4b74078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:18.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.195+0000 7f4b93fff640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4b74076350 0x7f4b74078860 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009560 tx=0x7f4b8402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:18.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.199+0000 7f4b90ff9640 1 -- 172.21.15.1:0/1893669459 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f4b7c061230 con 0x7f4b94153700 2024-05-09T05:19:18.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.682+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4b78005850 con 0x7f4b94153700 2024-05-09T05:19:18.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.683+0000 7f4b90ff9640 1 -- 172.21.15.1:0/1893669459 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f4b7c060b30 con 0x7f4b94153700 2024-05-09T05:19:18.682 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":98,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4b74076350 msgr2=0x7f4b74078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4b74076350 0x7f4b74078860 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4b84009560 tx=0x7f4b8402e000 comp rx=0 tx=0).stop 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b94153700 msgr2=0x7f4b9414b1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b94153700 0x7f4b9414b1e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4b7c00d5d0 tx=0x7f4b7c00da90 comp rx=0 tx=0).stop 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 shutdown_connections 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4b74076350 0x7f4b74078860 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4b94153700 0x7f4b9414b1e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4b9415c8b0 0x7f4b9414f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 --2- 172.21.15.1:0/1893669459 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4b940cd280 0x7f4b941e71c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:18.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.687+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 >> 172.21.15.1:0/1893669459 conn(0x7f4b94068650 msgr2=0x7f4b940ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.688+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 shutdown_connections 2024-05-09T05:19:18.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:18.688+0000 7f4b9a31f640 1 -- 172.21.15.1:0/1893669459 wait complete. 2024-05-09T05:19:21.704 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:19:21.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.885+0000 7fd423cbd640 1 Processor -- start 2024-05-09T05:19:21.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.885+0000 7fd423cbd640 1 -- start start 2024-05-09T05:19:21.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.887+0000 7fd423cbd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd41c0ca340 0x7fd41c0c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.888+0000 7fd423cbd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd41c15b110 con 0x7fd41c072120 2024-05-09T05:19:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.888+0000 7fd423cbd640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd41c15b3b0 con 0x7fd41c0c62e0 2024-05-09T05:19:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.888+0000 7fd421258640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd41c072120 0x7fd41c1547b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32904/0 (socket says 172.21.15.1:32904) 2024-05-09T05:19:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.888+0000 7fd421258640 1 -- 172.21.15.1:0/3447655778 learned_addr learned my addr 172.21.15.1:0/3447655778 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.888+0000 7fd423cbd640 1 -- 172.21.15.1:0/3447655778 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd41c0c53f0 con 0x7fd41c0ca340 2024-05-09T05:19:21.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.888+0000 7fd42225a640 1 --1- 172.21.15.1:0/3447655778 >> v1:172.21.15.196:6789/0 conn(0x7fd41c0c62e0 0x7fd41c070e20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:40166/0 (socket says 172.21.15.1:40166) 2024-05-09T05:19:21.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.888+0000 7fd421a59640 1 --2- 172.21.15.1:0/3447655778 >> v2:172.21.15.196:3300/0 conn(0x7fd41c0ca340 0x7fd41c0c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:21.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.889+0000 7fd421a59640 1 -- 172.21.15.1:0/3447655778 >> v1:172.21.15.196:6789/0 conn(0x7fd41c0c62e0 legacy=0x7fd41c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:21.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.889+0000 7fd421a59640 1 -- 172.21.15.1:0/3447655778 >> v1:172.21.15.1:6789/0 conn(0x7fd41c072120 legacy=0x7fd41c1547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:21.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.889+0000 7fd420a57640 1 -- 172.21.15.1:0/3447655778 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 933108190 0 0) 0x7fd41c15b3b0 con 0x7fd41c0c62e0 2024-05-09T05:19:21.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.889+0000 7fd421a59640 1 -- 172.21.15.1:0/3447655778 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c0c5800 con 0x7fd41c0ca340 2024-05-09T05:19:21.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.890+0000 7fd421a59640 1 --2- 172.21.15.1:0/3447655778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0ca340 0x7fd41c0c4eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd4180097a0 tx=0x7fd41802a0a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f4720aee84930151 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:21.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.890+0000 7fd420a57640 1 -- 172.21.15.1:0/3447655778 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418037070 con 0x7fd41c0ca340 2024-05-09T05:19:21.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.890+0000 7fd420a57640 1 -- 172.21.15.1:0/3447655778 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd41802ad30 con 0x7fd41c0ca340 2024-05-09T05:19:21.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.891+0000 7fd420a57640 1 -- 172.21.15.1:0/3447655778 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd418032660 con 0x7fd41c0ca340 2024-05-09T05:19:21.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.892+0000 7fd423cbd640 1 -- 172.21.15.1:0/3447655778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0ca340 msgr2=0x7fd41c0c4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:21.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.892+0000 7fd423cbd640 1 --2- 172.21.15.1:0/3447655778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0ca340 0x7fd41c0c4eb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fd4180097a0 tx=0x7fd41802a0a0 comp rx=0 tx=0).stop 2024-05-09T05:19:21.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.893+0000 7fd423cbd640 1 -- 172.21.15.1:0/3447655778 shutdown_connections 2024-05-09T05:19:21.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.893+0000 7fd423cbd640 1 --2- 172.21.15.1:0/3447655778 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0ca340 0x7fd41c0c4eb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:21.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.893+0000 7fd423cbd640 1 -- 172.21.15.1:0/3447655778 >> 172.21.15.1:0/3447655778 conn(0x7fd41c0731d0 msgr2=0x7fd41c0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:21.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.893+0000 7fd423cbd640 1 -- 172.21.15.1:0/3447655778 shutdown_connections 2024-05-09T05:19:21.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.893+0000 7fd423cbd640 1 -- 172.21.15.1:0/3447655778 wait complete. 2024-05-09T05:19:21.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.896+0000 7fd423cbd640 1 Processor -- start 2024-05-09T05:19:21.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.896+0000 7fd423cbd640 1 -- start start 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.899+0000 7fd423cbd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd41c072120 0x7fd41c1f0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.899+0000 7fd423cbd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0c62e0 0x7fd41c151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.899+0000 7fd423cbd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd41c0ca340 0x7fd41c14d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.900+0000 7fd423cbd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c0c5dc0 con 0x7fd41c0ca340 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.900+0000 7fd421258640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0c62e0 0x7fd41c151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.900+0000 7fd421258640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0c62e0 0x7fd41c151d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:51750/0 (socket says 172.21.15.1:51750) 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.900+0000 7fd421258640 1 -- 172.21.15.1:0/852990661 learned_addr learned my addr 172.21.15.1:0/852990661 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.900+0000 7fd42225a640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd41c0ca340 0x7fd41c14d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.900+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd41c0c55b0 con 0x7fd41c0c62e0 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.900+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd41c0c66f0 con 0x7fd41c072120 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.901+0000 7fd42225a640 1 -- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd41c072120 msgr2=0x7fd41c1f0720 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.901+0000 7fd42225a640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd41c072120 0x7fd41c1f0720 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:21.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.901+0000 7fd42225a640 1 -- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0c62e0 msgr2=0x7fd41c151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.901+0000 7fd42225a640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0c62e0 0x7fd41c151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.901+0000 7fd42225a640 1 -- 172.21.15.1:0/852990661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd41c1f0c60 con 0x7fd41c0ca340 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.901+0000 7fd42225a640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd41c0ca340 0x7fd41c14d1a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd41401d740 tx=0x7fd41401dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.901+0000 7fd4127fc640 1 -- 172.21.15.1:0/852990661 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd414029070 con 0x7fd41c0ca340 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.902+0000 7fd4127fc640 1 -- 172.21.15.1:0/852990661 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd414004b60 con 0x7fd41c0ca340 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.902+0000 7fd4127fc640 1 -- 172.21.15.1:0/852990661 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd41400a640 con 0x7fd41c0ca340 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.902+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd41c0ce8d0 con 0x7fd41c0ca340 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.902+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd41c0ce080 con 0x7fd41c0ca340 2024-05-09T05:19:21.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.903+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd41c153570 con 0x7fd41c0ca340 2024-05-09T05:19:21.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.904+0000 7fd4127fc640 1 -- 172.21.15.1:0/852990661 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd41402c070 con 0x7fd41c0ca340 2024-05-09T05:19:21.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.905+0000 7fd4127fc640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd400076350 0x7fd400078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:21.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.905+0000 7fd4127fc640 1 -- 172.21.15.1:0/852990661 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd4140a7760 con 0x7fd41c0ca340 2024-05-09T05:19:21.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.906+0000 7fd421a59640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd400076350 0x7fd400078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:21.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.908+0000 7fd421a59640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd400076350 0x7fd400078860 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd418034000 tx=0x7fd41802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:21.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:21.912+0000 7fd4127fc640 1 -- 172.21.15.1:0/852990661 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd414070a40 con 0x7fd41c0ca340 2024-05-09T05:19:22.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.410+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd41c050870 con 0x7fd41c0ca340 2024-05-09T05:19:22.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.410+0000 7fd4127fc640 1 -- 172.21.15.1:0/852990661 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7fd414079110 con 0x7fd41c0ca340 2024-05-09T05:19:22.410 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:19:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd400076350 msgr2=0x7fd400078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd400076350 0x7fd400078860 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fd418034000 tx=0x7fd41802e000 comp rx=0 tx=0).stop 2024-05-09T05:19:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd41c0ca340 msgr2=0x7fd41c14d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd41c0ca340 0x7fd41c14d1a0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fd41401d740 tx=0x7fd41401dc00 comp rx=0 tx=0).stop 2024-05-09T05:19:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 shutdown_connections 2024-05-09T05:19:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd400076350 0x7fd400078860 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd41c0c62e0 0x7fd41c151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd41c0ca340 0x7fd41c14d1a0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 --2- 172.21.15.1:0/852990661 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd41c072120 0x7fd41c1f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.414+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 >> 172.21.15.1:0/852990661 conn(0x7fd41c0731d0 msgr2=0x7fd41c1502d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.415+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 shutdown_connections 2024-05-09T05:19:22.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.415+0000 7fd423cbd640 1 -- 172.21.15.1:0/852990661 wait complete. 2024-05-09T05:19:22.432 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:19:22.432 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:19:22.432 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:19:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.614+0000 7f0622d5e640 1 Processor -- start 2024-05-09T05:19:22.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.614+0000 7f0622d5e640 1 -- start start 2024-05-09T05:19:22.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.616+0000 7f0622d5e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f061c0cdfe0 0x7f061c0ced20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.616+0000 7f0622d5e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c165170 0x7f061c167590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.616+0000 7f0622d5e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f061c0ce540 con 0x7f061c0c6b10 2024-05-09T05:19:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.616+0000 7f0622d5e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f061c0b5720 con 0x7f061c0cdfe0 2024-05-09T05:19:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.616+0000 7f0622d5e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f061c0ce890 con 0x7f061c165170 2024-05-09T05:19:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.616+0000 7f061bfff640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f061c0c6b10 0x7f061c0d1f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:40168/0 (socket says 172.21.15.1:40168) 2024-05-09T05:19:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.616+0000 7f061bfff640 1 -- 172.21.15.1:0/3283372065 learned_addr learned my addr 172.21.15.1:0/3283372065 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.616+0000 7f0620afa640 1 --2- 172.21.15.1:0/3283372065 >> v2:172.21.15.1:3300/0 conn(0x7f061c0cdfe0 0x7f061c0ced20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.615 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.617+0000 7f061b7fe640 1 -- 172.21.15.1:0/3283372065 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2035909204 0 0) 0x7f061c0ce540 con 0x7f061c0c6b10 2024-05-09T05:19:22.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.617+0000 7f061b7fe640 1 -- 172.21.15.1:0/3283372065 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0608003520 con 0x7f061c0c6b10 2024-05-09T05:19:22.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.617+0000 7f0620afa640 1 -- 172.21.15.1:0/3283372065 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c165170 msgr2=0x7f061c167590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:22.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.617+0000 7f0620afa640 1 --2- 172.21.15.1:0/3283372065 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c165170 0x7f061c167590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.617+0000 7f0620afa640 1 -- 172.21.15.1:0/3283372065 >> v1:172.21.15.196:6789/0 conn(0x7f061c0c6b10 legacy=0x7f061c0d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.617+0000 7f0620afa640 1 -- 172.21.15.1:0/3283372065 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c15d400 con 0x7f061c0cdfe0 2024-05-09T05:19:22.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.618+0000 7f0620afa640 1 --2- 172.21.15.1:0/3283372065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c0cdfe0 0x7f061c0ced20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f06040097a0 tx=0x7f060402a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60e06e2fd943d3d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.619+0000 7f061b7fe640 1 -- 172.21.15.1:0/3283372065 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604037070 con 0x7f061c0cdfe0 2024-05-09T05:19:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.619+0000 7f061b7fe640 1 -- 172.21.15.1:0/3283372065 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0604032440 con 0x7f061c0cdfe0 2024-05-09T05:19:22.617 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.619+0000 7f061b7fe640 1 -- 172.21.15.1:0/3283372065 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604032810 con 0x7f061c0cdfe0 2024-05-09T05:19:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.620+0000 7f0622d5e640 1 -- 172.21.15.1:0/3283372065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c0cdfe0 msgr2=0x7f061c0ced20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.620+0000 7f0622d5e640 1 --2- 172.21.15.1:0/3283372065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c0cdfe0 0x7f061c0ced20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f06040097a0 tx=0x7f060402a180 comp rx=0 tx=0).stop 2024-05-09T05:19:22.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.621+0000 7f0622d5e640 1 -- 172.21.15.1:0/3283372065 shutdown_connections 2024-05-09T05:19:22.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.621+0000 7f0622d5e640 1 --2- 172.21.15.1:0/3283372065 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c165170 0x7f061c167590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.621+0000 7f0622d5e640 1 --2- 172.21.15.1:0/3283372065 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c0cdfe0 0x7f061c0ced20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.621+0000 7f0622d5e640 1 -- 172.21.15.1:0/3283372065 >> 172.21.15.1:0/3283372065 conn(0x7f061c06dca0 msgr2=0x7f061c06e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.621+0000 7f0622d5e640 1 -- 172.21.15.1:0/3283372065 shutdown_connections 2024-05-09T05:19:22.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.621+0000 7f0622d5e640 1 -- 172.21.15.1:0/3283372065 wait complete. 2024-05-09T05:19:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.624+0000 7f0622d5e640 1 Processor -- start 2024-05-09T05:19:22.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.624+0000 7f0622d5e640 1 -- start start 2024-05-09T05:19:22.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.626+0000 7f0622d5e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f061c0c6b10 0x7f061c1f3110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.626+0000 7f0622d5e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c0cdfe0 0x7f061c155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.627+0000 7f0622d5e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c165170 0x7f061c14d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.627+0000 7f0622d5e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f061c15d6e0 con 0x7f061c165170 2024-05-09T05:19:22.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.627+0000 7f0622d5e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f061c0cd600 con 0x7f061c0c6b10 2024-05-09T05:19:22.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.627+0000 7f0622d5e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f061c0c7330 con 0x7f061c0cdfe0 2024-05-09T05:19:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f0620afa640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f061c0c6b10 0x7f061c1f3110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f061bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c0cdfe0 0x7f061c155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f0620afa640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f061c0c6b10 0x7f061c1f3110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:51758/0 (socket says 172.21.15.1:51758) 2024-05-09T05:19:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f0620afa640 1 -- 172.21.15.1:0/81766079 learned_addr learned my addr 172.21.15.1:0/81766079 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f06212fb640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c165170 0x7f061c14d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f061bfff640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c0cdfe0 msgr2=0x7f061c155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:19:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f061bfff640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c0cdfe0 msgr2=0x7f061c155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:19:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f061bfff640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c0cdfe0 0x7f061c155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:22.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f061bfff640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c0cdfe0 0x7f061c155de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f0620afa640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c0cdfe0 msgr2=0x7f061c155de0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f0620afa640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f061c0cdfe0 0x7f061c155de0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f0620afa640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c165170 msgr2=0x7f061c14d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f0620afa640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c165170 0x7f061c14d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f0620afa640 1 -- 172.21.15.1:0/81766079 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f061c1f3650 con 0x7f061c0c6b10 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.628+0000 7f06212fb640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c165170 0x7f061c14d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.629+0000 7f0620afa640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f061c0c6b10 0x7f061c1f3110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f06040098c0 tx=0x7f06040046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.629+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604037040 con 0x7f061c0c6b10 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.629+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0604004230 con 0x7f061c0c6b10 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.629+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f061c15bb90 con 0x7f061c0c6b10 2024-05-09T05:19:22.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.629+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0604003610 con 0x7f061c0c6b10 2024-05-09T05:19:22.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.629+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f061c1f2910 con 0x7f061c0c6b10 2024-05-09T05:19:22.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.631+0000 7f0622d5e640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c15b080 0x7f061c14dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.631+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f061c1f2700 con 0x7f061c15b080 2024-05-09T05:19:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.632+0000 7f061bfff640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c15b080 0x7f061c14dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.632+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f06040037b0 con 0x7f061c0c6b10 2024-05-09T05:19:22.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.632+0000 7f061bfff640 1 -- 172.21.15.1:0/81766079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f061c050870 con 0x7f061c15b080 2024-05-09T05:19:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.633+0000 7f061bfff640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c15b080 0x7f061c14dd30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0614007a60 tx=0x7f061400e960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.633+0000 7f06197fa640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f05fc0762d0 0x7f05fc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.633+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f06040b87e0 con 0x7f061c0c6b10 2024-05-09T05:19:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.633+0000 7f06212fb640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f05fc0762d0 0x7f05fc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.633+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614019070 con 0x7f061c15b080 2024-05-09T05:19:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.634+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0614005ac0 con 0x7f061c15b080 2024-05-09T05:19:22.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.634+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c15b080 msgr2=0x7f061c14dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.634+0000 7f06197fa640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c15b080 0x7f061c14dd30 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0614007a60 tx=0x7f061400e960 comp rx=0 tx=0).stop 2024-05-09T05:19:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.634+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c15b080 msgr2=0x7f061c14dd30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.634+0000 7f06197fa640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c15b080 0x7f061c14dd30 secure :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f0614007a60 tx=0x7f061400e960 comp rx=0 tx=0).stop 2024-05-09T05:19:22.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.634+0000 7f06212fb640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f05fc0762d0 0x7f05fc0787e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f061c06f6b0 tx=0x7f060c00d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.650+0000 7f0622d5e640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c159ad0 0x7f061c14e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.650+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f061c0cddc0 con 0x7f061c159ad0 2024-05-09T05:19:22.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.650+0000 7f061bfff640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c159ad0 0x7f061c14e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.651+0000 7f061bfff640 1 -- 172.21.15.1:0/81766079 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f061c0b71e0 con 0x7f061c159ad0 2024-05-09T05:19:22.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.651+0000 7f061bfff640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c159ad0 0x7f061c14e8f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0614016000 tx=0x7f06140057f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.651+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0614019040 con 0x7f061c159ad0 2024-05-09T05:19:22.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7f06140073d0 con 0x7f061c159ad0 2024-05-09T05:19:22.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c159ad0 msgr2=0x7f061c14e8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f06197fa640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c159ad0 0x7f061c14e8f0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0614016000 tx=0x7f06140057f0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f06197fa640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c159ad0 msgr2=0x7f061c14e8f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:22.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f06197fa640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f061c159ad0 0x7f061c14e8f0 secure :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f0614016000 tx=0x7f06140057f0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.651 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:19:22.651 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:22.651 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:22.651 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:19:22.651 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 100, 2024-05-09T05:19:22.651 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 364847, 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:22.652 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:22.653 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:22.654 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:19:22.655 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:19:22.656 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:22.657 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:19:22.657 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:22.657 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:22.657 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:22.657 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:22.657 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:22.658 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:22.658 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:22.658 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:22.658 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:22.658 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:22.658 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:22.658 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:22.659 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:22.660 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.661 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:22.662 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.663 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:22.664 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:19:22.665 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:22.666 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:19:22.667 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.667 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.667 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.667 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:19:22.667 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:19:22.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f05fc0762d0 msgr2=0x7f05fc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f0622d5e640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f05fc0762d0 0x7f05fc0787e0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f061c06f6b0 tx=0x7f060c00d040 comp rx=0 tx=0).stop 2024-05-09T05:19:22.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f061c0c6b10 msgr2=0x7f061c1f3110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f0622d5e640 1 --2- 172.21.15.1:0/81766079 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f061c0c6b10 0x7f061c1f3110 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f06040098c0 tx=0x7f06040046c0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.652+0000 7f06212fb640 1 -- 172.21.15.1:0/81766079 reap_dead start 2024-05-09T05:19:22.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.653+0000 7f06212fb640 1 -- 172.21.15.1:0/81766079 reap_dead start 2024-05-09T05:19:22.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.653+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 shutdown_connections 2024-05-09T05:19:22.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.653+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 >> 172.21.15.1:0/81766079 conn(0x7f061c06dca0 msgr2=0x7f061c06e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:22.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.653+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 shutdown_connections 2024-05-09T05:19:22.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.653+0000 7f0622d5e640 1 -- 172.21.15.1:0/81766079 wait complete. 2024-05-09T05:19:22.669 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:19:22.669 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:19:22.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:19:22.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.841+0000 7f039353f640 1 Processor -- start 2024-05-09T05:19:22.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.841+0000 7f039353f640 1 -- start start 2024-05-09T05:19:22.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.843+0000 7f039353f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f038c0c9520 0x7f038c0717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.843+0000 7f039353f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f038c071d30 0x7f038c159700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.843+0000 7f039353f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f038c0cb450 con 0x7f038c0c8b70 2024-05-09T05:19:22.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.843+0000 7f039353f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f038c0c9e50 con 0x7f038c0c9520 2024-05-09T05:19:22.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.843+0000 7f039353f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f038c0caed0 con 0x7f038c071d30 2024-05-09T05:19:22.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.843+0000 7f0390ada640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f038c0c9520 0x7f038c0717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.844+0000 7f0390ada640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f038c0c9520 0x7f038c0717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47510/0 (socket says 172.21.15.1:47510) 2024-05-09T05:19:22.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.844+0000 7f0390ada640 1 -- 172.21.15.1:0/3888963741 learned_addr learned my addr 172.21.15.1:0/3888963741 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.844+0000 7f0391adc640 1 --2- 172.21.15.1:0/3888963741 >> v2:172.21.15.196:3300/0 conn(0x7f038c071d30 0x7f038c159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.844+0000 7f0390ada640 1 -- 172.21.15.1:0/3888963741 >> v2:172.21.15.196:3300/0 conn(0x7f038c071d30 msgr2=0x7f038c159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.844+0000 7f0390ada640 1 --2- 172.21.15.1:0/3888963741 >> v2:172.21.15.196:3300/0 conn(0x7f038c071d30 0x7f038c159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.844+0000 7f0390ada640 1 -- 172.21.15.1:0/3888963741 >> v1:172.21.15.1:6789/0 conn(0x7f038c0c8b70 legacy=0x7f038c156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.844+0000 7f0390ada640 1 -- 172.21.15.1:0/3888963741 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c0cb170 con 0x7f038c0c9520 2024-05-09T05:19:22.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.845+0000 7f0390ada640 1 --2- 172.21.15.1:0/3888963741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c0c9520 0x7f038c0717f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f038c072330 tx=0x7f037c02a280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=821930c97f4187bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.845+0000 7f0383fff640 1 -- 172.21.15.1:0/3888963741 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c037070 con 0x7f038c0c9520 2024-05-09T05:19:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.846+0000 7f0383fff640 1 -- 172.21.15.1:0/3888963741 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f037c032440 con 0x7f038c0c9520 2024-05-09T05:19:22.844 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.846+0000 7f0383fff640 1 -- 172.21.15.1:0/3888963741 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f037c032810 con 0x7f038c0c9520 2024-05-09T05:19:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.847+0000 7f039353f640 1 -- 172.21.15.1:0/3888963741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c0c9520 msgr2=0x7f038c0717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.845 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.847+0000 7f039353f640 1 --2- 172.21.15.1:0/3888963741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c0c9520 0x7f038c0717f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f038c072330 tx=0x7f037c02a280 comp rx=0 tx=0).stop 2024-05-09T05:19:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.847+0000 7f039353f640 1 -- 172.21.15.1:0/3888963741 shutdown_connections 2024-05-09T05:19:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.847+0000 7f039353f640 1 --2- 172.21.15.1:0/3888963741 >> v2:172.21.15.196:3300/0 conn(0x7f038c071d30 0x7f038c159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.847+0000 7f039353f640 1 --2- 172.21.15.1:0/3888963741 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c0c9520 0x7f038c0717f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.847+0000 7f039353f640 1 -- 172.21.15.1:0/3888963741 >> 172.21.15.1:0/3888963741 conn(0x7f038c06d800 msgr2=0x7f038c06dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.847+0000 7f039353f640 1 -- 172.21.15.1:0/3888963741 shutdown_connections 2024-05-09T05:19:22.846 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.847+0000 7f039353f640 1 -- 172.21.15.1:0/3888963741 wait complete. 2024-05-09T05:19:22.847 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.849+0000 7f039353f640 1 Processor -- start 2024-05-09T05:19:22.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.850+0000 7f039353f640 1 -- start start 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.851+0000 7f039353f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c071d30 0x7f038c1f5f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.851+0000 7f039353f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f038c0c8b70 0x7f038c14b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.851+0000 7f039353f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c0c9520 0x7f038c1509a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.851+0000 7f039353f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f038c071210 con 0x7f038c071d30 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.851+0000 7f039353f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0c3f30 con 0x7f038c0c9520 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.851+0000 7f039353f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f038c0c9a30 con 0x7f038c0c8b70 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f03912db640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c071d30 0x7f038c1f5f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f03912db640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c071d30 0x7f038c1f5f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47514/0 (socket says 172.21.15.1:47514) 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f03912db640 1 -- 172.21.15.1:0/4272364640 learned_addr learned my addr 172.21.15.1:0/4272364640 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:22.850 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f0391adc640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c0c9520 0x7f038c1509a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f0390ada640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f038c0c8b70 0x7f038c14b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f0391adc640 1 -- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f038c0c8b70 msgr2=0x7f038c14b730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f0391adc640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f038c0c8b70 0x7f038c14b730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f0391adc640 1 -- 172.21.15.1:0/4272364640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c071d30 msgr2=0x7f038c1f5f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f0391adc640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f038c071d30 0x7f038c1f5f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.852+0000 7f0391adc640 1 -- 172.21.15.1:0/4272364640 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f038c1f6490 con 0x7f038c0c9520 2024-05-09T05:19:22.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.854+0000 7f0391adc640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c0c9520 0x7f038c1509a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f038400eb70 tx=0x7f038400c490 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.854+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0384018020 con 0x7f038c0c9520 2024-05-09T05:19:22.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.855+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f038400cd20 con 0x7f038c0c9520 2024-05-09T05:19:22.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.855+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03840146c0 con 0x7f038c0c9520 2024-05-09T05:19:22.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.855+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f038c1e3ac0 con 0x7f038c0c9520 2024-05-09T05:19:22.853 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.855+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f038c1f2ca0 con 0x7f038c0c9520 2024-05-09T05:19:22.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.856+0000 7f039353f640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1f6660 0x7f038c150ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.856+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f038c1f5ac0 con 0x7f038c1f6660 2024-05-09T05:19:22.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.857+0000 7f03912db640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1f6660 0x7f038c150ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.857+0000 7f03912db640 1 -- 172.21.15.1:0/4272364640 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f038c1f2730 con 0x7f038c1f6660 2024-05-09T05:19:22.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.858+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0384005d10 con 0x7f038c0c9520 2024-05-09T05:19:22.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.858+0000 7f03912db640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1f6660 0x7f038c150ee0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f038800f590 tx=0x7f038800fa50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.858+0000 7f0381ffb640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0370076320 0x7f0370078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.858+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f03840980f0 con 0x7f038c0c9520 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.858+0000 7f0390ada640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0370076320 0x7f0370078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.859+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03880052e0 con 0x7f038c1f6660 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.859+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0388018070 con 0x7f038c1f6660 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.859+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1f6660 msgr2=0x7f038c150ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.859+0000 7f0381ffb640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1f6660 0x7f038c150ee0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f038800f590 tx=0x7f038800fa50 comp rx=0 tx=0).stop 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.859+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1f6660 msgr2=0x7f038c150ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:22.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.859+0000 7f0381ffb640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1f6660 0x7f038c150ee0 secure :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f038800f590 tx=0x7f038800fa50 comp rx=0 tx=0).stop 2024-05-09T05:19:22.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.859+0000 7f0390ada640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0370076320 0x7f0370078830 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f037c034000 tx=0x7f037c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.870+0000 7f039353f640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1e1b70 0x7f038c1f2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:22.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.870+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f038c0696b0 con 0x7f038c1e1b70 2024-05-09T05:19:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.871+0000 7f03912db640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1e1b70 0x7f038c1f2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.871+0000 7f03912db640 1 -- 172.21.15.1:0/4272364640 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f038c050870 con 0x7f038c1e1b70 2024-05-09T05:19:22.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.872+0000 7f03912db640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1e1b70 0x7f038c1f2ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f038800f590 tx=0x7f0388000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:22.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.872+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0388003910 con 0x7f038c1e1b70 2024-05-09T05:19:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.874+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f0388003ab0 con 0x7f038c1e1b70 2024-05-09T05:19:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.874+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1e1b70 msgr2=0x7f038c1f2ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.874+0000 7f0381ffb640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1e1b70 0x7f038c1f2ee0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f038800f590 tx=0x7f0388000ed0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.874+0000 7f0381ffb640 1 -- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1e1b70 msgr2=0x7f038c1f2ee0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:22.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.874+0000 7f0381ffb640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c1e1b70 0x7f038c1f2ee0 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f038800f590 tx=0x7f0388000ed0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 100, 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 113279, 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:19:22.874 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:22.875 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:22.876 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:19:22.877 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:22.878 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.879 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:22.880 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:22.881 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:22.882 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:19:22.883 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:22.884 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:19:22.885 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:19:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.876+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0370076320 msgr2=0x7f0370078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.876+0000 7f039353f640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0370076320 0x7f0370078830 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f037c034000 tx=0x7f037c02e000 comp rx=0 tx=0).stop 2024-05-09T05:19:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.876+0000 7f0391adc640 1 -- 172.21.15.1:0/4272364640 reap_dead start 2024-05-09T05:19:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.876+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c0c9520 msgr2=0x7f038c1509a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.876+0000 7f039353f640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c0c9520 0x7f038c1509a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f038400eb70 tx=0x7f038400c490 comp rx=0 tx=0).stop 2024-05-09T05:19:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.877+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 shutdown_connections 2024-05-09T05:19:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.877+0000 7f039353f640 1 --2- 172.21.15.1:0/4272364640 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f038c0c9520 0x7f038c1509a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:22.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.877+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 >> 172.21.15.1:0/4272364640 conn(0x7f038c06d800 msgr2=0x7f038c0c4f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.877+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 shutdown_connections 2024-05-09T05:19:22.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:22.877+0000 7f039353f640 1 -- 172.21.15.1:0/4272364640 wait complete. 2024-05-09T05:19:22.892 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:19:22.892 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:19:22.892 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:19:23.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.075+0000 7fae452a7640 1 Processor -- start 2024-05-09T05:19:23.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.075+0000 7fae452a7640 1 -- start start 2024-05-09T05:19:23.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae452a7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae400cbf60 0x7fae400cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae3e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae400cbf60 0x7fae400cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae3e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fae400cbf60 0x7fae400cd4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47524/0 (socket says 172.21.15.1:47524) 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae3e7fc640 1 -- 172.21.15.1:0/161581104 learned_addr learned my addr 172.21.15.1:0/161581104 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae452a7640 1 -- 172.21.15.1:0/161581104 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae400c7970 con 0x7fae400747f0 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae452a7640 1 -- 172.21.15.1:0/161581104 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fae400c7be0 con 0x7fae400cc370 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae452a7640 1 -- 172.21.15.1:0/161581104 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fae400c5600 con 0x7fae400cbf60 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae3dffb640 1 -- 172.21.15.1:0/161581104 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3580675387 0 0) 0x7fae400c7be0 con 0x7fae400cc370 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.078+0000 7fae3dffb640 1 -- 172.21.15.1:0/161581104 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fae28003520 con 0x7fae400cc370 2024-05-09T05:19:23.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.079+0000 7fae3e7fc640 1 -- 172.21.15.1:0/161581104 >> v1:172.21.15.196:6789/0 conn(0x7fae400cc370 legacy=0x7fae400c5a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.079+0000 7fae3e7fc640 1 -- 172.21.15.1:0/161581104 >> v1:172.21.15.1:6789/0 conn(0x7fae400747f0 legacy=0x7fae40152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:23.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.079+0000 7fae3e7fc640 1 -- 172.21.15.1:0/161581104 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae400c7ec0 con 0x7fae400cbf60 2024-05-09T05:19:23.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.080+0000 7fae3e7fc640 1 --2- 172.21.15.1:0/161581104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae400cbf60 0x7fae400cd4c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fae3400bb00 tx=0x7fae3402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e19f399fb28990b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:23.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.080+0000 7fae3dffb640 1 -- 172.21.15.1:0/161581104 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3403a070 con 0x7fae400cbf60 2024-05-09T05:19:23.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.080+0000 7fae3dffb640 1 -- 172.21.15.1:0/161581104 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae34032480 con 0x7fae400cbf60 2024-05-09T05:19:23.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.080+0000 7fae3dffb640 1 -- 172.21.15.1:0/161581104 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae34032850 con 0x7fae400cbf60 2024-05-09T05:19:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.081+0000 7fae452a7640 1 -- 172.21.15.1:0/161581104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae400cbf60 msgr2=0x7fae400cd4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.081+0000 7fae452a7640 1 --2- 172.21.15.1:0/161581104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae400cbf60 0x7fae400cd4c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fae3400bb00 tx=0x7fae3402c650 comp rx=0 tx=0).stop 2024-05-09T05:19:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.081+0000 7fae452a7640 1 -- 172.21.15.1:0/161581104 shutdown_connections 2024-05-09T05:19:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.081+0000 7fae452a7640 1 --2- 172.21.15.1:0/161581104 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae400cbf60 0x7fae400cd4c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.081+0000 7fae452a7640 1 -- 172.21.15.1:0/161581104 >> 172.21.15.1:0/161581104 conn(0x7fae400ce180 msgr2=0x7fae400ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.082+0000 7fae452a7640 1 -- 172.21.15.1:0/161581104 shutdown_connections 2024-05-09T05:19:23.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.082+0000 7fae452a7640 1 -- 172.21.15.1:0/161581104 wait complete. 2024-05-09T05:19:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.084+0000 7fae452a7640 1 Processor -- start 2024-05-09T05:19:23.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.084+0000 7fae452a7640 1 -- start start 2024-05-09T05:19:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae452a7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae400747f0 0x7fae40156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae452a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae400cbf60 0x7fae4014b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae452a7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae400cc370 0x7fae40147b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae452a7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fae400740c0 con 0x7fae400cbf60 2024-05-09T05:19:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae452a7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fae400c8480 con 0x7fae400cc370 2024-05-09T05:19:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae452a7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fae400c61a0 con 0x7fae400747f0 2024-05-09T05:19:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae400cc370 0x7fae40147b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae3effd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae400747f0 0x7fae40156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae3f7fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae400cc370 0x7fae40147b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:51800/0 (socket says 172.21.15.1:51800) 2024-05-09T05:19:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.086+0000 7fae3f7fe640 1 -- 172.21.15.1:0/1320524721 learned_addr learned my addr 172.21.15.1:0/1320524721 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:23.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.087+0000 7fae3e7fc640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae400cbf60 0x7fae4014b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.087+0000 7fae3f7fe640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae400747f0 msgr2=0x7fae40156bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.087+0000 7fae3f7fe640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae400747f0 0x7fae40156bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.087+0000 7fae3f7fe640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae400cbf60 msgr2=0x7fae4014b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.087+0000 7fae3f7fe640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fae400cbf60 0x7fae4014b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.087+0000 7fae3f7fe640 1 -- 172.21.15.1:0/1320524721 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fae4014ca70 con 0x7fae400cc370 2024-05-09T05:19:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.087+0000 7fae3f7fe640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae400cc370 0x7fae40147b30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fae3800c730 tx=0x7fae3800cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.087+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae38003b70 con 0x7fae400cc370 2024-05-09T05:19:23.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.088+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fae38003d30 con 0x7fae400cc370 2024-05-09T05:19:23.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.088+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae3800aa90 con 0x7fae400cc370 2024-05-09T05:19:23.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.088+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fae4014b640 con 0x7fae400cc370 2024-05-09T05:19:23.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.088+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fae40153ac0 con 0x7fae400cc370 2024-05-09T05:19:23.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.089+0000 7fae452a7640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 0x7fae20004f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.089+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fae20007f10 con 0x7fae20002a70 2024-05-09T05:19:23.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.089+0000 7fae3effd640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 0x7fae20004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.090+0000 7fae3effd640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 msgr2=0x7fae20004f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 20 2024-05-09T05:19:23.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.090+0000 7fae3effd640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 msgr2=0x7fae20004f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:19:23.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.090+0000 7fae3effd640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 0x7fae20004f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:23.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.090+0000 7fae3effd640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 0x7fae20004f80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:23.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.090+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fae3800ac30 con 0x7fae400cc370 2024-05-09T05:19:23.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.091+0000 7fae277fe640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fae2c0762d0 0x7fae2c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.091+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fae38097520 con 0x7fae400cc370 2024-05-09T05:19:23.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.092+0000 7fae3e7fc640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fae2c0762d0 0x7fae2c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.092+0000 7fae3e7fc640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fae2c0762d0 0x7fae2c0787e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fae34037000 tx=0x7fae34030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:23.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.291+0000 7fae3effd640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 0x7fae20004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.291+0000 7fae3effd640 1 -- 172.21.15.1:0/1320524721 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fae20008110 con 0x7fae20002a70 2024-05-09T05:19:23.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.292+0000 7fae3effd640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 0x7fae20004f80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fae3000f410 tx=0x7fae3000f8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:23.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.292+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30019020 con 0x7fae20002a70 2024-05-09T05:19:23.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.292+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fae30014870 con 0x7fae20002a70 2024-05-09T05:19:23.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.292+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 msgr2=0x7fae20004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.292+0000 7fae277fe640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 0x7fae20004f80 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fae3000f410 tx=0x7fae3000f8d0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.292+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 msgr2=0x7fae20004f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:23.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.292+0000 7fae277fe640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20002a70 0x7fae20004f80 secure :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fae3000f410 tx=0x7fae3000f8d0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.305+0000 7fae452a7640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20006f30 0x7fae200073d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.306+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fae2000a4f0 con 0x7fae20006f30 2024-05-09T05:19:23.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.306+0000 7fae3effd640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20006f30 0x7fae200073d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.306+0000 7fae3effd640 1 -- 172.21.15.1:0/1320524721 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fae20007910 con 0x7fae20006f30 2024-05-09T05:19:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.309+0000 7fae3effd640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20006f30 0x7fae200073d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae3000f410 tx=0x7fae30004680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.310+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fae30019070 con 0x7fae20006f30 2024-05-09T05:19:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.310+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4550 (secure 0 0 0) 0x7fae300035f0 con 0x7fae20006f30 2024-05-09T05:19:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.310+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20006f30 msgr2=0x7fae200073d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.310+0000 7fae277fe640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20006f30 0x7fae200073d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae3000f410 tx=0x7fae30004680 comp rx=0 tx=0).stop 2024-05-09T05:19:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.310+0000 7fae277fe640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20006f30 msgr2=0x7fae200073d0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:23.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.310+0000 7fae277fe640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fae20006f30 0x7fae200073d0 secure :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fae3000f410 tx=0x7fae30004680 comp rx=0 tx=0).stop 2024-05-09T05:19:23.309 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:19:23.309 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:23.309 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:23.309 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:19:23.309 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 100, 2024-05-09T05:19:23.309 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 5200, 2024-05-09T05:19:23.309 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:19:23.309 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:23.310 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:23.311 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:23.312 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:23.313 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:23.314 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:23.315 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:23.316 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:23.317 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.318 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.319 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:23.320 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:23.321 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:23.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:23.321 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:19:23.321 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:19:23.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.311+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fae2c0762d0 msgr2=0x7fae2c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.321 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.311+0000 7fae452a7640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fae2c0762d0 0x7fae2c0787e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fae34037000 tx=0x7fae34030000 comp rx=0 tx=0).stop 2024-05-09T05:19:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.311+0000 7fae3f7fe640 1 -- 172.21.15.1:0/1320524721 reap_dead start 2024-05-09T05:19:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.311+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae400cc370 msgr2=0x7fae40147b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.311+0000 7fae452a7640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae400cc370 0x7fae40147b30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fae3800c730 tx=0x7fae3800cbf0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.311+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 shutdown_connections 2024-05-09T05:19:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.311+0000 7fae452a7640 1 --2- 172.21.15.1:0/1320524721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fae400cc370 0x7fae40147b30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.311+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 >> 172.21.15.1:0/1320524721 conn(0x7fae400ce180 msgr2=0x7fae4006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.312+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 shutdown_connections 2024-05-09T05:19:23.322 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.312+0000 7fae452a7640 1 -- 172.21.15.1:0/1320524721 wait complete. 2024-05-09T05:19:23.327 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:19:23.327 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:19:23.328 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:19:23.502 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.504+0000 7fb22331f640 1 Processor -- start 2024-05-09T05:19:23.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.504+0000 7fb22331f640 1 -- start start 2024-05-09T05:19:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.507+0000 7fb22331f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 0x7fb21c152e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.507+0000 7fb22331f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb21c15d080 con 0x7fb21c06ffa0 2024-05-09T05:19:23.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.507+0000 7fb22331f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb21c0c90f0 con 0x7fb21c06e8a0 2024-05-09T05:19:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.507+0000 7fb22331f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb21c0cb580 con 0x7fb21c159410 2024-05-09T05:19:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.507+0000 7fb2210bb640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb21c06ffa0 0x7fb21c158d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:32944/0 (socket says 172.21.15.1:32944) 2024-05-09T05:19:23.506 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.507+0000 7fb2210bb640 1 -- 172.21.15.1:0/2277579703 learned_addr learned my addr 172.21.15.1:0/2277579703 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.508+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2213403501 0 0) 0x7fb21c15d080 con 0x7fb21c06ffa0 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.508+0000 7fb2218bc640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 0x7fb21c152e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.508+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb208003520 con 0x7fb21c06ffa0 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.508+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2901726356 0 0) 0x7fb208003520 con 0x7fb21c06ffa0 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.508+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 msgr2=0x7fb21c152e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.508+0000 7fb20ffff640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 0x7fb21c152e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.508+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 >> v1:172.21.15.196:6789/0 conn(0x7fb21c06e8a0 legacy=0x7fb21c0cae70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.508+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21c15cda0 con 0x7fb21c06ffa0 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fb2180016f0 con 0x7fb21c06ffa0 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 >> v1:172.21.15.1:6789/0 conn(0x7fb21c06ffa0 legacy=0x7fb21c158d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb20ffff640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb208003a80 0x7fb208003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb20ffff640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb208004800 0x7fb208004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb20ffff640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 0x7fb20800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c0cb580 con 0x7fb208004800 2024-05-09T05:19:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb2080055a0 con 0x7fb208003a80 2024-05-09T05:19:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb20800bf00 con 0x7fb21c159410 2024-05-09T05:19:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb2208ba640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb208004800 0x7fb208004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb2218bc640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 0x7fb20800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb2210bb640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb208003a80 0x7fb208003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb2218bc640 1 -- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb208003a80 msgr2=0x7fb208003ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb2218bc640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb208003a80 0x7fb208003ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb2218bc640 1 -- 172.21.15.1:0/2277579703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb208004800 msgr2=0x7fb208004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb2218bc640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb208004800 0x7fb208004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.509+0000 7fb2218bc640 1 -- 172.21.15.1:0/2277579703 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb208003520 con 0x7fb21c159410 2024-05-09T05:19:23.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.511+0000 7fb2218bc640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 0x7fb20800b9c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb214009fc0 tx=0x7fb208005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf0db0b2bb637de8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.511+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb214009c10 con 0x7fb21c159410 2024-05-09T05:19:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.511+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb214036070 con 0x7fb21c159410 2024-05-09T05:19:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.511+0000 7fb20ffff640 1 -- 172.21.15.1:0/2277579703 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21402d070 con 0x7fb21c159410 2024-05-09T05:19:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.512+0000 7fb22331f640 1 -- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 msgr2=0x7fb20800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.512+0000 7fb22331f640 1 --2- 172.21.15.1:0/2277579703 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c159410 0x7fb20800b9c0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb214009fc0 tx=0x7fb208005830 comp rx=0 tx=0).stop 2024-05-09T05:19:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.512+0000 7fb2218bc640 1 -- 172.21.15.1:0/2277579703 reap_dead start 2024-05-09T05:19:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.512+0000 7fb22331f640 1 -- 172.21.15.1:0/2277579703 shutdown_connections 2024-05-09T05:19:23.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.512+0000 7fb22331f640 1 -- 172.21.15.1:0/2277579703 >> 172.21.15.1:0/2277579703 conn(0x7fb21c0ce290 msgr2=0x7fb21c0ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.514+0000 7fb22331f640 1 -- 172.21.15.1:0/2277579703 shutdown_connections 2024-05-09T05:19:23.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.514+0000 7fb22331f640 1 -- 172.21.15.1:0/2277579703 wait complete. 2024-05-09T05:19:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.516+0000 7fb22331f640 1 Processor -- start 2024-05-09T05:19:23.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.516+0000 7fb22331f640 1 -- start start 2024-05-09T05:19:23.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.518+0000 7fb22331f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb21c159410 0x7fb21c147a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.518+0000 7fb22331f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c167760 0x7fb21c15b040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb22331f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb21c1f9c70 0x7fb21c1fa080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb22331f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c0c8660 con 0x7fb21c159410 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb22331f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb21c06f650 con 0x7fb21c1f9c70 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb22331f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb21c0c87e0 con 0x7fb21c167760 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2208ba640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c167760 0x7fb21c15b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2208ba640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c167760 0x7fb21c15b040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:54938/0 (socket says 172.21.15.1:54938) 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2208ba640 1 -- 172.21.15.1:0/1619481552 learned_addr learned my addr 172.21.15.1:0/1619481552 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2210bb640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb21c159410 0x7fb21c147a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2218bc640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb21c1f9c70 0x7fb21c1fa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2208ba640 1 -- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb21c1f9c70 msgr2=0x7fb21c1fa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2208ba640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb21c1f9c70 0x7fb21c1fa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2208ba640 1 -- 172.21.15.1:0/1619481552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb21c159410 msgr2=0x7fb21c147a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2208ba640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb21c159410 0x7fb21c147a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.519+0000 7fb2208ba640 1 -- 172.21.15.1:0/1619481552 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb21c149750 con 0x7fb21c167760 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.520+0000 7fb2208ba640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c167760 0x7fb21c15b040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb21000f850 tx=0x7fb21000fd10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:23.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.520+0000 7fb20dffb640 1 -- 172.21.15.1:0/1619481552 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21000fe40 con 0x7fb21c167760 2024-05-09T05:19:23.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.520+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb21c143c10 con 0x7fb21c167760 2024-05-09T05:19:23.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.520+0000 7fb20dffb640 1 -- 172.21.15.1:0/1619481552 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb210018070 con 0x7fb21c167760 2024-05-09T05:19:23.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.520+0000 7fb20dffb640 1 -- 172.21.15.1:0/1619481552 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb21000d730 con 0x7fb21c167760 2024-05-09T05:19:23.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.520+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb21c1460b0 con 0x7fb21c167760 2024-05-09T05:19:23.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.522+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb21c145ea0 con 0x7fb21c167760 2024-05-09T05:19:23.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.524+0000 7fb20dffb640 1 -- 172.21.15.1:0/1619481552 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb21000d8d0 con 0x7fb21c167760 2024-05-09T05:19:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.524+0000 7fb20dffb640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb200076300 0x7fb200078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.524+0000 7fb20dffb640 1 -- 172.21.15.1:0/1619481552 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb210097390 con 0x7fb21c167760 2024-05-09T05:19:23.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.524+0000 7fb2210bb640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb200076300 0x7fb200078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:23.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.525+0000 7fb2210bb640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb200076300 0x7fb200078810 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb218007d00 tx=0x7fb218001e30 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:23.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:23.531+0000 7fb20dffb640 1 -- 172.21.15.1:0/1619481552 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb210060610 con 0x7fb21c167760 2024-05-09T05:19:24.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.030+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb21c050870 con 0x7fb21c167760 2024-05-09T05:19:24.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.032+0000 7fb20dffb640 1 -- 172.21.15.1:0/1619481552 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb210064910 con 0x7fb21c167760 2024-05-09T05:19:24.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.039+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb200076300 msgr2=0x7fb200078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:24.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.039+0000 7fb22331f640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb200076300 0x7fb200078810 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb218007d00 tx=0x7fb218001e30 comp rx=0 tx=0).stop 2024-05-09T05:19:24.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.039+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c167760 msgr2=0x7fb21c15b040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:24.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.039+0000 7fb22331f640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c167760 0x7fb21c15b040 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb21000f850 tx=0x7fb21000fd10 comp rx=0 tx=0).stop 2024-05-09T05:19:24.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.039+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 shutdown_connections 2024-05-09T05:19:24.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.039+0000 7fb22331f640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb200076300 0x7fb200078810 secure :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fb218007d00 tx=0x7fb218001e30 comp rx=0 tx=0).stop 2024-05-09T05:19:24.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.039+0000 7fb22331f640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb21c1f9c70 0x7fb21c1fa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:24.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.040+0000 7fb22331f640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb21c167760 0x7fb21c15b040 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:24.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.040+0000 7fb22331f640 1 --2- 172.21.15.1:0/1619481552 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb21c159410 0x7fb21c147a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:24.039 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.040+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 >> 172.21.15.1:0/1619481552 conn(0x7fb21c0ce290 msgr2=0x7fb21c156a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:24.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.042+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 shutdown_connections 2024-05-09T05:19:24.040 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:24.042+0000 7fb22331f640 1 -- 172.21.15.1:0/1619481552 wait complete. 2024-05-09T05:19:24.060 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:19:34.062 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:19:34.062 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:19:34.247 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.250+0000 7fc3b9300640 1 Processor -- start 2024-05-09T05:19:34.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.250+0000 7fc3b9300640 1 -- start start 2024-05-09T05:19:34.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.252+0000 7fc3b9300640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b4160600 0x7fc3b4156b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.253+0000 7fc3b9300640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3b406f5c0 con 0x7fc3b40cec10 2024-05-09T05:19:34.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.253+0000 7fc3b9300640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3b40cc6b0 con 0x7fc3b40cf990 2024-05-09T05:19:34.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.253+0000 7fc3b9300640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b40ccfe0 con 0x7fc3b4160600 2024-05-09T05:19:34.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.253+0000 7fc3b2ffd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fc3b40cf990 0x7fc3b40ce500 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:40994/0 (socket says 172.21.15.1:40994) 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.253+0000 7fc3b2ffd640 1 -- 172.21.15.1:0/496806267 learned_addr learned my addr 172.21.15.1:0/496806267 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.253+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b4160600 0x7fc3b4156b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.253+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4221199002 0 0) 0x7fc3b40cc6b0 con 0x7fc3b40cf990 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.253+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc39c003520 con 0x7fc3b40cf990 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3460252206 0 0) 0x7fc3b406f5c0 con 0x7fc3b40cec10 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc3b40cc6b0 con 0x7fc3b40cec10 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2286379921 0 0) 0x7fc39c003520 con 0x7fc3b40cf990 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b4160600 msgr2=0x7fc3b4156b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b4160600 0x7fc3b4156b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 >> v1:172.21.15.1:6789/0 conn(0x7fc3b40cec10 legacy=0x7fc3b415fef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b40cc990 con 0x7fc3b40cf990 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fc3a40016f0 con 0x7fc3b40cf990 2024-05-09T05:19:34.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 >> v1:172.21.15.196:6789/0 conn(0x7fc3b40cf990 legacy=0x7fc3b40ce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc39c003b90 0x7fc39c004020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c004800 0x7fc39c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc39c005210 0x7fc39c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b40ccfe0 con 0x7fc39c004800 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc39c00c0c0 con 0x7fc39c003b90 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc39c00c2c0 con 0x7fc39c005210 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b2ffd640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc39c003b90 0x7fc39c004020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc39c005210 0x7fc39c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.254+0000 7fc3b27fc640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c004800 0x7fc39c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.255+0000 7fc3b37fe640 1 -- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc39c003b90 msgr2=0x7fc39c004020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:34.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.255+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc39c003b90 0x7fc39c004020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.255+0000 7fc3b37fe640 1 -- 172.21.15.1:0/496806267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c004800 msgr2=0x7fc39c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:34.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.255+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c004800 0x7fc39c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.255+0000 7fc3b37fe640 1 -- 172.21.15.1:0/496806267 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b40cc990 con 0x7fc39c005210 2024-05-09T05:19:34.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.255+0000 7fc3b2ffd640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc39c003b90 0x7fc39c004020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.256+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc39c005210 0x7fc39c00b9c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40cc190 tx=0x7fc39c005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8eae2de46008cc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.256+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac037070 con 0x7fc39c005210 2024-05-09T05:19:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.256+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ac032640 con 0x7fc39c005210 2024-05-09T05:19:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.256+0000 7fc3b1ffb640 1 -- 172.21.15.1:0/496806267 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac009bd0 con 0x7fc39c005210 2024-05-09T05:19:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.257+0000 7fc3b9300640 1 -- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc39c005210 msgr2=0x7fc39c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.257+0000 7fc3b9300640 1 --2- 172.21.15.1:0/496806267 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc39c005210 0x7fc39c00b9c0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fc3b40cc190 tx=0x7fc39c005830 comp rx=0 tx=0).stop 2024-05-09T05:19:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.257+0000 7fc3b37fe640 1 -- 172.21.15.1:0/496806267 reap_dead start 2024-05-09T05:19:34.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.257+0000 7fc3b9300640 1 -- 172.21.15.1:0/496806267 shutdown_connections 2024-05-09T05:19:34.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.257+0000 7fc3b9300640 1 -- 172.21.15.1:0/496806267 >> 172.21.15.1:0/496806267 conn(0x7fc3b40b36c0 msgr2=0x7fc3b40b3ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:34.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.259+0000 7fc3b9300640 1 -- 172.21.15.1:0/496806267 shutdown_connections 2024-05-09T05:19:34.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.259+0000 7fc3b9300640 1 -- 172.21.15.1:0/496806267 wait complete. 2024-05-09T05:19:34.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.262+0000 7fc3b9300640 1 Processor -- start 2024-05-09T05:19:34.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.262+0000 7fc3b9300640 1 -- start start 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b9300640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c005210 0x7fc3b4151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b9300640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3b40b6020 0x7fc3b4158ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b9300640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b40d1e30 0x7fc3b40d2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b9300640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b406f370 con 0x7fc39c005210 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c005210 0x7fc3b4151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c005210 0x7fc3b4151230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36572/0 (socket says 172.21.15.1:36572) 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b2ffd640 1 -- 172.21.15.1:0/1384265392 learned_addr learned my addr 172.21.15.1:0/1384265392 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc3b406f1f0 con 0x7fc3b40b6020 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc3b4156b80 con 0x7fc3b40d1e30 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b40d1e30 0x7fc3b40d2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.264+0000 7fc3b27fc640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3b40b6020 0x7fc3b4158ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.265+0000 7fc3b37fe640 1 -- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3b40b6020 msgr2=0x7fc3b4158ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.265+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3b40b6020 0x7fc3b4158ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.265+0000 7fc3b37fe640 1 -- 172.21.15.1:0/1384265392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c005210 msgr2=0x7fc3b4151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.265+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c005210 0x7fc3b4151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.265+0000 7fc3b37fe640 1 -- 172.21.15.1:0/1384265392 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3b4151770 con 0x7fc3b40d1e30 2024-05-09T05:19:34.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.265+0000 7fc3b27fc640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3b40b6020 0x7fc3b4158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.265+0000 7fc3b37fe640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b40d1e30 0x7fc3b40d2240 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac031000 tx=0x7fc3ac009750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.266+0000 7fc39b7fe640 1 -- 172.21.15.1:0/1384265392 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3ac037040 con 0x7fc3b40d1e30 2024-05-09T05:19:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.266+0000 7fc39b7fe640 1 -- 172.21.15.1:0/1384265392 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3ac043070 con 0x7fc3b40d1e30 2024-05-09T05:19:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.266+0000 7fc39b7fe640 1 -- 172.21.15.1:0/1384265392 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3a4007e00 con 0x7fc3b40d1e30 2024-05-09T05:19:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.266+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3b40b54a0 con 0x7fc3b40d1e30 2024-05-09T05:19:34.264 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.266+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3b40b6c20 con 0x7fc3b40d1e30 2024-05-09T05:19:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.267+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3b4050870 con 0x7fc3b40d1e30 2024-05-09T05:19:34.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.268+0000 7fc39b7fe640 1 -- 172.21.15.1:0/1384265392 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc3ac007960 con 0x7fc3b40d1e30 2024-05-09T05:19:34.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.269+0000 7fc39b7fe640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3a00762d0 0x7fc3a00787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.270+0000 7fc39b7fe640 1 -- 172.21.15.1:0/1384265392 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc3ac0b7d00 con 0x7fc3b40d1e30 2024-05-09T05:19:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.270+0000 7fc3b2ffd640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3a00762d0 0x7fc3a00787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.267 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.270+0000 7fc3b2ffd640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3a00762d0 0x7fc3a00787e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4011920 tx=0x7fc3a4007d70 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:34.274 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.277+0000 7fc39b7fe640 1 -- 172.21.15.1:0/1384265392 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc3b4151770 con 0x7fc3b40d1e30 2024-05-09T05:19:34.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.780+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc3b40b6960 con 0x7fc3b40d1e30 2024-05-09T05:19:34.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.781+0000 7fc39b7fe640 1 -- 172.21.15.1:0/1384265392 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fc3ac081020 con 0x7fc3b40d1e30 2024-05-09T05:19:34.778 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:19:34.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3a00762d0 msgr2=0x7fc3a00787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3a00762d0 0x7fc3a00787e0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fc3a4011920 tx=0x7fc3a4007d70 comp rx=0 tx=0).stop 2024-05-09T05:19:34.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b40d1e30 msgr2=0x7fc3b40d2240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.781 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b40d1e30 0x7fc3b40d2240 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc3ac031000 tx=0x7fc3ac009750 comp rx=0 tx=0).stop 2024-05-09T05:19:34.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 shutdown_connections 2024-05-09T05:19:34.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3a00762d0 0x7fc3a00787e0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3b40b6020 0x7fc3b4158ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.782 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3b40d1e30 0x7fc3b40d2240 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 --2- 172.21.15.1:0/1384265392 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc39c005210 0x7fc3b4151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 >> 172.21.15.1:0/1384265392 conn(0x7fc3b40b36c0 msgr2=0x7fc3b40ca6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:34.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 shutdown_connections 2024-05-09T05:19:34.783 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.784+0000 7fc3b9300640 1 -- 172.21.15.1:0/1384265392 wait complete. 2024-05-09T05:19:34.801 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:19:34.801 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:19:34.801 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:19:34.801 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:19:34.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.985+0000 7f6947853640 1 Processor -- start 2024-05-09T05:19:34.983 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.985+0000 7f6947853640 1 -- start start 2024-05-09T05:19:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6947853640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f69400c46e0 0x7f6940159650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6947853640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f69400c4af0 0x7f6940159b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6947853640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f69400c3be0 con 0x7f69400c89a0 2024-05-09T05:19:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6947853640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f69400cb040 con 0x7f69400c46e0 2024-05-09T05:19:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6947853640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f69400c9290 con 0x7f69400c4af0 2024-05-09T05:19:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f69455ef640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f69400c89a0 0x7f69400ca930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:41004/0 (socket says 172.21.15.1:41004) 2024-05-09T05:19:34.985 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f69455ef640 1 -- 172.21.15.1:0/1561611001 learned_addr learned my addr 172.21.15.1:0/1561611001 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:34.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6944dee640 1 --2- 172.21.15.1:0/1561611001 >> v2:172.21.15.1:3300/0 conn(0x7f69400c46e0 0x7f6940159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6945df0640 1 --2- 172.21.15.1:0/1561611001 >> v2:172.21.15.196:3300/0 conn(0x7f69400c4af0 0x7f6940159b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6937fff640 1 -- 172.21.15.1:0/1561611001 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2482144483 0 0) 0x7f693c001110 con 0x7f69400c89a0 2024-05-09T05:19:34.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6937fff640 1 -- 172.21.15.1:0/1561611001 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f692c003520 con 0x7f69400c89a0 2024-05-09T05:19:34.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6945df0640 1 -- 172.21.15.1:0/1561611001 >> v2:172.21.15.1:3300/0 conn(0x7f69400c46e0 msgr2=0x7f6940159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:34.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6945df0640 1 --2- 172.21.15.1:0/1561611001 >> v2:172.21.15.1:3300/0 conn(0x7f69400c46e0 0x7f6940159650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6945df0640 1 -- 172.21.15.1:0/1561611001 >> v1:172.21.15.196:6789/0 conn(0x7f69400c89a0 legacy=0x7f69400ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.988+0000 7f6945df0640 1 -- 172.21.15.1:0/1561611001 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f694006e890 con 0x7f69400c4af0 2024-05-09T05:19:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.990+0000 7f6945df0640 1 --2- 172.21.15.1:0/1561611001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 0x7f6940159b90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6940073d30 tx=0x7f693802a540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94635609b9c060a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.990+0000 7f6937fff640 1 -- 172.21.15.1:0/1561611001 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6938007740 con 0x7f69400c4af0 2024-05-09T05:19:34.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.990+0000 7f6937fff640 1 -- 172.21.15.1:0/1561611001 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f69380078e0 con 0x7f69400c4af0 2024-05-09T05:19:34.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.991+0000 7f6937fff640 1 -- 172.21.15.1:0/1561611001 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6938007c90 con 0x7f69400c4af0 2024-05-09T05:19:34.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.991+0000 7f6947853640 1 -- 172.21.15.1:0/1561611001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 msgr2=0x7f6940159b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.991+0000 7f6947853640 1 --2- 172.21.15.1:0/1561611001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 0x7f6940159b90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6940073d30 tx=0x7f693802a540 comp rx=0 tx=0).stop 2024-05-09T05:19:34.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.993+0000 7f6947853640 1 -- 172.21.15.1:0/1561611001 shutdown_connections 2024-05-09T05:19:34.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.993+0000 7f6947853640 1 --2- 172.21.15.1:0/1561611001 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 0x7f6940159b90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.993+0000 7f6947853640 1 --2- 172.21.15.1:0/1561611001 >> v2:172.21.15.1:3300/0 conn(0x7f69400c46e0 0x7f6940159650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.993+0000 7f6947853640 1 -- 172.21.15.1:0/1561611001 >> 172.21.15.1:0/1561611001 conn(0x7f69400747b0 msgr2=0x7f69400ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:34.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.994+0000 7f6947853640 1 -- 172.21.15.1:0/1561611001 shutdown_connections 2024-05-09T05:19:34.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.994+0000 7f6947853640 1 -- 172.21.15.1:0/1561611001 wait complete. 2024-05-09T05:19:34.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.996+0000 7f6947853640 1 Processor -- start 2024-05-09T05:19:34.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.997+0000 7f6947853640 1 -- start start 2024-05-09T05:19:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.998+0000 7f6947853640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f69400c46e0 0x7f694014a800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.998+0000 7f6947853640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 0x7f69401596f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.998+0000 7f6947853640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69400c89a0 0x7f69400c7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:34.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.998+0000 7f6947853640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f69400c94e0 con 0x7f69400c89a0 2024-05-09T05:19:34.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.998+0000 7f6947853640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f69400c4140 con 0x7f69400c4af0 2024-05-09T05:19:34.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.998+0000 7f6947853640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f69400c97c0 con 0x7f69400c46e0 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f6944dee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 0x7f69401596f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f6944dee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 0x7f69401596f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:38890/0 (socket says 172.21.15.1:38890) 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f6944dee640 1 -- 172.21.15.1:0/91703383 learned_addr learned my addr 172.21.15.1:0/91703383 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f69455ef640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f69400c46e0 0x7f694014a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f6945df0640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69400c89a0 0x7f69400c7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f69455ef640 1 -- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 msgr2=0x7f69401596f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f69455ef640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 0x7f69401596f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f69455ef640 1 -- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69400c89a0 msgr2=0x7f69400c7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:34.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f69455ef640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69400c89a0 0x7f69400c7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f69455ef640 1 -- 172.21.15.1:0/91703383 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6940159c30 con 0x7f69400c46e0 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f6945df0640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69400c89a0 0x7f69400c7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:34.999+0000 7f6944dee640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f69400c4af0 0x7f69401596f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.000+0000 7f69455ef640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f69400c46e0 0x7f694014a800 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f693c00c3f0 tx=0x7f693c00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.000+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c0074e0 con 0x7f69400c46e0 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.000+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f693c0047e0 con 0x7f69400c46e0 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.000+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6940156360 con 0x7f69400c46e0 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.001+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f69401568e0 con 0x7f69400c46e0 2024-05-09T05:19:34.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.001+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693c004bd0 con 0x7f69400c46e0 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.002+0000 7f6947853640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6940154f20 0x7f69400c7eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.002+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6940050870 con 0x7f6940154f20 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.002+0000 7f6944dee640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6940154f20 0x7f69400c7eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.002+0000 7f6944dee640 1 -- 172.21.15.1:0/91703383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f69400ca360 con 0x7f6940154f20 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.003+0000 7f6944dee640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6940154f20 0x7f69400c7eb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6930009fd0 tx=0x7f693000f610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.003+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6930019070 con 0x7f6940154f20 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.003+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f693c011070 con 0x7f69400c46e0 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.004+0000 7f6935ffb640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69240761d0 0x7f69240786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.004+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f693c096fc0 con 0x7f69400c46e0 2024-05-09T05:19:35.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.004+0000 7f6945df0640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69240761d0 0x7f69240786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.004+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6930004030 con 0x7f6940154f20 2024-05-09T05:19:35.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.004+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6940154f20 msgr2=0x7f69400c7eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.004+0000 7f6935ffb640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6940154f20 0x7f69400c7eb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f6930009fd0 tx=0x7f693000f610 comp rx=0 tx=0).stop 2024-05-09T05:19:35.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.004+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6940154f20 msgr2=0x7f69400c7eb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:35.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.004+0000 7f6935ffb640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6940154f20 0x7f69400c7eb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.005+0000 7f6945df0640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69240761d0 0x7f69240786e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6938034000 tx=0x7f693802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.016+0000 7f6947853640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69401555a0 0x7f69401559b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.016+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f694006ccb0 con 0x7f69401555a0 2024-05-09T05:19:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.017+0000 7f6944dee640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69401555a0 0x7f69401559b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.017+0000 7f6944dee640 1 -- 172.21.15.1:0/91703383 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f694006c9d0 con 0x7f69401555a0 2024-05-09T05:19:35.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.018+0000 7f6944dee640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69401555a0 0x7f69401559b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f693001b000 tx=0x7f69300046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.018+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f693002c020 con 0x7f69401555a0 2024-05-09T05:19:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.018+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5065 (secure 0 0 0) 0x7f69300133a0 con 0x7f69401555a0 2024-05-09T05:19:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.018+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69401555a0 msgr2=0x7f69401559b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.018+0000 7f6935ffb640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69401555a0 0x7f69401559b0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f693001b000 tx=0x7f69300046c0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.019+0000 7f6935ffb640 1 -- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69401555a0 msgr2=0x7f69401559b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:35.016 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.019+0000 7f6935ffb640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f69401555a0 0x7f69401559b0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 100, 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 377213, 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.017 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.018 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.019 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:19:35.020 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.021 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:35.022 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.023 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.023 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:19:35.023 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.023 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.023 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.023 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:35.024 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.025 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:19:35.026 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.026 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.026 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.026 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.026 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:19:35.026 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:19:35.026 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.027 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.028 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:19:35.028 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.028 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.028 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.028 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.028 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:19:35.028 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.029 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.029 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.029 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.029 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:19:35.029 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:19:35.029 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.030 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.030 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.030 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.030 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.030 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.030 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:19:35.031 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:19:35.031 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.031 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.031 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.031 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:35.031 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.031 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.032 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:19:35.032 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.032 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.032 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.032 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:19:35.032 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.032 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.032 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:19:35.033 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.033 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.033 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.033 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 8 2024-05-09T05:19:35.033 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.033 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.033 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:19:35.034 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.034 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.034 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.034 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:19:35.034 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.035 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.035 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:19:35.035 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.035 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.035 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.035 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:19:35.037 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.037 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.038 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:19:35.038 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.019+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69240761d0 msgr2=0x7f69240786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.019+0000 7f6947853640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f69240761d0 0x7f69240786e0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f6938034000 tx=0x7f693802f000 comp rx=0 tx=0).stop 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.020+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f69400c46e0 msgr2=0x7f694014a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.020+0000 7f6947853640 1 --2- 172.21.15.1:0/91703383 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f69400c46e0 0x7f694014a800 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f693c00c3f0 tx=0x7f693c00c8b0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.021+0000 7f6945df0640 1 -- 172.21.15.1:0/91703383 reap_dead start 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.021+0000 7f6945df0640 1 -- 172.21.15.1:0/91703383 reap_dead start 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.021+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 shutdown_connections 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.021+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 >> 172.21.15.1:0/91703383 conn(0x7f69400747b0 msgr2=0x7f6940071470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.021+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 shutdown_connections 2024-05-09T05:19:35.041 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.021+0000 7f6947853640 1 -- 172.21.15.1:0/91703383 wait complete. 2024-05-09T05:19:35.042 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:19:35.042 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:19:35.042 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:19:35.141 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:19:35.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:19:35.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.228+0000 7fdd70140640 1 Processor -- start 2024-05-09T05:19:35.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.228+0000 7fdd70140640 1 -- start start 2024-05-09T05:19:35.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.230+0000 7fdd70140640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68152fe0 0x7fdd68072c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.230+0000 7fdd70140640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd68159410 0x7fdd68159820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.230+0000 7fdd70140640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdd6815d290 con 0x7fdd680cafb0 2024-05-09T05:19:35.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.230+0000 7fdd70140640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdd68073160 con 0x7fdd68159410 2024-05-09T05:19:35.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.230+0000 7fdd70140640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdd68158f80 con 0x7fdd68152fe0 2024-05-09T05:19:35.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.231+0000 7fdd6e6dd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd68159410 0x7fdd68159820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.231+0000 7fdd6d6db640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68152fe0 0x7fdd68072c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.231+0000 7fdd6e6dd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdd68159410 0x7fdd68159820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36612/0 (socket says 172.21.15.1:36612) 2024-05-09T05:19:35.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.231+0000 7fdd6e6dd640 1 -- 172.21.15.1:0/445261833 learned_addr learned my addr 172.21.15.1:0/445261833 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:35.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.231+0000 7fdd6d6db640 1 -- 172.21.15.1:0/445261833 >> v2:172.21.15.1:3300/0 conn(0x7fdd68159410 msgr2=0x7fdd68159820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:35.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.231+0000 7fdd6d6db640 1 --2- 172.21.15.1:0/445261833 >> v2:172.21.15.1:3300/0 conn(0x7fdd68159410 0x7fdd68159820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.231+0000 7fdd6d6db640 1 -- 172.21.15.1:0/445261833 >> v1:172.21.15.1:6789/0 conn(0x7fdd680cafb0 legacy=0x7fdd680c7230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:35.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.231+0000 7fdd6d6db640 1 -- 172.21.15.1:0/445261833 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd6815c710 con 0x7fdd68152fe0 2024-05-09T05:19:35.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.232+0000 7fdd6d6db640 1 --2- 172.21.15.1:0/445261833 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68152fe0 0x7fdd68072c20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdd680742d0 tx=0x7fdd5802a150 comp rx=0 tx=0).ready entity=mon.2 client_cookie=afbbc63fe2bedd3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.232+0000 7fdd6ceda640 1 -- 172.21.15.1:0/445261833 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58037070 con 0x7fdd68152fe0 2024-05-09T05:19:35.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.232+0000 7fdd6ceda640 1 -- 172.21.15.1:0/445261833 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd58032440 con 0x7fdd68152fe0 2024-05-09T05:19:35.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.232+0000 7fdd6ceda640 1 -- 172.21.15.1:0/445261833 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58032810 con 0x7fdd68152fe0 2024-05-09T05:19:35.230 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.234+0000 7fdd70140640 1 -- 172.21.15.1:0/445261833 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68152fe0 msgr2=0x7fdd68072c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.234+0000 7fdd70140640 1 --2- 172.21.15.1:0/445261833 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68152fe0 0x7fdd68072c20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fdd680742d0 tx=0x7fdd5802a150 comp rx=0 tx=0).stop 2024-05-09T05:19:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.234+0000 7fdd70140640 1 -- 172.21.15.1:0/445261833 shutdown_connections 2024-05-09T05:19:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.234+0000 7fdd70140640 1 --2- 172.21.15.1:0/445261833 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68152fe0 0x7fdd68072c20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.234+0000 7fdd70140640 1 --2- 172.21.15.1:0/445261833 >> v2:172.21.15.1:3300/0 conn(0x7fdd68159410 0x7fdd68159820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.234+0000 7fdd70140640 1 -- 172.21.15.1:0/445261833 >> 172.21.15.1:0/445261833 conn(0x7fdd68070f40 msgr2=0x7fdd6806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:35.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.234+0000 7fdd70140640 1 -- 172.21.15.1:0/445261833 shutdown_connections 2024-05-09T05:19:35.232 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.234+0000 7fdd70140640 1 -- 172.21.15.1:0/445261833 wait complete. 2024-05-09T05:19:35.234 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.237+0000 7fdd70140640 1 Processor -- start 2024-05-09T05:19:35.235 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.237+0000 7fdd70140640 1 -- start start 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.239+0000 7fdd70140640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd680cafb0 0x7fdd680c2a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.239+0000 7fdd70140640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd68152fe0 0x7fdd680c95e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.239+0000 7fdd70140640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68159410 0x7fdd6814f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.239+0000 7fdd70140640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6815cfb0 con 0x7fdd680cafb0 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.239+0000 7fdd70140640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdd6815ccd0 con 0x7fdd68152fe0 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.239+0000 7fdd70140640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdd681533f0 con 0x7fdd68159410 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6dedc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd680cafb0 0x7fdd680c2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6d6db640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd68152fe0 0x7fdd680c95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.237 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6d6db640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd68152fe0 0x7fdd680c95e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:38898/0 (socket says 172.21.15.1:38898) 2024-05-09T05:19:35.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6d6db640 1 -- 172.21.15.1:0/2066409471 learned_addr learned my addr 172.21.15.1:0/2066409471 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:35.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6e6dd640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68159410 0x7fdd6814f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6dedc640 1 -- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68159410 msgr2=0x7fdd6814f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6dedc640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdd68159410 0x7fdd6814f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6dedc640 1 -- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd68152fe0 msgr2=0x7fdd680c95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6dedc640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd68152fe0 0x7fdd680c95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.240+0000 7fdd6dedc640 1 -- 172.21.15.1:0/2066409471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd680c2fa0 con 0x7fdd680cafb0 2024-05-09T05:19:35.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.241+0000 7fdd6dedc640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd680cafb0 0x7fdd680c2a60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000bd50 tx=0x7fdd6000be40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.243+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd60019070 con 0x7fdd680cafb0 2024-05-09T05:19:35.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.243+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd60014440 con 0x7fdd680cafb0 2024-05-09T05:19:35.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.243+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd60014850 con 0x7fdd680cafb0 2024-05-09T05:19:35.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.243+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd680cc930 con 0x7fdd680cafb0 2024-05-09T05:19:35.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.243+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd680ccef0 con 0x7fdd680cafb0 2024-05-09T05:19:35.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.244+0000 7fdd70140640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c3170 0x7fdd6814fca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.244+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdd680cd320 con 0x7fdd680c3170 2024-05-09T05:19:35.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.245+0000 7fdd6d6db640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c3170 0x7fdd6814fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.245+0000 7fdd6d6db640 1 -- 172.21.15.1:0/2066409471 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fdd681574d0 con 0x7fdd680c3170 2024-05-09T05:19:35.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.245+0000 7fdd6d6db640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c3170 0x7fdd6814fca0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdd680c9f40 tx=0x7fdd580046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.246+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58043020 con 0x7fdd680c3170 2024-05-09T05:19:35.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.246+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdd6001c070 con 0x7fdd680cafb0 2024-05-09T05:19:35.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.246+0000 7fdd5e7fc640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd4c076220 0x7fdd4c078730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.246+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdd60097450 con 0x7fdd680cafb0 2024-05-09T05:19:35.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.247+0000 7fdd6e6dd640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd4c076220 0x7fdd4c078730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.247+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fdd58038040 con 0x7fdd680c3170 2024-05-09T05:19:35.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.247+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c3170 msgr2=0x7fdd6814fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.247+0000 7fdd5e7fc640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c3170 0x7fdd6814fca0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdd680c9f40 tx=0x7fdd580046c0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.247+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c3170 msgr2=0x7fdd6814fca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:35.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.247+0000 7fdd5e7fc640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c3170 0x7fdd6814fca0 secure :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fdd680c9f40 tx=0x7fdd580046c0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.248+0000 7fdd6e6dd640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd4c076220 0x7fdd4c078730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400bac0 tx=0x7fdd64008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.264+0000 7fdd70140640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c4690 0x7fdd681fa020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.264+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdd680cc6e0 con 0x7fdd680c4690 2024-05-09T05:19:35.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.264+0000 7fdd6d6db640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c4690 0x7fdd681fa020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.265+0000 7fdd6d6db640 1 -- 172.21.15.1:0/2066409471 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fdd6806c6f0 con 0x7fdd680c4690 2024-05-09T05:19:35.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.265+0000 7fdd6d6db640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c4690 0x7fdd681fa020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd68078090 tx=0x7fdd58008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.262 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.265+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd58037040 con 0x7fdd680c4690 2024-05-09T05:19:35.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7fdd5804a0d0 con 0x7fdd680c4690 2024-05-09T05:19:35.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c4690 msgr2=0x7fdd681fa020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd5e7fc640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c4690 0x7fdd681fa020 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd68078090 tx=0x7fdd58008030 comp rx=0 tx=0).stop 2024-05-09T05:19:35.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd5e7fc640 1 -- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c4690 msgr2=0x7fdd681fa020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:35.263 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd5e7fc640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdd680c4690 0x7fdd681fa020 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.263 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:19:35.263 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:35.263 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 100, 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 125669, 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:19:35.264 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:19:35.265 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.266 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:19:35.267 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.268 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.269 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.270 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.271 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:35.272 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.273 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.274 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:19:35.275 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:19:35.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd4c076220 msgr2=0x7fdd4c078730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd70140640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd4c076220 0x7fdd4c078730 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fdd6400bac0 tx=0x7fdd64008040 comp rx=0 tx=0).stop 2024-05-09T05:19:35.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd680cafb0 msgr2=0x7fdd680c2a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.276 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd70140640 1 --2- 172.21.15.1:0/2066409471 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdd680cafb0 0x7fdd680c2a60 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fdd6000bd50 tx=0x7fdd6000be40 comp rx=0 tx=0).stop 2024-05-09T05:19:35.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd6e6dd640 1 -- 172.21.15.1:0/2066409471 reap_dead start 2024-05-09T05:19:35.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd6e6dd640 1 -- 172.21.15.1:0/2066409471 reap_dead start 2024-05-09T05:19:35.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 shutdown_connections 2024-05-09T05:19:35.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 >> 172.21.15.1:0/2066409471 conn(0x7fdd68070f40 msgr2=0x7fdd681507a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:35.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 shutdown_connections 2024-05-09T05:19:35.277 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.266+0000 7fdd70140640 1 -- 172.21.15.1:0/2066409471 wait complete. 2024-05-09T05:19:35.282 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:19:35.282 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:19:35.282 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:19:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.507+0000 7fcbec1c0640 1 Processor -- start 2024-05-09T05:19:35.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.508+0000 7fcbec1c0640 1 -- start start 2024-05-09T05:19:35.507 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbec1c0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbec1c0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fcbe40740a0 0x7fcbe40744b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbec1c0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcbe4073ac0 con 0x7fcbe40c86d0 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbec1c0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fcbe40c8ae0 con 0x7fcbe40740a0 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbec1c0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe40c91f0 con 0x7fcbe415b1e0 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbe975b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fcbe40c86d0 0x7fcbe406c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:38924/0 (socket says 172.21.15.1:38924) 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbe975b640 1 -- 172.21.15.1:0/3293543374 learned_addr learned my addr 172.21.15.1:0/3293543374 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbe9f5c640 1 --2- 172.21.15.1:0/3293543374 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.510+0000 7fcbea75d640 1 --2- 172.21.15.1:0/3293543374 >> v2:172.21.15.196:3300/0 conn(0x7fcbe40740a0 0x7fcbe40744b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.511+0000 7fcbe8f5a640 1 -- 172.21.15.1:0/3293543374 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 796688717 0 0) 0x7fcbd4001110 con 0x7fcbe40c86d0 2024-05-09T05:19:35.508 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.511+0000 7fcbe8f5a640 1 -- 172.21.15.1:0/3293543374 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fcbd0003520 con 0x7fcbe40c86d0 2024-05-09T05:19:35.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.511+0000 7fcbea75d640 1 -- 172.21.15.1:0/3293543374 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 msgr2=0x7fcbe40c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:35.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.511+0000 7fcbea75d640 1 --2- 172.21.15.1:0/3293543374 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.511+0000 7fcbea75d640 1 -- 172.21.15.1:0/3293543374 >> v1:172.21.15.1:6789/0 conn(0x7fcbe40c86d0 legacy=0x7fcbe406c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.511+0000 7fcbea75d640 1 -- 172.21.15.1:0/3293543374 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe40c9490 con 0x7fcbe40740a0 2024-05-09T05:19:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.512+0000 7fcbea75d640 1 --2- 172.21.15.1:0/3293543374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40740a0 0x7fcbe40744b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009d90 tx=0x7fcbdc02a710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba1fa009fc0c2cff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.513+0000 7fcbe8f5a640 1 -- 172.21.15.1:0/3293543374 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc007740 con 0x7fcbe40740a0 2024-05-09T05:19:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.513+0000 7fcbe8f5a640 1 -- 172.21.15.1:0/3293543374 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbdc0078e0 con 0x7fcbe40740a0 2024-05-09T05:19:35.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.513+0000 7fcbe8f5a640 1 -- 172.21.15.1:0/3293543374 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbdc007c90 con 0x7fcbe40740a0 2024-05-09T05:19:35.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.514+0000 7fcbec1c0640 1 -- 172.21.15.1:0/3293543374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40740a0 msgr2=0x7fcbe40744b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.514+0000 7fcbec1c0640 1 --2- 172.21.15.1:0/3293543374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40740a0 0x7fcbe40744b0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc009d90 tx=0x7fcbdc02a710 comp rx=0 tx=0).stop 2024-05-09T05:19:35.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.514+0000 7fcbec1c0640 1 -- 172.21.15.1:0/3293543374 shutdown_connections 2024-05-09T05:19:35.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.514+0000 7fcbec1c0640 1 --2- 172.21.15.1:0/3293543374 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40740a0 0x7fcbe40744b0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.514+0000 7fcbec1c0640 1 --2- 172.21.15.1:0/3293543374 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.514+0000 7fcbec1c0640 1 -- 172.21.15.1:0/3293543374 >> 172.21.15.1:0/3293543374 conn(0x7fcbe40cdcd0 msgr2=0x7fcbe40ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:35.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.514+0000 7fcbec1c0640 1 -- 172.21.15.1:0/3293543374 shutdown_connections 2024-05-09T05:19:35.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.514+0000 7fcbec1c0640 1 -- 172.21.15.1:0/3293543374 wait complete. 2024-05-09T05:19:35.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.517+0000 7fcbec1c0640 1 Processor -- start 2024-05-09T05:19:35.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.518+0000 7fcbec1c0640 1 -- start start 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbec1c0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40740a0 0x7fcbe41faa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbec1c0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40c86d0 0x7fcbe41faf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbec1c0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbec1c0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe40c9d30 con 0x7fcbe40740a0 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbec1c0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcbe40c9a50 con 0x7fcbe40c86d0 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbec1c0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe40749f0 con 0x7fcbe415b1e0 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbe9f5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40740a0 0x7fcbe41faa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbe975b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40c86d0 0x7fcbe41faf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbea75d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbe975b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40c86d0 0x7fcbe41faf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:38924/0 (socket says 172.21.15.1:38924) 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbea75d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:36626/0 (socket says 172.21.15.1:36626) 2024-05-09T05:19:35.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbe9f5c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40740a0 0x7fcbe41faa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:36632/0 (socket says 172.21.15.1:36632) 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.520+0000 7fcbe975b640 1 -- 172.21.15.1:0/2116022080 learned_addr learned my addr 172.21.15.1:0/2116022080 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbe975b640 1 -- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 msgr2=0x7fcbe40c4b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbe975b640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbe975b640 1 -- 172.21.15.1:0/2116022080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40740a0 msgr2=0x7fcbe41faa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbe975b640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcbe40740a0 0x7fcbe41faa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbe975b640 1 -- 172.21.15.1:0/2116022080 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbe40cda60 con 0x7fcbe40c86d0 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbea75d640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe415b1e0 0x7fcbe40c4b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbe975b640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40c86d0 0x7fcbe41faf80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400c730 tx=0x7fcbd400cbf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4005ab0 con 0x7fcbe40c86d0 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcbd4001d30 con 0x7fcbe40c86d0 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbe414bd50 con 0x7fcbe40c86d0 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.521+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbd4005e50 con 0x7fcbe40c86d0 2024-05-09T05:19:35.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.522+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbe40c7af0 con 0x7fcbe40c86d0 2024-05-09T05:19:35.521 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.523+0000 7fcbec1c0640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe41594a0 0x7fcbe40c50a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.523+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe4159c80 con 0x7fcbe41594a0 2024-05-09T05:19:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.524+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcbd4007070 con 0x7fcbe40c86d0 2024-05-09T05:19:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.525+0000 7fcbda7fc640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcbc8076570 0x7fcbc8078a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.525+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcbd4097160 con 0x7fcbe40c86d0 2024-05-09T05:19:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.525+0000 7fcbea75d640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcbc8076570 0x7fcbc8078a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.526+0000 7fcbe9f5c640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe41594a0 0x7fcbe40c50a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.526+0000 7fcbea75d640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcbc8076570 0x7fcbc8078a80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc034000 tx=0x7fcbdc02f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.526+0000 7fcbe9f5c640 1 -- 172.21.15.1:0/2116022080 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fcbe4050870 con 0x7fcbe41594a0 2024-05-09T05:19:35.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.527+0000 7fcbe9f5c640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe41594a0 0x7fcbe40c50a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0011890 tx=0x7fcbe0011d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.527+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0019070 con 0x7fcbe41594a0 2024-05-09T05:19:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.527+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fcbe0004240 con 0x7fcbe41594a0 2024-05-09T05:19:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.528+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe41594a0 msgr2=0x7fcbe40c50a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.528+0000 7fcbda7fc640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe41594a0 0x7fcbe40c50a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0011890 tx=0x7fcbe0011d50 comp rx=0 tx=0).stop 2024-05-09T05:19:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.528+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe41594a0 msgr2=0x7fcbe40c50a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:35.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.528+0000 7fcbda7fc640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe41594a0 0x7fcbe40c50a0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fcbe0011890 tx=0x7fcbe0011d50 comp rx=0 tx=0).stop 2024-05-09T05:19:35.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.543+0000 7fcbec1c0640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe4157850 0x7fcbe4157c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.540 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.543+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcbe40c3590 con 0x7fcbe4157850 2024-05-09T05:19:35.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.544+0000 7fcbe9f5c640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe4157850 0x7fcbe4157c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.541 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.544+0000 7fcbe9f5c640 1 -- 172.21.15.1:0/2116022080 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fcbe4073cd0 con 0x7fcbe4157850 2024-05-09T05:19:35.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.545+0000 7fcbe9f5c640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe4157850 0x7fcbe4157c60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbe001b000 tx=0x7fcbe00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.545+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcbe0019040 con 0x7fcbe4157850 2024-05-09T05:19:35.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.545+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fcbe00044d0 con 0x7fcbe4157850 2024-05-09T05:19:35.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.545+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe4157850 msgr2=0x7fcbe4157c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.545+0000 7fcbda7fc640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe4157850 0x7fcbe4157c60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbe001b000 tx=0x7fcbe00046c0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.545+0000 7fcbda7fc640 1 -- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe4157850 msgr2=0x7fcbe4157c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:35.542 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.545+0000 7fcbda7fc640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcbe4157850 0x7fcbe4157c60 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fcbe001b000 tx=0x7fcbe00046c0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 100, 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 17434, 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:19:35.543 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.544 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.545 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.545 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.545 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:19:35.545 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:19:35.545 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.545 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.545 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.546 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.546 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.546 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.546 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.546 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.547 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.547 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.547 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.547 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.547 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.547 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:19:35.548 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:19:35.549 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:19:35.549 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:19:35.549 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:19:35.549 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:19:35.549 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:19:35.549 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:35.549 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:19:35.549 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:35.550 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:35.550 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:35.550 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:35.550 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:35.550 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:35.550 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:35.550 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:35.550 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:35.551 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.552 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:19:35.553 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.554 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:19:35.555 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.556 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:19:35.557 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:19:35.557 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.557 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.557 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.557 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:35.557 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.557 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:35.557 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:19:35.558 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:19:35.559 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.546+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcbc8076570 msgr2=0x7fcbc8078a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.546+0000 7fcbec1c0640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcbc8076570 0x7fcbc8078a80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7fcbdc034000 tx=0x7fcbdc02f000 comp rx=0 tx=0).stop 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.546+0000 7fcbea75d640 1 -- 172.21.15.1:0/2116022080 reap_dead start 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.546+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40c86d0 msgr2=0x7fcbe41faf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.546+0000 7fcbec1c0640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40c86d0 0x7fcbe41faf80 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fcbd400c730 tx=0x7fcbd400cbf0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.547+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 shutdown_connections 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.547+0000 7fcbec1c0640 1 --2- 172.21.15.1:0/2116022080 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcbe40c86d0 0x7fcbe41faf80 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.547+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 >> 172.21.15.1:0/2116022080 conn(0x7fcbe40cdcd0 msgr2=0x7fcbe406f010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.547+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 shutdown_connections 2024-05-09T05:19:35.560 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.547+0000 7fcbec1c0640 1 -- 172.21.15.1:0/2116022080 wait complete. 2024-05-09T05:19:35.562 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:19:35.563 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['a'] 2024-05-09T05:19:35.563 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:19:35.563 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.a 2024-05-09T05:19:35.563 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-05-09T05:19:35.563 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T05:19:35.563 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:19:35.585 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T05:19:35.585 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:19:35.585 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:19:35.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.798+0000 7f8e7c7c7640 1 Processor -- start 2024-05-09T05:19:35.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.798+0000 7f8e7c7c7640 1 -- start start 2024-05-09T05:19:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.800+0000 7f8e7c7c7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e7406c6f0 0x7f8e740cd330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.800+0000 7f8e7c7c7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e7406d8a0 con 0x7f8e74068a50 2024-05-09T05:19:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.800+0000 7f8e7c7c7640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8e7415a680 con 0x7f8e741546f0 2024-05-09T05:19:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.800+0000 7f8e7c7c7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8e7406db40 con 0x7f8e7406c6f0 2024-05-09T05:19:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.800+0000 7f8e7a563640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f8e74068a50 legacy=0x7f8e740c77a0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:19:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.801+0000 7f8e7ad64640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f8e741546f0 0x7f8e7415ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:41018/0 (socket says 172.21.15.1:41018) 2024-05-09T05:19:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.801+0000 7f8e7ad64640 1 -- 172.21.15.1:0/942904667 learned_addr learned my addr 172.21.15.1:0/942904667 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:35.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.801+0000 7f8e79d62640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e7406c6f0 0x7f8e740cd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.801+0000 7f8e79561640 1 -- 172.21.15.1:0/942904667 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3745644400 0 0) 0x7f8e7415a680 con 0x7f8e741546f0 2024-05-09T05:19:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.801+0000 7f8e79561640 1 -- 172.21.15.1:0/942904667 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f8e60003520 con 0x7f8e741546f0 2024-05-09T05:19:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.801+0000 7f8e79d62640 1 -- 172.21.15.1:0/942904667 >> v1:172.21.15.196:6789/0 conn(0x7f8e741546f0 legacy=0x7f8e7415ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.801+0000 7f8e79d62640 1 -- 172.21.15.1:0/942904667 >> v1:172.21.15.1:6789/0 conn(0x7f8e74068a50 legacy=0x7f8e740c77a0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:35.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.801+0000 7f8e79d62640 1 -- 172.21.15.1:0/942904667 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7406de20 con 0x7f8e7406c6f0 2024-05-09T05:19:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.803+0000 7f8e79d62640 1 --2- 172.21.15.1:0/942904667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e7406c6f0 0x7f8e740cd330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e680097d0 tx=0x7f8e6802a030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3525bf442dca111e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.803+0000 7f8e79561640 1 -- 172.21.15.1:0/942904667 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68037070 con 0x7f8e7406c6f0 2024-05-09T05:19:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.803+0000 7f8e79561640 1 -- 172.21.15.1:0/942904667 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e6802ad80 con 0x7f8e7406c6f0 2024-05-09T05:19:35.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.803+0000 7f8e79561640 1 -- 172.21.15.1:0/942904667 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e68037070 con 0x7f8e7406c6f0 2024-05-09T05:19:35.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.805+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/942904667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e7406c6f0 msgr2=0x7f8e740cd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.805+0000 7f8e7c7c7640 1 --2- 172.21.15.1:0/942904667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e7406c6f0 0x7f8e740cd330 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f8e680097d0 tx=0x7f8e6802a030 comp rx=0 tx=0).stop 2024-05-09T05:19:35.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.806+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/942904667 shutdown_connections 2024-05-09T05:19:35.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.806+0000 7f8e7c7c7640 1 --2- 172.21.15.1:0/942904667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e7406c6f0 0x7f8e740cd330 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.806+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/942904667 >> 172.21.15.1:0/942904667 conn(0x7f8e740ce110 msgr2=0x7f8e740ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:35.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.806+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/942904667 shutdown_connections 2024-05-09T05:19:35.803 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.806+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/942904667 wait complete. 2024-05-09T05:19:35.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.809+0000 7f8e7c7c7640 1 Processor -- start 2024-05-09T05:19:35.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.809+0000 7f8e7c7c7640 1 -- start start 2024-05-09T05:19:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.810+0000 7f8e7c7c7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e74068a50 0x7f8e741f3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.810+0000 7f8e7c7c7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e7406c6f0 0x7f8e74158990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.810+0000 7f8e7c7c7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e741546f0 0x7f8e74142f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7c7c7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8e740cc4b0 con 0x7f8e74068a50 2024-05-09T05:19:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7a563640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e74068a50 msgr2=0x7f8e741f3930 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:35.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7a563640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e74068a50 0x7f8e741f3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7c7c7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8e740cc1d0 con 0x7f8e7406c6f0 2024-05-09T05:19:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7c7c7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8e74154b00 con 0x7f8e741546f0 2024-05-09T05:19:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7ad64640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e741546f0 0x7f8e74142f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e79d62640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e7406c6f0 0x7f8e74158990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7ad64640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e741546f0 0x7f8e74142f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:36738/0 (socket says 172.21.15.1:36738) 2024-05-09T05:19:35.809 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7ad64640 1 -- 172.21.15.1:0/1900235969 learned_addr learned my addr 172.21.15.1:0/1900235969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7ad64640 1 -- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e7406c6f0 msgr2=0x7f8e74158990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7ad64640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e7406c6f0 0x7f8e74158990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7ad64640 1 -- 172.21.15.1:0/1900235969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e74068a50 msgr2=0x7f8e741f3930 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7ad64640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e74068a50 0x7f8e741f3930 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.811+0000 7f8e7ad64640 1 -- 172.21.15.1:0/1900235969 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e7414c800 con 0x7f8e741546f0 2024-05-09T05:19:35.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.812+0000 7f8e7ad64640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e741546f0 0x7f8e74142f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00c3f0 tx=0x7f8e6c00c8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.812+0000 7f8e66ffd640 1 -- 172.21.15.1:0/1900235969 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c0074e0 con 0x7f8e741546f0 2024-05-09T05:19:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.812+0000 7f8e66ffd640 1 -- 172.21.15.1:0/1900235969 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8e6c0047e0 con 0x7f8e741546f0 2024-05-09T05:19:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.812+0000 7f8e66ffd640 1 -- 172.21.15.1:0/1900235969 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e6c004bd0 con 0x7f8e741546f0 2024-05-09T05:19:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.813+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e74158ed0 con 0x7f8e741546f0 2024-05-09T05:19:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.813+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e7415a230 con 0x7f8e741546f0 2024-05-09T05:19:35.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.814+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e7414c9d0 con 0x7f8e741546f0 2024-05-09T05:19:35.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.816+0000 7f8e66ffd640 1 -- 172.21.15.1:0/1900235969 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8e6c011070 con 0x7f8e741546f0 2024-05-09T05:19:35.814 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.817+0000 7f8e66ffd640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8e58076300 0x7f8e58078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.817+0000 7f8e66ffd640 1 -- 172.21.15.1:0/1900235969 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8e6c0975c0 con 0x7f8e741546f0 2024-05-09T05:19:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.817+0000 7f8e7a563640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8e58076300 0x7f8e58078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:35.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.818+0000 7f8e7a563640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8e58076300 0x7f8e58078810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8e70003d00 tx=0x7f8e70009270 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:35.820 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:35.823+0000 7f8e66ffd640 1 -- 172.21.15.1:0/1900235969 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f8e6c061010 con 0x7f8e741546f0 2024-05-09T05:19:36.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.306+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f8e7415eb10 con 0x7f8e741546f0 2024-05-09T05:19:36.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.306+0000 7f8e66ffd640 1 -- 172.21.15.1:0/1900235969 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f8e6c060910 con 0x7f8e741546f0 2024-05-09T05:19:36.304 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:19:36.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8e58076300 msgr2=0x7f8e58078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8e58076300 0x7f8e58078810 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8e70003d00 tx=0x7f8e70009270 comp rx=0 tx=0).stop 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e741546f0 msgr2=0x7f8e74142f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e741546f0 0x7f8e74142f70 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8e6c00c3f0 tx=0x7f8e6c00c8b0 comp rx=0 tx=0).stop 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 shutdown_connections 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8e58076300 0x7f8e58078810 secure :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f8e70003d00 tx=0x7f8e70009270 comp rx=0 tx=0).stop 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8e7406c6f0 0x7f8e74158990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8e741546f0 0x7f8e74142f70 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 --2- 172.21.15.1:0/1900235969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8e74068a50 0x7f8e741f3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:36.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 >> 172.21.15.1:0/1900235969 conn(0x7f8e740ce110 msgr2=0x7f8e740735c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:36.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 shutdown_connections 2024-05-09T05:19:36.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:36.310+0000 7f8e7c7c7640 1 -- 172.21.15.1:0/1900235969 wait complete. 2024-05-09T05:19:39.327 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:19:39.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.507+0000 7f177c1b7640 1 Processor -- start 2024-05-09T05:19:39.505 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.507+0000 7f177c1b7640 1 -- start start 2024-05-09T05:19:39.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.509+0000 7f177c1b7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f17740d16b0 0x7f17740d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:39.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.510+0000 7f177c1b7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f177406f1b0 0x7f17740ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:39.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.510+0000 7f1779f53640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f17740d16b0 0x7f17740d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:39.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.510+0000 7f177c1b7640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f177406f6d0 con 0x7f177415ca90 2024-05-09T05:19:39.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.510+0000 7f177c1b7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f17740cd0e0 con 0x7f177406f1b0 2024-05-09T05:19:39.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.510+0000 7f177c1b7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f17740cc4d0 con 0x7f17740d16b0 2024-05-09T05:19:39.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.510+0000 7f177a754640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f177415ca90 0x7f177415cea0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:41026/0 (socket says 172.21.15.1:41026) 2024-05-09T05:19:39.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.510+0000 7f1779f53640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f17740d16b0 0x7f17740d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:36744/0 (socket says 172.21.15.1:36744) 2024-05-09T05:19:39.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.510+0000 7f1779f53640 1 -- 172.21.15.1:0/235274335 learned_addr learned my addr 172.21.15.1:0/235274335 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:39.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.511+0000 7f1779752640 1 -- 172.21.15.1:0/235274335 >> v2:172.21.15.1:3300/0 conn(0x7f177406f1b0 msgr2=0x7f17740ce740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:39.510 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.511+0000 7f1779752640 1 --2- 172.21.15.1:0/235274335 >> v2:172.21.15.1:3300/0 conn(0x7f177406f1b0 0x7f17740ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:39.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.511+0000 7f1778f51640 1 -- 172.21.15.1:0/235274335 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1886984567 0 0) 0x7f177406f6d0 con 0x7f177415ca90 2024-05-09T05:19:39.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.511+0000 7f1778f51640 1 -- 172.21.15.1:0/235274335 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1760003520 con 0x7f177415ca90 2024-05-09T05:19:39.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.511+0000 7f1779f53640 1 -- 172.21.15.1:0/235274335 >> v2:172.21.15.1:3300/0 conn(0x7f177406f1b0 msgr2=0x7f17740ce740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:39.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.511+0000 7f1779f53640 1 --2- 172.21.15.1:0/235274335 >> v2:172.21.15.1:3300/0 conn(0x7f177406f1b0 0x7f17740ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:39.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.511+0000 7f1779f53640 1 -- 172.21.15.1:0/235274335 >> v1:172.21.15.196:6789/0 conn(0x7f177415ca90 legacy=0x7f177415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:39.511 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.511+0000 7f1779f53640 1 -- 172.21.15.1:0/235274335 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17740cc7b0 con 0x7f17740d16b0 2024-05-09T05:19:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.513+0000 7f1779f53640 1 --2- 172.21.15.1:0/235274335 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f17740d16b0 0x7f17740d03d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1770009870 tx=0x7f177002a310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5960bd9456fb2cb4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:39.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.513+0000 7f1778f51640 1 -- 172.21.15.1:0/235274335 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1770037070 con 0x7f17740d16b0 2024-05-09T05:19:39.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.513+0000 7f1778f51640 1 -- 172.21.15.1:0/235274335 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1770032640 con 0x7f17740d16b0 2024-05-09T05:19:39.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.513+0000 7f1778f51640 1 -- 172.21.15.1:0/235274335 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f177002ae60 con 0x7f17740d16b0 2024-05-09T05:19:39.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.515+0000 7f177c1b7640 1 -- 172.21.15.1:0/235274335 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f17740d16b0 msgr2=0x7f17740d03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:39.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.515+0000 7f177c1b7640 1 --2- 172.21.15.1:0/235274335 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f17740d16b0 0x7f17740d03d0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f1770009870 tx=0x7f177002a310 comp rx=0 tx=0).stop 2024-05-09T05:19:39.514 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.515+0000 7f177c1b7640 1 -- 172.21.15.1:0/235274335 shutdown_connections 2024-05-09T05:19:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.515+0000 7f177c1b7640 1 --2- 172.21.15.1:0/235274335 >> v2:172.21.15.1:3300/0 conn(0x7f177406f1b0 0x7f17740ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.515+0000 7f177c1b7640 1 --2- 172.21.15.1:0/235274335 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f17740d16b0 0x7f17740d03d0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.515+0000 7f177c1b7640 1 -- 172.21.15.1:0/235274335 >> 172.21.15.1:0/235274335 conn(0x7f17740b3660 msgr2=0x7f17740b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.515+0000 7f177c1b7640 1 -- 172.21.15.1:0/235274335 shutdown_connections 2024-05-09T05:19:39.515 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.515+0000 7f177c1b7640 1 -- 172.21.15.1:0/235274335 wait complete. 2024-05-09T05:19:39.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.518+0000 7f177c1b7640 1 Processor -- start 2024-05-09T05:19:39.516 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.518+0000 7f177c1b7640 1 -- start start 2024-05-09T05:19:39.518 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f177c1b7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f177406f1b0 0x7f17741f60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:39.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f177c1b7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17740d16b0 0x7f17740b47a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:39.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f177c1b7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f177415ca90 0x7f1774151d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:39.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f177c1b7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f17740d0910 con 0x7f177415ca90 2024-05-09T05:19:39.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f177c1b7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f17740ccd70 con 0x7f17740d16b0 2024-05-09T05:19:39.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f177c1b7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f17740cec80 con 0x7f177406f1b0 2024-05-09T05:19:39.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f1779752640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17740d16b0 0x7f17740b47a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:39.519 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f1779752640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17740d16b0 0x7f17740b47a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:39016/0 (socket says 172.21.15.1:39016) 2024-05-09T05:19:39.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f1779752640 1 -- 172.21.15.1:0/2032146306 learned_addr learned my addr 172.21.15.1:0/2032146306 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:39.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.520+0000 7f177a754640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f177415ca90 msgr2=0x7f1774151d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:39.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.521+0000 7f177a754640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f177415ca90 0x7f1774151d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:39.520 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.521+0000 7f1779f53640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f177406f1b0 0x7f17741f60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:39.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.522+0000 7f1779752640 1 -- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f177406f1b0 msgr2=0x7f17741f60f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:39.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.522+0000 7f1779752640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f177406f1b0 0x7f17741f60f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:39.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.522+0000 7f1779752640 1 -- 172.21.15.1:0/2032146306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f177415ca90 msgr2=0x7f1774151d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:39.522 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.522+0000 7f1779752640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f177415ca90 0x7f1774151d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:39.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.522+0000 7f1779752640 1 -- 172.21.15.1:0/2032146306 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1774159ed0 con 0x7f17740d16b0 2024-05-09T05:19:39.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.523+0000 7f1779752640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17740d16b0 0x7f17740b47a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f176400b450 tx=0x7f176400b910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:39.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.523+0000 7f176a7fc640 1 -- 172.21.15.1:0/2032146306 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764003cf0 con 0x7f17740d16b0 2024-05-09T05:19:39.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.523+0000 7f176a7fc640 1 -- 172.21.15.1:0/2032146306 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1764015070 con 0x7f17740d16b0 2024-05-09T05:19:39.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.524+0000 7f176a7fc640 1 -- 172.21.15.1:0/2032146306 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1764005290 con 0x7f17740d16b0 2024-05-09T05:19:39.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.524+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f177406e220 con 0x7f17740d16b0 2024-05-09T05:19:39.524 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.524+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17741f58f0 con 0x7f17740d16b0 2024-05-09T05:19:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.526+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1774154750 con 0x7f17740d16b0 2024-05-09T05:19:39.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.527+0000 7f176a7fc640 1 -- 172.21.15.1:0/2032146306 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f176400f070 con 0x7f17740d16b0 2024-05-09T05:19:39.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.527+0000 7f176a7fc640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1758076300 0x7f1758078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:39.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.527+0000 7f176a7fc640 1 -- 172.21.15.1:0/2032146306 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1764096da0 con 0x7f17740d16b0 2024-05-09T05:19:39.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.528+0000 7f1779f53640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1758076300 0x7f1758078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:39.527 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.529+0000 7f1779f53640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1758076300 0x7f1758078810 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1770034000 tx=0x7f177002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:39.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:39.535+0000 7f176a7fc640 1 -- 172.21.15.1:0/2032146306 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1774154750 con 0x7f17740d16b0 2024-05-09T05:19:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.037+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1774050870 con 0x7f17740d16b0 2024-05-09T05:19:40.034 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.037+0000 7f176a7fc640 1 -- 172.21.15.1:0/2032146306 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f1774159ed0 con 0x7f17740d16b0 2024-05-09T05:19:40.034 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1758076300 msgr2=0x7f1758078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1758076300 0x7f1758078810 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f1770034000 tx=0x7f177002e000 comp rx=0 tx=0).stop 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17740d16b0 msgr2=0x7f17740b47a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17740d16b0 0x7f17740b47a0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f176400b450 tx=0x7f176400b910 comp rx=0 tx=0).stop 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 shutdown_connections 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1758076300 0x7f1758078810 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f17740d16b0 0x7f17740b47a0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f177415ca90 0x7f1774151d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 --2- 172.21.15.1:0/2032146306 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f177406f1b0 0x7f17741f60f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 >> 172.21.15.1:0/2032146306 conn(0x7f17740b3660 msgr2=0x7f177415a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 shutdown_connections 2024-05-09T05:19:40.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:40.040+0000 7f177c1b7640 1 -- 172.21.15.1:0/2032146306 wait complete. 2024-05-09T05:19:43.055 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:19:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.243+0000 7ff0b9d57640 1 Processor -- start 2024-05-09T05:19:43.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.243+0000 7ff0b9d57640 1 -- start start 2024-05-09T05:19:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.246+0000 7ff0b9d57640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0b40c87d0 0x7ff0b40c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.246+0000 7ff0b9d57640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff0b41538b0 0x7ff0b40c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.246+0000 7ff0b37fe640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7ff0b40c87d0 msgr2=0x7ff0b40c5ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.246+0000 7ff0b9d57640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b40c9810 0x7ff0b415cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.246+0000 7ff0b37fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff0b40c87d0 0x7ff0b40c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:43.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.246+0000 7ff0b9d57640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b415a380 con 0x7ff0b40c87d0 2024-05-09T05:19:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.246+0000 7ff0b9d57640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff0b41594f0 con 0x7ff0b41538b0 2024-05-09T05:19:43.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.246+0000 7ff0b9d57640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b4159770 con 0x7ff0b40c9810 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b2ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff0b41538b0 0x7ff0b40c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b2ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff0b41538b0 0x7ff0b40c92d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:42892/0 (socket says 172.21.15.1:42892) 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b2ffd640 1 -- 172.21.15.1:0/4273716624 learned_addr learned my addr 172.21.15.1:0/4273716624 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b3fff640 1 --2- 172.21.15.1:0/4273716624 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b40c9810 0x7ff0b415cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b2ffd640 1 -- 172.21.15.1:0/4273716624 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b40c9810 msgr2=0x7ff0b415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b2ffd640 1 --2- 172.21.15.1:0/4273716624 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b40c9810 0x7ff0b415cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b2ffd640 1 -- 172.21.15.1:0/4273716624 >> v2:172.21.15.1:3300/0 conn(0x7ff0b40c87d0 msgr2=0x7ff0b40c5ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b2ffd640 1 --2- 172.21.15.1:0/4273716624 >> v2:172.21.15.1:3300/0 conn(0x7ff0b40c87d0 0x7ff0b40c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.247+0000 7ff0b2ffd640 1 -- 172.21.15.1:0/4273716624 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b4159a50 con 0x7ff0b41538b0 2024-05-09T05:19:43.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.249+0000 7ff0b2ffd640 1 --2- 172.21.15.1:0/4273716624 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b41538b0 0x7ff0b40c92d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff0b415dbe0 tx=0x7ff0a802c420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8a58e91f7d2a662 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.249+0000 7ff0b27fc640 1 -- 172.21.15.1:0/4273716624 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8038070 con 0x7ff0b41538b0 2024-05-09T05:19:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.249+0000 7ff0b27fc640 1 -- 172.21.15.1:0/4273716624 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0a8031710 con 0x7ff0b41538b0 2024-05-09T05:19:43.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.249+0000 7ff0b27fc640 1 -- 172.21.15.1:0/4273716624 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0a8031ae0 con 0x7ff0b41538b0 2024-05-09T05:19:43.248 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.250+0000 7ff0b9d57640 1 -- 172.21.15.1:0/4273716624 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b41538b0 msgr2=0x7ff0b40c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.250+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/4273716624 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b41538b0 0x7ff0b40c92d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ff0b415dbe0 tx=0x7ff0a802c420 comp rx=0 tx=0).stop 2024-05-09T05:19:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.250+0000 7ff0b9d57640 1 -- 172.21.15.1:0/4273716624 shutdown_connections 2024-05-09T05:19:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.250+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/4273716624 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b40c9810 0x7ff0b415cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.250+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/4273716624 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b41538b0 0x7ff0b40c92d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.250+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/4273716624 >> v2:172.21.15.1:3300/0 conn(0x7ff0b40c87d0 0x7ff0b40c5ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.251+0000 7ff0b9d57640 1 -- 172.21.15.1:0/4273716624 >> 172.21.15.1:0/4273716624 conn(0x7ff0b40745a0 msgr2=0x7ff0b4071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.251+0000 7ff0b9d57640 1 -- 172.21.15.1:0/4273716624 shutdown_connections 2024-05-09T05:19:43.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.251+0000 7ff0b9d57640 1 -- 172.21.15.1:0/4273716624 wait complete. 2024-05-09T05:19:43.251 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.254+0000 7ff0b9d57640 1 Processor -- start 2024-05-09T05:19:43.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.255+0000 7ff0b9d57640 1 -- start start 2024-05-09T05:19:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.257+0000 7ff0b9d57640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b40c87d0 0x7ff0b41f2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.257+0000 7ff0b9d57640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b40c9810 0x7ff0b406cf10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.257+0000 7ff0b9d57640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b41538b0 0x7ff0b414c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.257+0000 7ff0b9d57640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b40c61e0 con 0x7ff0b40c9810 2024-05-09T05:19:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.257+0000 7ff0b9d57640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff0b415a010 con 0x7ff0b40c87d0 2024-05-09T05:19:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.257+0000 7ff0b9d57640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff0b40c70c0 con 0x7ff0b41538b0 2024-05-09T05:19:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b2ffd640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b40c9810 msgr2=0x7ff0b406cf10 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:43.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b37fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b40c87d0 0x7ff0b41f2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b41538b0 0x7ff0b414c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b40c9810 0x7ff0b406cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b37fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b40c87d0 0x7ff0b41f2fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:42900/0 (socket says 172.21.15.1:42900) 2024-05-09T05:19:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b41538b0 0x7ff0b414c9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:33140/0 (socket says 172.21.15.1:33140) 2024-05-09T05:19:43.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b37fe640 1 -- 172.21.15.1:0/721836117 learned_addr learned my addr 172.21.15.1:0/721836117 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b3fff640 1 -- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b40c87d0 msgr2=0x7ff0b41f2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b3fff640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b40c87d0 0x7ff0b41f2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b3fff640 1 -- 172.21.15.1:0/721836117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b40c9810 msgr2=0x7ff0b406cf10 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b3fff640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b40c9810 0x7ff0b406cf10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b3fff640 1 -- 172.21.15.1:0/721836117 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff0b4152c00 con 0x7ff0b41538b0 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b37fe640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b40c87d0 0x7ff0b41f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.258+0000 7ff0b3fff640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b41538b0 0x7ff0b414c9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac007cf0 tx=0x7ff0ac00e4b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.259+0000 7ff09bfff640 1 -- 172.21.15.1:0/721836117 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac00edd0 con 0x7ff0b41538b0 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.259+0000 7ff09bfff640 1 -- 172.21.15.1:0/721836117 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff0ac003990 con 0x7ff0b41538b0 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.259+0000 7ff09bfff640 1 -- 172.21.15.1:0/721836117 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff0ac003d80 con 0x7ff0b41538b0 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.259+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff0b4152ec0 con 0x7ff0b41538b0 2024-05-09T05:19:43.259 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.259+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff0b415c280 con 0x7ff0b41538b0 2024-05-09T05:19:43.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.262+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff0b4050870 con 0x7ff0b41538b0 2024-05-09T05:19:43.260 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.263+0000 7ff09bfff640 1 -- 172.21.15.1:0/721836117 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff0ac01b020 con 0x7ff0b41538b0 2024-05-09T05:19:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.263+0000 7ff09bfff640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff0a0076280 0x7ff0a0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.264+0000 7ff09bfff640 1 -- 172.21.15.1:0/721836117 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff0ac0977d0 con 0x7ff0b41538b0 2024-05-09T05:19:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.264+0000 7ff0b37fe640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff0a0076280 0x7ff0a0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:43.261 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.265+0000 7ff0b37fe640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff0a0076280 0x7ff0a0078790 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009810 tx=0x7ff0a4009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:43.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.270+0000 7ff09bfff640 1 -- 172.21.15.1:0/721836117 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff0ac060f80 con 0x7ff0b41538b0 2024-05-09T05:19:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.768+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff0b40c55a0 con 0x7ff0b41538b0 2024-05-09T05:19:43.765 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.768+0000 7ff09bfff640 1 -- 172.21.15.1:0/721836117 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7ff0ac060880 con 0x7ff0b41538b0 2024-05-09T05:19:43.765 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":100,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:19:43.768 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff0a0076280 msgr2=0x7ff0a0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff0a0076280 0x7ff0a0078790 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7ff0a4009810 tx=0x7ff0a4009190 comp rx=0 tx=0).stop 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b41538b0 msgr2=0x7ff0b414c9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b41538b0 0x7ff0b414c9a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ff0ac007cf0 tx=0x7ff0ac00e4b0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 shutdown_connections 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff0a0076280 0x7ff0a0078790 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff0b41538b0 0x7ff0b414c9a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff0b40c9810 0x7ff0b406cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 --2- 172.21.15.1:0/721836117 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff0b40c87d0 0x7ff0b41f2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:43.769 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 >> 172.21.15.1:0/721836117 conn(0x7ff0b40745a0 msgr2=0x7ff0b40ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:43.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 shutdown_connections 2024-05-09T05:19:43.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:43.772+0000 7ff0b9d57640 1 -- 172.21.15.1:0/721836117 wait complete. 2024-05-09T05:19:46.789 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:19:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.970+0000 7ff3b4a69640 1 Processor -- start 2024-05-09T05:19:46.967 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.970+0000 7ff3b4a69640 1 -- start start 2024-05-09T05:19:46.969 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.972+0000 7ff3b4a69640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.972+0000 7ff3b4a69640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.972+0000 7ff3b4a69640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3b0074d10 con 0x7ff3b0158890 2024-05-09T05:19:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.972+0000 7ff3b4a69640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b00c9710 con 0x7ff3b00c4570 2024-05-09T05:19:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.972+0000 7ff3b4a69640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b00c99f0 con 0x7ff3b00c3650 2024-05-09T05:19:46.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.973+0000 7ff3ae59c640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7ff3b0158890 legacy=0x7ff3b0152aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:19:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.973+0000 7ff3add9b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.973+0000 7ff3add9b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:42906/0 (socket says 172.21.15.1:42906) 2024-05-09T05:19:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.973+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 learned_addr learned my addr 172.21.15.1:0/1458829906 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.973+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:19:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.973+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.973+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.973+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:46.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.974+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.975+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:19:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.975+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.975+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:46.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:46.975+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.174+0000 7ff3ae59c640 1 -- 172.21.15.1:0/1458829906 >> v1:172.21.15.1:6789/0 conn(0x7ff3b0158890 legacy=0x7ff3b0152aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:19:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.174+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:47.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.174+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:19:47.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.174+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:47.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.174+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:47.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.174+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:19:47.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.176+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:47.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.176+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:19:47.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.176+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:47.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.176+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:47.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.176+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:19:47.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.575+0000 7ff3ae59c640 1 -- 172.21.15.1:0/1458829906 >> v1:172.21.15.1:6789/0 conn(0x7ff3b0158890 legacy=0x7ff3b0152aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:19:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.575+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.575+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:19:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.575+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.575+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:47.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.575+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:19:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.577+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.577+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:19:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.577+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:47.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.577+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:47.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:47.577+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:19:48.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.376+0000 7ff3ae59c640 1 -- 172.21.15.1:0/1458829906 >> v1:172.21.15.1:6789/0 conn(0x7ff3b0158890 legacy=0x7ff3b0152aa0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:19:48.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.376+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:48.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.376+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:19:48.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.376+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:48.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.376+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:48.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.376+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:19:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.378+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:48.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.379+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:19:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.379+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:19:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.379+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:19:48.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:48.379+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:19:49.970 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.972+0000 7ff39ffff640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff39ffff640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3b00c3650 0x7ff3b00c3a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff39ffff640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 msgr2=0x7ff3b00cdd20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff39ffff640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.196:3300/0 conn(0x7ff3b00c4570 0x7ff3b00cdd20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff39ffff640 1 -- 172.21.15.1:0/1458829906 >> v1:172.21.15.1:6789/0 conn(0x7ff3b0158890 legacy=0x7ff3b0152aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff39ffff640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.1:3300/0 conn(0x7ff390007380 0x7ff390009890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.1:3300/0 conn(0x7ff390007380 msgr2=0x7ff390009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.1:3300/0 conn(0x7ff390007380 0x7ff390009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff39ffff640 1 -- 172.21.15.1:0/1458829906 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff3b0072ec0 con 0x7ff390003680 2024-05-09T05:19:49.971 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:49.973+0000 7ff39ffff640 1 -- 172.21.15.1:0/1458829906 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff3b00c9710 con 0x7ff390007380 2024-05-09T05:19:50.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.174+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.1:3300/0 conn(0x7ff390007380 msgr2=0x7ff390009890 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:50.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.174+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.1:3300/0 conn(0x7ff390007380 0x7ff390009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:19:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.177+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2383314652 0 0) 0x7ff3b0072ec0 con 0x7ff390003680 2024-05-09T05:19:50.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.177+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff398003520 con 0x7ff390003680 2024-05-09T05:19:50.196 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2956430221 0 0) 0x7ff398003520 con 0x7ff390003680 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 >> v2:172.21.15.1:3300/0 conn(0x7ff390007380 msgr2=0x7ff390009890 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 --2- 172.21.15.1:0/1458829906 >> v2:172.21.15.1:3300/0 conn(0x7ff390007380 0x7ff390009890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b0074d10 con 0x7ff390003680 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7ff3a8004600 con 0x7ff390003680 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 >> v1:172.21.15.196:6789/0 conn(0x7ff390003680 legacy=0x7ff390005b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3aed9d640 1 -- 172.21.15.1:0/1458829906 reap_dead start 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff398003e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff398004800 0x7ff398004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3980099b0 0x7ff39800bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b00c9710 con 0x7ff398004800 2024-05-09T05:19:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff39800e5d0 con 0x7ff3980099b0 2024-05-09T05:19:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff39800e7d0 con 0x7ff3980039b0 2024-05-09T05:19:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3ae59c640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff398003e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3add9b640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff398004800 msgr2=0x7ff398004c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.186+0000 7ff3add9b640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff398004800 0x7ff398004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.187+0000 7ff3aed9d640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3980099b0 0x7ff39800bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.187+0000 7ff3ae59c640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3980099b0 msgr2=0x7ff39800bdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.187+0000 7ff3ae59c640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3980099b0 0x7ff39800bdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.187+0000 7ff3ae59c640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff398004800 msgr2=0x7ff398004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.187+0000 7ff3ae59c640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff398004800 0x7ff398004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.187+0000 7ff3ae59c640 1 -- 172.21.15.1:0/1458829906 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff398003520 con 0x7ff3980039b0 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.188+0000 7ff3ae59c640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff398003e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8003d00 tx=0x7ff398005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54e0686f5e509c64 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.188+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a800c220 con 0x7ff3980039b0 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.189+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a8032020 con 0x7ff3980039b0 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.189+0000 7ff3ad59a640 1 -- 172.21.15.1:0/1458829906 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a80063d0 con 0x7ff3980039b0 2024-05-09T05:19:50.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 -- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 msgr2=0x7ff398003e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff398003e00 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff3a8003d00 tx=0x7ff398005830 comp rx=0 tx=0).stop 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 -- 172.21.15.1:0/1458829906 shutdown_connections 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff3980099b0 0x7ff39800bdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff398004800 0x7ff398004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/1458829906 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff398003e00 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 -- 172.21.15.1:0/1458829906 >> 172.21.15.1:0/1458829906 conn(0x7ff3b0070dd0 msgr2=0x7ff3b00711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 -- 172.21.15.1:0/1458829906 shutdown_connections 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.190+0000 7ff3b4a69640 1 -- 172.21.15.1:0/1458829906 wait complete. 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.193+0000 7ff3b4a69640 1 Processor -- start 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.194+0000 7ff3b4a69640 1 -- start start 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3b4a69640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff3b0157300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3b4a69640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff398004800 0x7ff3b01f16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3b4a69640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3980099b0 0x7ff3b00c5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3aed9d640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3980099b0 msgr2=0x7ff3b00c5150 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3aed9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3980099b0 0x7ff3b00c5150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3ae59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff3b0157300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3ae59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff3b0157300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:43996/0 (socket says 172.21.15.1:43996) 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3add9b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff398004800 0x7ff3b01f16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3add9b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff398004800 0x7ff3b01f16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33832/0 (socket says 172.21.15.1:33832) 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3b4a69640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b0073480 con 0x7ff3980099b0 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.196+0000 7ff3ae59c640 1 -- 172.21.15.1:0/3601912237 learned_addr learned my addr 172.21.15.1:0/3601912237 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.197+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff3b00cd800 con 0x7ff398004800 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.197+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff3b0073600 con 0x7ff3980039b0 2024-05-09T05:19:50.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.197+0000 7ff3ae59c640 1 -- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff398004800 msgr2=0x7ff3b01f16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.197+0000 7ff3ae59c640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff398004800 0x7ff3b01f16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.197+0000 7ff3ae59c640 1 -- 172.21.15.1:0/3601912237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3980099b0 msgr2=0x7ff3b00c5150 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.197+0000 7ff3ae59c640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3980099b0 0x7ff3b00c5150 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.197+0000 7ff3ae59c640 1 -- 172.21.15.1:0/3601912237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3b0157840 con 0x7ff3980039b0 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.197+0000 7ff3add9b640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff398004800 0x7ff3b01f16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.198+0000 7ff3ae59c640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff3b0157300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800c1b0 tx=0x7ff3a80067c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.198+0000 7ff39effd640 1 -- 172.21.15.1:0/3601912237 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a8031040 con 0x7ff3980039b0 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.198+0000 7ff39effd640 1 -- 172.21.15.1:0/3601912237 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff3a802cd10 con 0x7ff3980039b0 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.198+0000 7ff39effd640 1 -- 172.21.15.1:0/3601912237 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3a800c220 con 0x7ff3980039b0 2024-05-09T05:19:50.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.198+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3b01f2370 con 0x7ff3980039b0 2024-05-09T05:19:50.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.198+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3b0153580 con 0x7ff3980039b0 2024-05-09T05:19:50.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.199+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3b00696d0 con 0x7ff3980039b0 2024-05-09T05:19:50.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.201+0000 7ff39effd640 1 -- 172.21.15.1:0/3601912237 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff3a8006a70 con 0x7ff3980039b0 2024-05-09T05:19:50.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.202+0000 7ff39effd640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff390076940 0x7ff390078e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.202+0000 7ff39effd640 1 -- 172.21.15.1:0/3601912237 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff3a80b8030 con 0x7ff3980039b0 2024-05-09T05:19:50.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.202+0000 7ff3add9b640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff390076940 0x7ff390078e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.203+0000 7ff3add9b640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff390076940 0x7ff390078e50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff3b014cc90 tx=0x7ff3a4016040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:50.205 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.207+0000 7ff39effd640 1 -- 172.21.15.1:0/3601912237 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff3a80818f0 con 0x7ff3980039b0 2024-05-09T05:19:50.713 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.716+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff3b00ca040 con 0x7ff3980039b0 2024-05-09T05:19:50.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.717+0000 7ff39effd640 1 -- 172.21.15.1:0/3601912237 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7ff3a80811f0 con 0x7ff3980039b0 2024-05-09T05:19:50.715 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:19:50.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff390076940 msgr2=0x7ff390078e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff390076940 0x7ff390078e50 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7ff3b014cc90 tx=0x7ff3a4016040 comp rx=0 tx=0).stop 2024-05-09T05:19:50.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 msgr2=0x7ff3b0157300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff3b0157300 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff3a800c1b0 tx=0x7ff3a80067c0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 shutdown_connections 2024-05-09T05:19:50.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff390076940 0x7ff390078e50 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff398004800 0x7ff3b01f16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff3980099b0 0x7ff3b00c5150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 --2- 172.21.15.1:0/3601912237 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff3980039b0 0x7ff3b0157300 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 >> 172.21.15.1:0/3601912237 conn(0x7ff3b0070dd0 msgr2=0x7ff3b00c3a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:50.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 shutdown_connections 2024-05-09T05:19:50.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.721+0000 7ff3b4a69640 1 -- 172.21.15.1:0/3601912237 wait complete. 2024-05-09T05:19:50.734 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:19:50.734 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:19:50.734 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:19:50.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.914+0000 7f76a1a61640 1 Processor -- start 2024-05-09T05:19:50.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.914+0000 7f76a1a61640 1 -- start start 2024-05-09T05:19:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.916+0000 7f76a1a61640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f769c0c46e0 0x7f769c159650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.916+0000 7f769affd640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f769c0c89a0 legacy=0x7f769c0ca930 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:19:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f76a1a61640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f769c0c3be0 con 0x7f769c0c89a0 2024-05-09T05:19:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f76a1a61640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f769c06e890 con 0x7f769c0c4af0 2024-05-09T05:19:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f76a1a61640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f769c0cb040 con 0x7f769c0c46e0 2024-05-09T05:19:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f769a7fc640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f769c0c46e0 0x7f769c159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f769b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f769c0c4af0 0x7f769c153490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:52868/0 (socket says 172.21.15.1:52868) 2024-05-09T05:19:50.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f769b7fe640 1 -- 172.21.15.1:0/2209320446 learned_addr learned my addr 172.21.15.1:0/2209320446 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:19:50.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f7699ffb640 1 -- 172.21.15.1:0/2209320446 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2442864652 0 0) 0x7f769c06e890 con 0x7f769c0c4af0 2024-05-09T05:19:50.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f769a7fc640 1 -- 172.21.15.1:0/2209320446 >> v1:172.21.15.196:6789/0 conn(0x7f769c0c4af0 legacy=0x7f769c153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f769a7fc640 1 -- 172.21.15.1:0/2209320446 >> v1:172.21.15.1:6789/0 conn(0x7f769c0c89a0 legacy=0x7f769c0ca930 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:50.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.917+0000 7f769a7fc640 1 -- 172.21.15.1:0/2209320446 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769c0c3e60 con 0x7f769c0c46e0 2024-05-09T05:19:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.918+0000 7f769a7fc640 1 --2- 172.21.15.1:0/2209320446 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c46e0 0x7f769c159650 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f76900097a0 tx=0x7f769002c310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc441279e125aaa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.919+0000 7f7699ffb640 1 -- 172.21.15.1:0/2209320446 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7690039070 con 0x7f769c0c46e0 2024-05-09T05:19:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.919+0000 7f7699ffb640 1 -- 172.21.15.1:0/2209320446 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7690034640 con 0x7f769c0c46e0 2024-05-09T05:19:50.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.919+0000 7f7699ffb640 1 -- 172.21.15.1:0/2209320446 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769002ce60 con 0x7f769c0c46e0 2024-05-09T05:19:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.920+0000 7f76a1a61640 1 -- 172.21.15.1:0/2209320446 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c46e0 msgr2=0x7f769c159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.920+0000 7f76a1a61640 1 --2- 172.21.15.1:0/2209320446 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c46e0 0x7f769c159650 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f76900097a0 tx=0x7f769002c310 comp rx=0 tx=0).stop 2024-05-09T05:19:50.917 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.920+0000 7f76a1a61640 1 -- 172.21.15.1:0/2209320446 shutdown_connections 2024-05-09T05:19:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.920+0000 7f76a1a61640 1 --2- 172.21.15.1:0/2209320446 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c46e0 0x7f769c159650 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.920+0000 7f76a1a61640 1 -- 172.21.15.1:0/2209320446 >> 172.21.15.1:0/2209320446 conn(0x7f769c0747b0 msgr2=0x7f769c0ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.921+0000 7f76a1a61640 1 -- 172.21.15.1:0/2209320446 shutdown_connections 2024-05-09T05:19:50.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.921+0000 7f76a1a61640 1 -- 172.21.15.1:0/2209320446 wait complete. 2024-05-09T05:19:50.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.923+0000 7f76a1a61640 1 Processor -- start 2024-05-09T05:19:50.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.924+0000 7f76a1a61640 1 -- start start 2024-05-09T05:19:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.926+0000 7f76a1a61640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f769c0c46e0 0x7f769c14e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.926+0000 7f76a1a61640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c4af0 0x7f769c157760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.926+0000 7f76a1a61640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f769c0c89a0 0x7f769c14c710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.926+0000 7f769b7fe640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f769c0c89a0 msgr2=0x7f769c14c710 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.926+0000 7f769b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f769c0c89a0 0x7f769c14c710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.927+0000 7f76a1a61640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f769c0c94e0 con 0x7f769c0c89a0 2024-05-09T05:19:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.927+0000 7f76a1a61640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f769c0c9290 con 0x7f769c0c4af0 2024-05-09T05:19:50.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.927+0000 7f769affd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f769c0c46e0 0x7f769c14e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.927+0000 7f769a7fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c4af0 0x7f769c157760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.927+0000 7f769a7fc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c4af0 0x7f769c157760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33848/0 (socket says 172.21.15.1:33848) 2024-05-09T05:19:50.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.927+0000 7f769a7fc640 1 -- 172.21.15.1:0/258226251 learned_addr learned my addr 172.21.15.1:0/258226251 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:50.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.927+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f769c0c97c0 con 0x7f769c0c46e0 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.928+0000 7f769a7fc640 1 -- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f769c0c46e0 msgr2=0x7f769c14e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.928+0000 7f769a7fc640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f769c0c46e0 0x7f769c14e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.928+0000 7f769a7fc640 1 -- 172.21.15.1:0/258226251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f769c0c89a0 msgr2=0x7f769c14c710 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.928+0000 7f769a7fc640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f769c0c89a0 0x7f769c14c710 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.928+0000 7f769a7fc640 1 -- 172.21.15.1:0/258226251 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f769c158360 con 0x7f769c0c4af0 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.928+0000 7f769a7fc640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c4af0 0x7f769c157760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f76900098c0 tx=0x7f76900046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.928+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7690039040 con 0x7f769c0c4af0 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.929+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7690044070 con 0x7f769c0c4af0 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.929+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769002ce60 con 0x7f769c0c4af0 2024-05-09T05:19:50.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.929+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f769c14dea0 con 0x7f769c0c4af0 2024-05-09T05:19:50.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.929+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f769c156000 con 0x7f769c0c4af0 2024-05-09T05:19:50.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.932+0000 7f76a1a61640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c14cc50 0x7f769c159450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.932+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f769c14eba0 con 0x7f769c14cc50 2024-05-09T05:19:50.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.932+0000 7f769affd640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c14cc50 0x7f769c159450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.933+0000 7f769affd640 1 -- 172.21.15.1:0/258226251 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f769c050870 con 0x7f769c14cc50 2024-05-09T05:19:50.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.933+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7690032be0 con 0x7f769c0c4af0 2024-05-09T05:19:50.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.934+0000 7f7683fff640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7688076280 0x7f7688078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.934+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f76900b83c0 con 0x7f769c0c4af0 2024-05-09T05:19:50.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.934+0000 7f769affd640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c14cc50 0x7f769c159450 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f768c009610 tx=0x7f768c009ad0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:50.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.934+0000 7f769b7fe640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7688076280 0x7f7688078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.934+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c016070 con 0x7f769c14cc50 2024-05-09T05:19:50.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.935+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f768c005a40 con 0x7f769c14cc50 2024-05-09T05:19:50.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.935+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c14cc50 msgr2=0x7f769c159450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.935+0000 7f7683fff640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c14cc50 0x7f769c159450 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f768c009610 tx=0x7f768c009ad0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.935+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c14cc50 msgr2=0x7f769c159450 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:50.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.935+0000 7f7683fff640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c14cc50 0x7f769c159450 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.935+0000 7f769b7fe640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7688076280 0x7f7688078790 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f769c0c61e0 tx=0x7f7694007c50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.945+0000 7f76a1a61640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c159990 0x7f769c159da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:50.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.945+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f769c14be10 con 0x7f769c159990 2024-05-09T05:19:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.945+0000 7f769affd640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c159990 0x7f769c159da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.946+0000 7f769affd640 1 -- 172.21.15.1:0/258226251 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f769c0ca360 con 0x7f769c159990 2024-05-09T05:19:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.946+0000 7f769affd640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c159990 0x7f769c159da0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f768c009610 tx=0x7f768c0059d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:50.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.947+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f768c016050 con 0x7f769c159990 2024-05-09T05:19:50.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.947+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5053 (secure 0 0 0) 0x7f768c00e3f0 con 0x7f769c159990 2024-05-09T05:19:50.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.947+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c159990 msgr2=0x7f769c159da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.947+0000 7f7683fff640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c159990 0x7f769c159da0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f768c009610 tx=0x7f768c0059d0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.947+0000 7f7683fff640 1 -- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c159990 msgr2=0x7f769c159da0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:50.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.947+0000 7f7683fff640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c159990 0x7f769c159da0 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f768c009610 tx=0x7f768c0059d0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 102, 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 141351, 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:19:50.945 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:50.946 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:19:50.946 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:19:50.946 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:50.946 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:50.946 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:50.946 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:50.946 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:50.946 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:50.947 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:50.948 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:50.949 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:50.950 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:50.951 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:50.952 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:50.953 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:19:50.954 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.955 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:50.956 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:50.957 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:50.958 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:50.958 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:50.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:50.958 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:19:50.958 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7688076280 msgr2=0x7f7688078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f76a1a61640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7688076280 0x7f7688078790 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f769c0c61e0 tx=0x7f7694007c50 comp rx=0 tx=0).stop 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c4af0 msgr2=0x7f769c157760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f76a1a61640 1 --2- 172.21.15.1:0/258226251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f769c0c4af0 0x7f769c157760 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f76900098c0 tx=0x7f76900046c0 comp rx=0 tx=0).stop 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f769b7fe640 1 -- 172.21.15.1:0/258226251 reap_dead start 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f769b7fe640 1 -- 172.21.15.1:0/258226251 reap_dead start 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 shutdown_connections 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 >> 172.21.15.1:0/258226251 conn(0x7f769c0747b0 msgr2=0x7f769c151b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 shutdown_connections 2024-05-09T05:19:50.959 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:50.948+0000 7f76a1a61640 1 -- 172.21.15.1:0/258226251 wait complete. 2024-05-09T05:19:50.963 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:19:50.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:19:50.963 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:19:51.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.149+0000 7f2367e8b640 1 Processor -- start 2024-05-09T05:19:51.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.149+0000 7f2367e8b640 1 -- start start 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.151+0000 7f2367e8b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c9d90 0x7f23601544a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.151+0000 7f2365426640 1 -- >> v1:172.21.15.1:6789/0 conn(0x7f23600c8620 legacy=0x7f23600c7010 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.1:6789/0 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2365c27640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c9d90 0x7f23601544a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2365c27640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c9d90 0x7f23601544a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:44014/0 (socket says 172.21.15.1:44014) 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2365c27640 1 -- 172.21.15.1:0/3605207528 learned_addr learned my addr 172.21.15.1:0/3605207528 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2367e8b640 1 -- 172.21.15.1:0/3605207528 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f23600c8e40 con 0x7f23600c8620 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2367e8b640 1 -- 172.21.15.1:0/3605207528 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2360073840 con 0x7f23600c8a30 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2367e8b640 1 -- 172.21.15.1:0/3605207528 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23600735f0 con 0x7f23600c9d90 2024-05-09T05:19:51.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2364c25640 1 -- 172.21.15.1:0/3605207528 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2941293741 0 0) 0x7f2360073840 con 0x7f23600c8a30 2024-05-09T05:19:51.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2365c27640 1 -- 172.21.15.1:0/3605207528 >> v1:172.21.15.196:6789/0 conn(0x7f23600c8a30 legacy=0x7f236015a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:51.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2365c27640 1 -- 172.21.15.1:0/3605207528 >> v1:172.21.15.1:6789/0 conn(0x7f23600c8620 legacy=0x7f23600c7010 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:19:51.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.152+0000 7f2365c27640 1 -- 172.21.15.1:0/3605207528 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2360073b20 con 0x7f23600c9d90 2024-05-09T05:19:51.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.153+0000 7f2365c27640 1 --2- 172.21.15.1:0/3605207528 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c9d90 0x7f23601544a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f235c0098d0 tx=0x7f235c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3ff3fb345645c46 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:51.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.154+0000 7f2364c25640 1 -- 172.21.15.1:0/3605207528 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c037070 con 0x7f23600c9d90 2024-05-09T05:19:51.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.154+0000 7f2364c25640 1 -- 172.21.15.1:0/3605207528 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f235c032440 con 0x7f23600c9d90 2024-05-09T05:19:51.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.154+0000 7f2364c25640 1 -- 172.21.15.1:0/3605207528 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c032810 con 0x7f23600c9d90 2024-05-09T05:19:51.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.155+0000 7f2367e8b640 1 -- 172.21.15.1:0/3605207528 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c9d90 msgr2=0x7f23601544a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:51.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.155+0000 7f2367e8b640 1 --2- 172.21.15.1:0/3605207528 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c9d90 0x7f23601544a0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f235c0098d0 tx=0x7f235c02a270 comp rx=0 tx=0).stop 2024-05-09T05:19:51.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.156+0000 7f2367e8b640 1 -- 172.21.15.1:0/3605207528 shutdown_connections 2024-05-09T05:19:51.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.156+0000 7f2367e8b640 1 --2- 172.21.15.1:0/3605207528 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c9d90 0x7f23601544a0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:51.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.156+0000 7f2367e8b640 1 -- 172.21.15.1:0/3605207528 >> 172.21.15.1:0/3605207528 conn(0x7f2360070f40 msgr2=0x7f236006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:51.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.157+0000 7f2367e8b640 1 -- 172.21.15.1:0/3605207528 shutdown_connections 2024-05-09T05:19:51.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.157+0000 7f2367e8b640 1 -- 172.21.15.1:0/3605207528 wait complete. 2024-05-09T05:19:51.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.159+0000 7f2367e8b640 1 Processor -- start 2024-05-09T05:19:51.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.159+0000 7f2367e8b640 1 -- start start 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2367e8b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23600c8620 0x7f23601f65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2367e8b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c8a30 0x7f23601e8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2367e8b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23600c9d90 0x7f2360146750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2367e8b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f23600743c0 con 0x7f23600c9d90 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2367e8b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f23600740e0 con 0x7f23600c8620 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2367e8b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23601549e0 con 0x7f23600c8a30 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2365c27640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23600c8620 0x7f23601f65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2365c27640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23600c8620 0x7f23601f65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33862/0 (socket says 172.21.15.1:33862) 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2365c27640 1 -- 172.21.15.1:0/1041877505 learned_addr learned my addr 172.21.15.1:0/1041877505 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2365426640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c8a30 0x7f23601e8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:51.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2366428640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23600c9d90 msgr2=0x7f2360146750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.162+0000 7f2366428640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23600c9d90 0x7f2360146750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.163+0000 7f2365c27640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c8a30 msgr2=0x7f23601e8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.163+0000 7f2365c27640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c8a30 0x7f23601e8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.163+0000 7f2365c27640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23600c9d90 msgr2=0x7f2360146750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.163+0000 7f2365c27640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f23600c9d90 0x7f2360146750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.163+0000 7f2365c27640 1 -- 172.21.15.1:0/1041877505 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23600c6d80 con 0x7f23600c8620 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.163+0000 7f2365c27640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23600c8620 0x7f23601f65a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f235c0099f0 tx=0x7f235c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.164+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c041070 con 0x7f23600c8620 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.164+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f235c0309b0 con 0x7f23600c8620 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.164+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f235c037040 con 0x7f23600c8620 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.164+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23601f2730 con 0x7f23600c8620 2024-05-09T05:19:51.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.164+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23601f2cb0 con 0x7f23600c8620 2024-05-09T05:19:51.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.165+0000 7f2367e8b640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2360146c90 0x7f23601486f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:51.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.165+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23601f6ae0 con 0x7f2360146c90 2024-05-09T05:19:51.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.166+0000 7f2365426640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2360146c90 0x7f23601486f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:51.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.167+0000 7f2365426640 1 -- 172.21.15.1:0/1041877505 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2360050870 con 0x7f2360146c90 2024-05-09T05:19:51.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.167+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f235c044070 con 0x7f23600c8620 2024-05-09T05:19:51.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.167+0000 7f2365426640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2360146c90 0x7f23601486f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23600c7730 tx=0x7f2350009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:51.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.168+0000 7f23567fc640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2344076280 0x7f2344078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:51.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.168+0000 7f2366428640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2344076280 0x7f2344078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:51.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.168+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f235c0b7ff0 con 0x7f23600c8620 2024-05-09T05:19:51.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.169+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350016070 con 0x7f2360146c90 2024-05-09T05:19:51.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.169+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2350012600 con 0x7f2360146c90 2024-05-09T05:19:51.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.169+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2360146c90 msgr2=0x7f23601486f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:51.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.169+0000 7f23567fc640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2360146c90 0x7f23601486f0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f23600c7730 tx=0x7f2350009b00 comp rx=0 tx=0).stop 2024-05-09T05:19:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.169+0000 7f2366428640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2344076280 0x7f2344078790 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f23600c6880 tx=0x7f2358007ca0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.169+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2360146c90 msgr2=0x7f23601486f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:51.167 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.169+0000 7f23567fc640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2360146c90 0x7f23601486f0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.179+0000 7f2367e8b640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c4e00 0x7f23600c5210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:19:51.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.179+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f23600c9160 con 0x7f23600c4e00 2024-05-09T05:19:51.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.180+0000 7f2365426640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c4e00 0x7f23600c5210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:19:51.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.180+0000 7f2365426640 1 -- 172.21.15.1:0/1041877505 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2360159380 con 0x7f23600c4e00 2024-05-09T05:19:51.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.181+0000 7f2365426640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c4e00 0x7f23600c5210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2350018000 tx=0x7f23500059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:19:51.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.181+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2350016040 con 0x7f23600c4e00 2024-05-09T05:19:51.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.181+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5050 (secure 0 0 0) 0x7f235001d0d0 con 0x7f23600c4e00 2024-05-09T05:19:51.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.181+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c4e00 msgr2=0x7f23600c5210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:51.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.181+0000 7f23567fc640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c4e00 0x7f23600c5210 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f2350018000 tx=0x7f23500059d0 comp rx=0 tx=0).stop 2024-05-09T05:19:51.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.181+0000 7f23567fc640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c4e00 msgr2=0x7f23600c5210 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:19:51.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.181+0000 7f23567fc640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f23600c4e00 0x7f23600c5210 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:19:51.179 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:19:51.179 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:51.179 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 102, 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 33071, 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:51.180 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:51.181 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:51.182 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:51.182 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:51.182 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:51.182 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:51.182 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:51.182 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:19:51.183 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:19:51.184 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:19:51.185 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:51.186 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:51.187 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.188 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:19:51.189 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:19:51.190 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:51.191 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.192 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:19:51.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:19:51.194 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:19:51.194 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:19:51.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2344076280 msgr2=0x7f2344078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:51.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2367e8b640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2344076280 0x7f2344078790 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f23600c6880 tx=0x7f2358007ca0 comp rx=0 tx=0).stop 2024-05-09T05:19:51.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23600c8620 msgr2=0x7f23601f65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:19:51.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2367e8b640 1 --2- 172.21.15.1:0/1041877505 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f23600c8620 0x7f23601f65a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f235c0099f0 tx=0x7f235c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:19:51.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2366428640 1 -- 172.21.15.1:0/1041877505 reap_dead start 2024-05-09T05:19:51.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2366428640 1 -- 172.21.15.1:0/1041877505 reap_dead start 2024-05-09T05:19:51.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 shutdown_connections 2024-05-09T05:19:51.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 >> 172.21.15.1:0/1041877505 conn(0x7f2360070f40 msgr2=0x7f23600cca80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:19:51.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 shutdown_connections 2024-05-09T05:19:51.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:19:51.182+0000 7f2367e8b640 1 -- 172.21.15.1:0/1041877505 wait complete. 2024-05-09T05:19:51.196 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:19:51.196 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:20:05.296 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:20:05.299 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:20:11.197 INFO:tasks.mon_thrash.mon_thrasher:killing mon.a 2024-05-09T05:20:11.197 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.a 2024-05-09T05:20:11.197 INFO:tasks.ceph.mon.a:Restarting daemon 2024-05-09T05:20:11.198 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2024-05-09T05:20:11.200 INFO:tasks.ceph.mon.a:Started 2024-05-09T05:20:11.200 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:20:11.201 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:20:11.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.398+0000 7f76ce6b8640 1 Processor -- start 2024-05-09T05:20:11.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.398+0000 7f76ce6b8640 1 -- start start 2024-05-09T05:20:11.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.400+0000 7f76ce6b8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80c47c0 0x7f76c8159290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:11.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.400+0000 7f76ce6b8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f76c8152c50 0x7f76c80cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:11.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.401+0000 7f76ce6b8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76c80cdca0 0x7f76c80c9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:11.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.401+0000 7f76ce6b8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f76c80c8de0 con 0x7f76c80cdca0 2024-05-09T05:20:11.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.401+0000 7f76ce6b8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f76c80c82f0 con 0x7f76c8152c50 2024-05-09T05:20:11.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.401+0000 7f76ce6b8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f76c81597d0 con 0x7f76c80c47c0 2024-05-09T05:20:11.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.401+0000 7f76ccc55640 1 -- >> v2:172.21.15.1:3300/0 conn(0x7f76c80cdca0 msgr2=0x7f76c80c9270 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:20:11.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.401+0000 7f76ccc55640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76c80cdca0 0x7f76c80c9270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:11.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.402+0000 7f76c77fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f76c8152c50 0x7f76c80cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:11.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.402+0000 7f76c7fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80c47c0 0x7f76c8159290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:11.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.403+0000 7f76c7fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80c47c0 0x7f76c8159290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:56418/0 (socket says 172.21.15.1:56418) 2024-05-09T05:20:11.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.403+0000 7f76c7fff640 1 -- 172.21.15.1:0/3941051290 learned_addr learned my addr 172.21.15.1:0/3941051290 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:11.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.403+0000 7f76c7fff640 1 -- 172.21.15.1:0/3941051290 >> v2:172.21.15.196:3300/0 conn(0x7f76c8152c50 msgr2=0x7f76c80cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:11.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.403+0000 7f76c7fff640 1 --2- 172.21.15.1:0/3941051290 >> v2:172.21.15.196:3300/0 conn(0x7f76c8152c50 0x7f76c80cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.403+0000 7f76c7fff640 1 -- 172.21.15.1:0/3941051290 >> v2:172.21.15.1:3300/0 conn(0x7f76c80cdca0 msgr2=0x7f76c80c9270 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:11.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.403+0000 7f76c7fff640 1 --2- 172.21.15.1:0/3941051290 >> v2:172.21.15.1:3300/0 conn(0x7f76c80cdca0 0x7f76c80c9270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.403+0000 7f76c7fff640 1 -- 172.21.15.1:0/3941051290 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c80c2f70 con 0x7f76c80c47c0 2024-05-09T05:20:11.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.405+0000 7f76c7fff640 1 --2- 172.21.15.1:0/3941051290 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80c47c0 0x7f76c8159290 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f76c8159980 tx=0x7f76b802b630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e2103c0e27704011 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:11.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.405+0000 7f76c6ffd640 1 -- 172.21.15.1:0/3941051290 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b800b040 con 0x7f76c80c47c0 2024-05-09T05:20:11.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.405+0000 7f76c6ffd640 1 -- 172.21.15.1:0/3941051290 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76b802be50 con 0x7f76c80c47c0 2024-05-09T05:20:11.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.405+0000 7f76c6ffd640 1 -- 172.21.15.1:0/3941051290 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76b8030930 con 0x7f76c80c47c0 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.406+0000 7f76ce6b8640 1 -- 172.21.15.1:0/3941051290 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80c47c0 msgr2=0x7f76c8159290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.406+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/3941051290 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80c47c0 0x7f76c8159290 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f76c8159980 tx=0x7f76b802b630 comp rx=0 tx=0).stop 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.407+0000 7f76ce6b8640 1 -- 172.21.15.1:0/3941051290 shutdown_connections 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.407+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/3941051290 >> v2:172.21.15.1:3300/0 conn(0x7f76c80cdca0 0x7f76c80c9270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.407+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/3941051290 >> v2:172.21.15.196:3300/0 conn(0x7f76c8152c50 0x7f76c80cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.407+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/3941051290 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80c47c0 0x7f76c8159290 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.407+0000 7f76ce6b8640 1 -- 172.21.15.1:0/3941051290 >> 172.21.15.1:0/3941051290 conn(0x7f76c8070dd0 msgr2=0x7f76c80711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.407+0000 7f76ce6b8640 1 -- 172.21.15.1:0/3941051290 shutdown_connections 2024-05-09T05:20:11.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.407+0000 7f76ce6b8640 1 -- 172.21.15.1:0/3941051290 wait complete. 2024-05-09T05:20:11.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.410+0000 7f76ce6b8640 1 Processor -- start 2024-05-09T05:20:11.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.410+0000 7f76ce6b8640 1 -- start start 2024-05-09T05:20:11.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.414+0000 7f76ce6b8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76c80c47c0 0x7f76c81f6bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:11.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.414+0000 7f76ce6b8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80cdca0 0x7f76c80c7400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:11.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.414+0000 7f76ce6b8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76c8152c50 0x7f76c814b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:11.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.414+0000 7f76ce6b8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76c80b3ba0 con 0x7f76c8152c50 2024-05-09T05:20:11.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.414+0000 7f76ce6b8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f76c80b38c0 con 0x7f76c80c47c0 2024-05-09T05:20:11.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.414+0000 7f76ce6b8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f76c80c2b30 con 0x7f76c80cdca0 2024-05-09T05:20:11.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76ccc55640 1 -- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76c8152c50 msgr2=0x7f76c814b860 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.1:3300/0 2024-05-09T05:20:11.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76ccc55640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76c8152c50 0x7f76c814b860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:11.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c77fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80cdca0 0x7f76c80c7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:11.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c7fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76c80c47c0 0x7f76c81f6bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:11.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c77fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80cdca0 0x7f76c80c7400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:56426/0 (socket says 172.21.15.1:56426) 2024-05-09T05:20:11.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c77fe640 1 -- 172.21.15.1:0/1600261967 learned_addr learned my addr 172.21.15.1:0/1600261967 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:11.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c77fe640 1 -- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76c80c47c0 msgr2=0x7f76c81f6bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:11.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c77fe640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76c80c47c0 0x7f76c81f6bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c77fe640 1 -- 172.21.15.1:0/1600261967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76c8152c50 msgr2=0x7f76c814b860 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:20:11.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c77fe640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76c8152c50 0x7f76c814b860 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c77fe640 1 -- 172.21.15.1:0/1600261967 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76c80c5450 con 0x7f76c80cdca0 2024-05-09T05:20:11.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.415+0000 7f76c7fff640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76c80c47c0 0x7f76c81f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:20:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.416+0000 7f76c77fe640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80cdca0 0x7f76c80c7400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00e870 tx=0x7f76bc00ec70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.416+0000 7f76c4ff9640 1 -- 172.21.15.1:0/1600261967 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc017070 con 0x7f76c80cdca0 2024-05-09T05:20:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.416+0000 7f76c4ff9640 1 -- 172.21.15.1:0/1600261967 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76bc00c940 con 0x7f76c80cdca0 2024-05-09T05:20:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.416+0000 7f76c4ff9640 1 -- 172.21.15.1:0/1600261967 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76bc00cd30 con 0x7f76c80cdca0 2024-05-09T05:20:11.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.417+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76c8159e50 con 0x7f76c80cdca0 2024-05-09T05:20:11.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.417+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f76c815bf50 con 0x7f76c80cdca0 2024-05-09T05:20:11.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.419+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76c80696d0 con 0x7f76c80cdca0 2024-05-09T05:20:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.421+0000 7f76c4ff9640 1 -- 172.21.15.1:0/1600261967 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f76bc01a070 con 0x7f76c80cdca0 2024-05-09T05:20:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.421+0000 7f76c4ff9640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76b4076280 0x7f76b4078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:11.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.421+0000 7f76c4ff9640 1 -- 172.21.15.1:0/1600261967 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f76bc0975a0 con 0x7f76c80cdca0 2024-05-09T05:20:11.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.423+0000 7f76c7fff640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76b4076280 0x7f76b4078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:11.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.424+0000 7f76c7fff640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76b4076280 0x7f76b4078790 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f76b8035000 tx=0x7f76b8030000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:11.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.428+0000 7f76c4ff9640 1 -- 172.21.15.1:0/1600261967 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f76bc060780 con 0x7f76c80cdca0 2024-05-09T05:20:11.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.909+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f76c8050870 con 0x7f76c80cdca0 2024-05-09T05:20:11.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.910+0000 7f76c4ff9640 1 -- 172.21.15.1:0/1600261967 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f76bc064a80 con 0x7f76c80cdca0 2024-05-09T05:20:11.908 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:20:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.913+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76b4076280 msgr2=0x7f76b4078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:11.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.913+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76b4076280 0x7f76b4078790 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f76b8035000 tx=0x7f76b8030000 comp rx=0 tx=0).stop 2024-05-09T05:20:11.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.913+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80cdca0 msgr2=0x7f76c80c7400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:11.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.913+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80cdca0 0x7f76c80c7400 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f76bc00e870 tx=0x7f76bc00ec70 comp rx=0 tx=0).stop 2024-05-09T05:20:11.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.914+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 shutdown_connections 2024-05-09T05:20:11.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.914+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76b4076280 0x7f76b4078790 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.914+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76c8152c50 0x7f76c814b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.914+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76c80cdca0 0x7f76c80c7400 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.914+0000 7f76ce6b8640 1 --2- 172.21.15.1:0/1600261967 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76c80c47c0 0x7f76c81f6bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:11.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.914+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 >> 172.21.15.1:0/1600261967 conn(0x7f76c8070dd0 msgr2=0x7f76c8151840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:11.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.914+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 shutdown_connections 2024-05-09T05:20:11.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:11.914+0000 7f76ce6b8640 1 -- 172.21.15.1:0/1600261967 wait complete. 2024-05-09T05:20:14.931 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:20:15.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.113+0000 7fd3204e9640 1 Processor -- start 2024-05-09T05:20:15.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.114+0000 7fd3204e9640 1 -- start start 2024-05-09T05:20:15.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd3204e9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd318152d50 0x7fd3180735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:15.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd3204e9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 0x7fd318073b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:15.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd3204e9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3180c7b20 con 0x7fd3180cbef0 2024-05-09T05:20:15.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd3204e9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd3180c5590 con 0x7fd318153160 2024-05-09T05:20:15.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd3204e9640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd3180c7e00 con 0x7fd318152d50 2024-05-09T05:20:15.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd31e285640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fd3180cbef0 0x7fd3180744c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:50050/0 (socket says 172.21.15.1:50050) 2024-05-09T05:20:15.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd31e285640 1 -- 172.21.15.1:0/1463288218 learned_addr learned my addr 172.21.15.1:0/1463288218 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:15.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd31da84640 1 --2- 172.21.15.1:0/1463288218 >> v2:172.21.15.196:3300/0 conn(0x7fd318152d50 0x7fd3180735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:15.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.116+0000 7fd31ea86640 1 --2- 172.21.15.1:0/1463288218 >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 0x7fd318073b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:15.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.117+0000 7fd31ea86640 1 -- 172.21.15.1:0/1463288218 >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 msgr2=0x7fd318073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:15.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.117+0000 7fd31ea86640 1 -- 172.21.15.1:0/1463288218 >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 msgr2=0x7fd318073b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:15.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.117+0000 7fd31ea86640 1 --2- 172.21.15.1:0/1463288218 >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 0x7fd318073b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:15.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.117+0000 7fd31ea86640 1 --2- 172.21.15.1:0/1463288218 >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 0x7fd318073b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:15.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.117+0000 7fd31da84640 1 -- 172.21.15.1:0/1463288218 >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 msgr2=0x7fd318073b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:15.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.117+0000 7fd31da84640 1 --2- 172.21.15.1:0/1463288218 >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 0x7fd318073b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.117+0000 7fd31da84640 1 -- 172.21.15.1:0/1463288218 >> v1:172.21.15.196:6789/0 conn(0x7fd3180cbef0 legacy=0x7fd3180744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:15.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.117+0000 7fd31da84640 1 -- 172.21.15.1:0/1463288218 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3180c80e0 con 0x7fd318152d50 2024-05-09T05:20:15.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.118+0000 7fd31da84640 1 --2- 172.21.15.1:0/1463288218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 0x7fd3180735c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd318156890 tx=0x7fd30802c800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dc93c0935e4a7dd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:15.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.118+0000 7fd31d283640 1 -- 172.21.15.1:0/1463288218 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308031710 con 0x7fd318152d50 2024-05-09T05:20:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.118+0000 7fd31d283640 1 -- 172.21.15.1:0/1463288218 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3080318b0 con 0x7fd318152d50 2024-05-09T05:20:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.119+0000 7fd31d283640 1 -- 172.21.15.1:0/1463288218 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd308031c80 con 0x7fd318152d50 2024-05-09T05:20:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.120+0000 7fd3204e9640 1 -- 172.21.15.1:0/1463288218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 msgr2=0x7fd3180735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.120+0000 7fd3204e9640 1 --2- 172.21.15.1:0/1463288218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 0x7fd3180735c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd318156890 tx=0x7fd30802c800 comp rx=0 tx=0).stop 2024-05-09T05:20:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.121+0000 7fd3204e9640 1 -- 172.21.15.1:0/1463288218 shutdown_connections 2024-05-09T05:20:15.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.121+0000 7fd3204e9640 1 --2- 172.21.15.1:0/1463288218 >> v2:172.21.15.1:3300/0 conn(0x7fd318153160 0x7fd318073b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.121+0000 7fd3204e9640 1 --2- 172.21.15.1:0/1463288218 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 0x7fd3180735c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.121+0000 7fd3204e9640 1 -- 172.21.15.1:0/1463288218 >> 172.21.15.1:0/1463288218 conn(0x7fd3180ce110 msgr2=0x7fd3180ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:15.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.121+0000 7fd3204e9640 1 -- 172.21.15.1:0/1463288218 shutdown_connections 2024-05-09T05:20:15.121 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.121+0000 7fd3204e9640 1 -- 172.21.15.1:0/1463288218 wait complete. 2024-05-09T05:20:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.124+0000 7fd3204e9640 1 Processor -- start 2024-05-09T05:20:15.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.124+0000 7fd3204e9640 1 -- start start 2024-05-09T05:20:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd3204e9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3180cbef0 0x7fd3181defb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:15.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd3204e9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 0x7fd3180c9f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd3204e9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 0x7fd3180ca470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd3204e9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd3180cdf40 con 0x7fd318153160 2024-05-09T05:20:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd3204e9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd3180cddc0 con 0x7fd318152d50 2024-05-09T05:20:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd3204e9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd318068650 con 0x7fd3180cbef0 2024-05-09T05:20:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31ea86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 0x7fd3180ca470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:15.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31ea86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 0x7fd3180ca470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:35378/0 (socket says 172.21.15.1:35378) 2024-05-09T05:20:15.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31ea86640 1 -- 172.21.15.1:0/3491031281 learned_addr learned my addr 172.21.15.1:0/3491031281 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:15.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31e285640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3180cbef0 0x7fd3181defb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:15.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31da84640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 0x7fd3180c9f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:15.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31ea86640 1 -- 172.21.15.1:0/3491031281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 msgr2=0x7fd3180ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 17 2024-05-09T05:20:15.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31ea86640 1 -- 172.21.15.1:0/3491031281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 msgr2=0x7fd3180ca470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:20:15.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31ea86640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 0x7fd3180ca470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:15.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.126+0000 7fd31ea86640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 0x7fd3180ca470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:15.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd31e285640 1 -- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 msgr2=0x7fd3180c9f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:15.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd31e285640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 0x7fd3180c9f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd31e285640 1 -- 172.21.15.1:0/3491031281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 msgr2=0x7fd3180ca470 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:20:15.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd31e285640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 0x7fd3180ca470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd31e285640 1 -- 172.21.15.1:0/3491031281 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3181df4f0 con 0x7fd3180cbef0 2024-05-09T05:20:15.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd31e285640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3180cbef0 0x7fd3181defb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd31000c8a0 tx=0x7fd31000cd60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:15.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd30effd640 1 -- 172.21.15.1:0/3491031281 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd310018070 con 0x7fd3180cbef0 2024-05-09T05:20:15.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd30effd640 1 -- 172.21.15.1:0/3491031281 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd31000a5e0 con 0x7fd3180cbef0 2024-05-09T05:20:15.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd30effd640 1 -- 172.21.15.1:0/3491031281 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31000a9d0 con 0x7fd3180cbef0 2024-05-09T05:20:15.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.127+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3181f3020 con 0x7fd3180cbef0 2024-05-09T05:20:15.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.128+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3181f35e0 con 0x7fd3180cbef0 2024-05-09T05:20:15.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.129+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd318050870 con 0x7fd3180cbef0 2024-05-09T05:20:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.130+0000 7fd30effd640 1 -- 172.21.15.1:0/3491031281 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd31001b070 con 0x7fd3180cbef0 2024-05-09T05:20:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.130+0000 7fd30effd640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd2fc0762d0 0x7fd2fc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:15.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.131+0000 7fd30effd640 1 -- 172.21.15.1:0/3491031281 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd310096c90 con 0x7fd3180cbef0 2024-05-09T05:20:15.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.131+0000 7fd31da84640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd2fc0762d0 0x7fd2fc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:15.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.132+0000 7fd31da84640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd2fc0762d0 0x7fd2fc0787e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd308036000 tx=0x7fd308008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:15.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.139+0000 7fd30effd640 1 -- 172.21.15.1:0/3491031281 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd31005fe90 con 0x7fd3180cbef0 2024-05-09T05:20:15.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.637+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd3181495f0 con 0x7fd3180cbef0 2024-05-09T05:20:15.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.638+0000 7fd30effd640 1 -- 172.21.15.1:0/3491031281 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fd310064190 con 0x7fd3180cbef0 2024-05-09T05:20:15.635 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":102,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:20:15.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd2fc0762d0 msgr2=0x7fd2fc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:15.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd2fc0762d0 0x7fd2fc0787e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fd308036000 tx=0x7fd308008040 comp rx=0 tx=0).stop 2024-05-09T05:20:15.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3180cbef0 msgr2=0x7fd3181defb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3180cbef0 0x7fd3181defb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fd31000c8a0 tx=0x7fd31000cd60 comp rx=0 tx=0).stop 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 shutdown_connections 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd2fc0762d0 0x7fd2fc0787e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd318152d50 0x7fd3180c9f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd318153160 0x7fd3180ca470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 --2- 172.21.15.1:0/3491031281 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3180cbef0 0x7fd3181defb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.642+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 >> 172.21.15.1:0/3491031281 conn(0x7fd3180ce110 msgr2=0x7fd3181e4800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.643+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 shutdown_connections 2024-05-09T05:20:15.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:15.643+0000 7fd3204e9640 1 -- 172.21.15.1:0/3491031281 wait complete. 2024-05-09T05:20:18.659 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:20:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.835+0000 7f6ff6b84640 1 Processor -- start 2024-05-09T05:20:18.832 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.835+0000 7f6ff6b84640 1 -- start start 2024-05-09T05:20:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6ff6b84640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6ff6b84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6ff6b84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff00737c0 con 0x7f6ff0073f60 2024-05-09T05:20:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6ff6b84640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6ff0153d20 con 0x7f6ff0153910 2024-05-09T05:20:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6ff6b84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff0073520 con 0x7f6ff00c2b30 2024-05-09T05:20:18.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6ff5121640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6ff5121640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:56504/0 (socket says 172.21.15.1:56504) 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 learned_addr learned my addr 172.21.15.1:0/1935455712 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.837+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.838+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.838+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.838+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.838+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.838+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.838+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.838+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:18.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:18.838+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:19.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:19.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:20:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:19.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.039+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:20:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:19.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:20:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:19.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:19.440+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:20:20.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:20.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:20.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:20.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:20.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:20.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:20.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:20:20.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:20.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:20.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:20.241+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:20:21.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.450+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v1:172.21.15.1:6789/0 conn(0x7f6ff0073f60 legacy=0x7f6ff01594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-09T05:20:21.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.450+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v1:172.21.15.1:6789/0 conn(0x7f6ff0073f60 legacy=0x7f6ff01594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:20:21.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.450+0000 7f6ff4920640 1 --1- 172.21.15.1:0/1935455712 >> v1:172.21.15.1:6789/0 conn(0x7f6ff0073f60 0x7f6ff01594d0 :-1 s=OPENED pgs=24 cs=1 l=1).handle_message read tag failed 2024-05-09T05:20:21.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.450+0000 7f6ff4920640 1 --1- 172.21.15.1:0/1935455712 >> v1:172.21.15.1:6789/0 conn(0x7f6ff0073f60 0x7f6ff01594d0 :-1 s=OPENED pgs=24 cs=1 l=1).fault on lossy channel, failing 2024-05-09T05:20:21.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.837+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 msgr2=0x7f6ff00ca6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:22.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.837+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff00c2b30 0x7f6ff00ca6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:22.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.837+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6ff0072fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:22.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.837+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6ff0072fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:22.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.837+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> v1:172.21.15.1:6789/0 conn(0x7f6ff0073f60 legacy=0x7f6ff01594d0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:22.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.837+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:22.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.837+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6ff00c4260 con 0x7f6fd4003610 2024-05-09T05:20:22.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.837+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f6ff0073520 con 0x7f6fd40073a0 2024-05-09T05:20:22.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.839+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:22.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.839+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:22.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.839+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:22.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.839+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:22.102 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:21.839+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:22.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.040+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:22.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.040+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:22.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.040+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:22.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.040+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:22.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.040+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:20:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.441+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.442+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.442+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.442+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:22.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:22.442+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:20:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:23.243+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:23.243+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:23.243+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:23.243+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:23.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:23.243+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:20:24.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.837+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 msgr2=0x7f6fd40098b0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.837+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd40073a0 0x7f6fd40098b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.837+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> v1:172.21.15.196:6789/0 conn(0x7f6fd4003610 legacy=0x7f6fd4005ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.837+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 reap_dead start 2024-05-09T05:20:24.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.837+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd400afd0 con 0x7f6fd400a570 2024-05-09T05:20:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.838+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd400a980 con 0x7f6fd40098b0 2024-05-09T05:20:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6ff0073520 con 0x7f6ff0153910 2024-05-09T05:20:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.838+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:20:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.838+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.838+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:24.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:24.838+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.039+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.039+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:20:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.039+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.039+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:25.036 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.039+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:20:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.440+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.440+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:20:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.440+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.440+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:25.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:25.440+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:20:26.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:26.241+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:26.241+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:20:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:26.241+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:26.241+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:26.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:26.241+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:20:27.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 msgr2=0x7f6fd400a030 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:27.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.838+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.1:3300/0 conn(0x7f6ff0153910 0x7f6fd400a030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:27.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> v1:172.21.15.196:6789/0 conn(0x7f6fd40098b0 legacy=0x7f6fd4006920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:27.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> v1:172.21.15.1:6789/0 conn(0x7f6fd400a570 legacy=0x7f6fd4006210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:27.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.838+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:27.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.838+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:27.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f6ff0073520 con 0x7f6fd4005af0 2024-05-09T05:20:27.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6fd4014250 con 0x7f6fd4011870 2024-05-09T05:20:27.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:27.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:27.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:27.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:27.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:27.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:27.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:20:27.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:27.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:27.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:27.839+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:28.037 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:20:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:20:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:28.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.040+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:28.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:28.441+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:20:29.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.242+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:29.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.242+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:20:29.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.242+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:29.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.242+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:29.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.242+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:20:29.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.243+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:29.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.243+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:20:29.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.243+0000 7f6ff4920640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:29.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.243+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:29.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:29.243+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:20:30.835 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 msgr2=0x7f6fd4013c90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.838+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4011870 0x7f6fd4013c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.838+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 msgr2=0x7f6fd4011330 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.838+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> v2:172.21.15.196:3300/0 conn(0x7f6fd4005af0 0x7f6fd4011330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.838+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 reap_dead start 2024-05-09T05:20:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.838+0000 7f6fee7fc640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.839+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd400abf0 con 0x7f6fd400a030 2024-05-09T05:20:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.839+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6fd4005430 con 0x7f6fd4005020 2024-05-09T05:20:30.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.839+0000 7f6fee7fc640 1 -- 172.21.15.1:0/1935455712 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff0073520 con 0x7f6fd4006920 2024-05-09T05:20:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.839+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.840+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.840+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.840+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:30.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:30.840+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:31.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.041+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:31.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.041+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:31.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.041+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:31.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.041+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:31.038 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.041+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:20:31.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.442+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:31.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.442+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:31.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.442+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:31.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.442+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:31.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:31.442+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:20:32.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.243+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:32.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.243+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:20:32.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.243+0000 7f6feffff640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:20:32.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.243+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:32.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.243+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:20:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.705+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3710379373 0 0) 0x7f6fd4005430 con 0x7f6fd4005020 2024-05-09T05:20:32.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.705+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6fe0003520 con 0x7f6fd4005020 2024-05-09T05:20:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 517726309 0 0) 0x7f6fe0003520 con 0x7f6fd4005020 2024-05-09T05:20:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fd4004ae0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fd4004ae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 >> v1:172.21.15.1:6789/0 conn(0x7f6fd400a030 legacy=0x7f6fd4006210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:32.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fd40056a0 con 0x7f6fd4005020 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f6fe8009700 con 0x7f6fd4005020 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 >> v1:172.21.15.196:6789/0 conn(0x7f6fd4005020 legacy=0x7f6fd40145d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fe00039b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0003fe0 0x7f6fe0004800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.710+0000 7f6fef7fe640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe0004d90 0x7f6fe0005220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.711+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff0073520 con 0x7f6fe0004d90 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.711+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6fe000bcc0 con 0x7f6fe0003fe0 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.711+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6fe000bec0 con 0x7f6fd4006920 2024-05-09T05:20:32.708 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.711+0000 7f6feffff640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0003fe0 0x7f6fe0004800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.711+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe0004d90 0x7f6fe0005220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.711+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fe00039b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.712+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 msgr2=0x7f6fe00039b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.712+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6fd4006920 0x7f6fe00039b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.712+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0003fe0 msgr2=0x7f6fe0004800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:32.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.712+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0003fe0 0x7f6fe0004800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:32.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.712+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6fe0003520 con 0x7f6fe0004d90 2024-05-09T05:20:32.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.713+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe0004d90 0x7f6fe0005220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00b3330 tx=0x7f6fe0005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73a6b5c5bd369c1c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:32.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.713+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8009c20 con 0x7f6fe0004d90 2024-05-09T05:20:32.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.713+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe8003990 con 0x7f6fe0004d90 2024-05-09T05:20:32.710 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.713+0000 7f6fef7fe640 1 -- 172.21.15.1:0/1935455712 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe8003d80 con 0x7f6fe0004d90 2024-05-09T05:20:32.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.715+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1935455712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe0004d90 msgr2=0x7f6fe0005220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:32.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.715+0000 7f6ff6b84640 1 --2- 172.21.15.1:0/1935455712 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6fe0004d90 0x7f6fe0005220 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00b3330 tx=0x7f6fe0005830 comp rx=0 tx=0).stop 2024-05-09T05:20:32.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.715+0000 7f6ff5121640 1 -- 172.21.15.1:0/1935455712 reap_dead start 2024-05-09T05:20:32.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.715+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1935455712 shutdown_connections 2024-05-09T05:20:32.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.715+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1935455712 >> 172.21.15.1:0/1935455712 conn(0x7f6ff0070dd0 msgr2=0x7f6ff00711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:32.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.715+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1935455712 shutdown_connections 2024-05-09T05:20:32.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.715+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1935455712 wait complete. 2024-05-09T05:20:32.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.719+0000 7f6ff6b84640 1 Processor -- start 2024-05-09T05:20:32.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.719+0000 7f6ff6b84640 1 -- start start 2024-05-09T05:20:32.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.721+0000 7f6ff6b84640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0004d90 0x7f6ff00c6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:32.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.721+0000 7f6ff6b84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff01f2f40 0x7f6ff01563e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.722+0000 7f6ff4920640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0004d90 0x7f6ff00c6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.722+0000 7f6ff4920640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0004d90 0x7f6ff00c6d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:49424/0 (socket says 172.21.15.1:49424) 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.722+0000 7f6feffff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff01f2f40 0x7f6ff01563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.722+0000 7f6feffff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff01f2f40 0x7f6ff01563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:48164/0 (socket says 172.21.15.1:48164) 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.722+0000 7f6ff6b84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 0x7f6ff0156920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.722+0000 7f6ff5121640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 0x7f6ff0156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff5121640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 0x7f6ff0156920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47378/0 (socket says 172.21.15.1:47378) 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff4920640 1 -- 172.21.15.1:0/1504803701 learned_addr learned my addr 172.21.15.1:0/1504803701 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff0073a40 con 0x7f6ff014ee00 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6ff00c4820 con 0x7f6fe0004d90 2024-05-09T05:20:32.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6ff0072fe0 con 0x7f6ff01f2f40 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff5121640 1 -- 172.21.15.1:0/1504803701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 msgr2=0x7f6ff0156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff5121640 1 -- 172.21.15.1:0/1504803701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 msgr2=0x7f6ff0156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 0x7f6ff0156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff5121640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 0x7f6ff0156920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff4920640 1 -- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff01f2f40 msgr2=0x7f6ff01563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff01f2f40 0x7f6ff01563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff4920640 1 -- 172.21.15.1:0/1504803701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 msgr2=0x7f6ff0156920 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 0x7f6ff0156920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.723+0000 7f6ff4920640 1 -- 172.21.15.1:0/1504803701 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ff0150a90 con 0x7f6fe0004d90 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.724+0000 7f6feffff640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff01f2f40 0x7f6ff01563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:20:32.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.724+0000 7f6ff4920640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0004d90 0x7f6ff00c6d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4001150 tx=0x7f6fe4001810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:32.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.727+0000 7f6fed7fa640 1 -- 172.21.15.1:0/1504803701 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe4016630 con 0x7f6fe0004d90 2024-05-09T05:20:32.724 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.727+0000 7f6fed7fa640 1 -- 172.21.15.1:0/1504803701 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6fe40167d0 con 0x7f6fe0004d90 2024-05-09T05:20:32.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.727+0000 7f6fed7fa640 1 -- 172.21.15.1:0/1504803701 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6fe40169a0 con 0x7f6fe0004d90 2024-05-09T05:20:32.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.728+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ff01f2460 con 0x7f6fe0004d90 2024-05-09T05:20:32.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.728+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ff01f2a20 con 0x7f6fe0004d90 2024-05-09T05:20:32.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.730+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ff00696d0 con 0x7f6fe0004d90 2024-05-09T05:20:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.731+0000 7f6fed7fa640 1 -- 172.21.15.1:0/1504803701 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6fe4021030 con 0x7f6fe0004d90 2024-05-09T05:20:32.728 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.732+0000 7f6fed7fa640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6fd4078ed0 0x7f6fd407b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.732+0000 7f6fed7fa640 1 -- 172.21.15.1:0/1504803701 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6fe4097db0 con 0x7f6fe0004d90 2024-05-09T05:20:32.729 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.732+0000 7f6feffff640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6fd4078ed0 0x7f6fd407b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:32.730 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.734+0000 7f6feffff640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6fd4078ed0 0x7f6fd407b3e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00c5130 tx=0x7f6fdc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:32.736 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:32.739+0000 7f6fed7fa640 1 -- 172.21.15.1:0/1504803701 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6fe409c070 con 0x7f6fe0004d90 2024-05-09T05:20:33.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.239+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f6ff01f2250 con 0x7f6fe0004d90 2024-05-09T05:20:33.236 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.240+0000 7f6fed7fa640 1 -- 172.21.15.1:0/1504803701 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f6ff00737c0 con 0x7f6fe0004d90 2024-05-09T05:20:33.237 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":112,"quorum":[1,2],"quorum_names":["b","c"],"quorum_leader_name":"b","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6fd4078ed0 msgr2=0x7f6fd407b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6fd4078ed0 0x7f6fd407b3e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f6ff00c5130 tx=0x7f6fdc008040 comp rx=0 tx=0).stop 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0004d90 msgr2=0x7f6ff00c6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0004d90 0x7f6ff00c6d80 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f6fe4001150 tx=0x7f6fe4001810 comp rx=0 tx=0).stop 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 shutdown_connections 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6fd4078ed0 0x7f6fd407b3e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6ff014ee00 0x7f6ff0156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6ff01f2f40 0x7f6ff01563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:33.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 --2- 172.21.15.1:0/1504803701 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6fe0004d90 0x7f6ff00c6d80 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:33.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 >> 172.21.15.1:0/1504803701 conn(0x7f6ff0070dd0 msgr2=0x7f6ff0159de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:33.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 shutdown_connections 2024-05-09T05:20:33.240 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:33.242+0000 7f6ff6b84640 1 -- 172.21.15.1:0/1504803701 wait complete. 2024-05-09T05:20:35.401 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:20:35.404 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:20:36.252 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:20:36.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.443+0000 7f7d1abc0640 1 Processor -- start 2024-05-09T05:20:36.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.443+0000 7f7d1abc0640 1 -- start start 2024-05-09T05:20:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d1abc0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7d140d16b0 0x7f7d140d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:36.442 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d1abc0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d1406f1b0 0x7f7d140ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d1abc0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1406f6d0 con 0x7f7d1415ca90 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d1abc0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f7d140cd0e0 con 0x7f7d140d16b0 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d1abc0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d140cc4d0 con 0x7f7d1406f1b0 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d1895c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7d140d16b0 0x7f7d140d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d1895c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f7d140d16b0 0x7f7d140d03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:49432/0 (socket says 172.21.15.1:49432) 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d1895c640 1 -- 172.21.15.1:0/376969361 learned_addr learned my addr 172.21.15.1:0/376969361 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.445+0000 7f7d13fff640 1 --2- 172.21.15.1:0/376969361 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d1406f1b0 0x7f7d140ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.446+0000 7f7d13fff640 1 -- 172.21.15.1:0/376969361 >> v2:172.21.15.196:3300/0 conn(0x7f7d140d16b0 msgr2=0x7f7d140d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.446+0000 7f7d13fff640 1 --2- 172.21.15.1:0/376969361 >> v2:172.21.15.196:3300/0 conn(0x7f7d140d16b0 0x7f7d140d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.446+0000 7f7d13fff640 1 -- 172.21.15.1:0/376969361 >> v1:172.21.15.1:6789/0 conn(0x7f7d1415ca90 legacy=0x7f7d1415cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.446+0000 7f7d13fff640 1 -- 172.21.15.1:0/376969361 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d140cc7b0 con 0x7f7d1406f1b0 2024-05-09T05:20:36.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.446+0000 7f7d1895c640 1 --2- 172.21.15.1:0/376969361 >> v2:172.21.15.196:3300/0 conn(0x7f7d140d16b0 0x7f7d140d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:20:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.447+0000 7f7d13fff640 1 --2- 172.21.15.1:0/376969361 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d1406f1b0 0x7f7d140ce740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d080097a0 tx=0x7f7d0802a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f0066d342bc0f55f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.447+0000 7f7d137fe640 1 -- 172.21.15.1:0/376969361 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08037070 con 0x7f7d1406f1b0 2024-05-09T05:20:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.447+0000 7f7d137fe640 1 -- 172.21.15.1:0/376969361 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d08032440 con 0x7f7d1406f1b0 2024-05-09T05:20:36.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.447+0000 7f7d137fe640 1 -- 172.21.15.1:0/376969361 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08032810 con 0x7f7d1406f1b0 2024-05-09T05:20:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.448+0000 7f7d1abc0640 1 -- 172.21.15.1:0/376969361 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d1406f1b0 msgr2=0x7f7d140ce740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.448+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/376969361 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d1406f1b0 0x7f7d140ce740 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d080097a0 tx=0x7f7d0802a210 comp rx=0 tx=0).stop 2024-05-09T05:20:36.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.449+0000 7f7d1abc0640 1 -- 172.21.15.1:0/376969361 shutdown_connections 2024-05-09T05:20:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.449+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/376969361 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d1406f1b0 0x7f7d140ce740 secure :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f7d080097a0 tx=0x7f7d0802a210 comp rx=0 tx=0).stop 2024-05-09T05:20:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.449+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/376969361 >> v2:172.21.15.196:3300/0 conn(0x7f7d140d16b0 0x7f7d140d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.449+0000 7f7d1abc0640 1 -- 172.21.15.1:0/376969361 >> 172.21.15.1:0/376969361 conn(0x7f7d140b3660 msgr2=0x7f7d140b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.449+0000 7f7d1abc0640 1 -- 172.21.15.1:0/376969361 shutdown_connections 2024-05-09T05:20:36.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.449+0000 7f7d1abc0640 1 -- 172.21.15.1:0/376969361 wait complete. 2024-05-09T05:20:36.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.451+0000 7f7d1abc0640 1 Processor -- start 2024-05-09T05:20:36.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.453+0000 7f7d1abc0640 1 -- start start 2024-05-09T05:20:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.455+0000 7f7d1abc0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 0x7f7d140ca600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.455+0000 7f7d1abc0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d1415ca90 0x7f7d140c97f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.455+0000 7f7d1abc0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d141e6590 0x7f7d141e80c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.455+0000 7f7d1abc0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d140d0910 con 0x7f7d140d16b0 2024-05-09T05:20:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.455+0000 7f7d1abc0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7d140ccd70 con 0x7f7d1415ca90 2024-05-09T05:20:36.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.455+0000 7f7d1abc0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d140cec80 con 0x7f7d141e6590 2024-05-09T05:20:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.456+0000 7f7d13fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d1415ca90 0x7f7d140c97f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:36.453 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.456+0000 7f7d13fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d1415ca90 0x7f7d140c97f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:49446/0 (socket says 172.21.15.1:49446) 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.456+0000 7f7d13fff640 1 -- 172.21.15.1:0/2065977911 learned_addr learned my addr 172.21.15.1:0/2065977911 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.456+0000 7f7d1915d640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d141e6590 0x7f7d141e80c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.456+0000 7f7d1895c640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 0x7f7d140ca600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d1895c640 1 -- 172.21.15.1:0/2065977911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 msgr2=0x7f7d140ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d1895c640 1 -- 172.21.15.1:0/2065977911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 msgr2=0x7f7d140ca600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d1895c640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 0x7f7d140ca600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d13fff640 1 -- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d141e6590 msgr2=0x7f7d141e80c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d13fff640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d141e6590 0x7f7d141e80c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d13fff640 1 -- 172.21.15.1:0/2065977911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 msgr2=0x7f7d140ca600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:20:36.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d1895c640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 0x7f7d140ca600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d13fff640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 0x7f7d140ca600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d13fff640 1 -- 172.21.15.1:0/2065977911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d140cab40 con 0x7f7d1415ca90 2024-05-09T05:20:36.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d13fff640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d1415ca90 0x7f7d140c97f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d080098c0 tx=0x7f7d080046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:36.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d117fa640 1 -- 172.21.15.1:0/2065977911 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08037050 con 0x7f7d1415ca90 2024-05-09T05:20:36.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d141e8f00 con 0x7f7d1415ca90 2024-05-09T05:20:36.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.457+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d141e94c0 con 0x7f7d1415ca90 2024-05-09T05:20:36.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.458+0000 7f7d117fa640 1 -- 172.21.15.1:0/2065977911 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d08038040 con 0x7f7d1415ca90 2024-05-09T05:20:36.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.458+0000 7f7d117fa640 1 -- 172.21.15.1:0/2065977911 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d08030af0 con 0x7f7d1415ca90 2024-05-09T05:20:36.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.459+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cf4005290 con 0x7f7d1415ca90 2024-05-09T05:20:36.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.460+0000 7f7d117fa640 1 -- 172.21.15.1:0/2065977911 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7d08030c90 con 0x7f7d1415ca90 2024-05-09T05:20:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.460+0000 7f7d117fa640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d00076300 0x7f7d00078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:36.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.461+0000 7f7d117fa640 1 -- 172.21.15.1:0/2065977911 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7d080b7eb0 con 0x7f7d1415ca90 2024-05-09T05:20:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.461+0000 7f7d1895c640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d00076300 0x7f7d00078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:36.458 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.461+0000 7f7d1895c640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d00076300 0x7f7d00078810 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009ce0 tx=0x7f7d0c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:36.464 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.468+0000 7f7d117fa640 1 -- 172.21.15.1:0/2065977911 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7d0804c020 con 0x7f7d1415ca90 2024-05-09T05:20:36.982 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.984+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f7cf4005580 con 0x7f7d1415ca90 2024-05-09T05:20:36.986 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.985+0000 7f7d117fa640 1 -- 172.21.15.1:0/2065977911 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f7d08081780 con 0x7f7d1415ca90 2024-05-09T05:20:36.986 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:20:36.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d00076300 msgr2=0x7f7d00078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d00076300 0x7f7d00078810 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c009ce0 tx=0x7f7d0c00b040 comp rx=0 tx=0).stop 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d1415ca90 msgr2=0x7f7d140c97f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d1415ca90 0x7f7d140c97f0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f7d080098c0 tx=0x7f7d080046c0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 shutdown_connections 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7d00076300 0x7f7d00078810 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d1415ca90 0x7f7d140c97f0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d141e6590 0x7f7d141e80c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 --2- 172.21.15.1:0/2065977911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d140d16b0 0x7f7d140ca600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.988+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 >> 172.21.15.1:0/2065977911 conn(0x7f7d140b3660 msgr2=0x7f7d14156280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.989+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 shutdown_connections 2024-05-09T05:20:36.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:36.989+0000 7f7d1abc0640 1 -- 172.21.15.1:0/2065977911 wait complete. 2024-05-09T05:20:37.004 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:20:37.004 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:20:37.006 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:20:37.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.187+0000 7f089571b640 1 Processor -- start 2024-05-09T05:20:37.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.188+0000 7f089571b640 1 -- start start 2024-05-09T05:20:37.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f089571b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f08900c6870 0x7f0890156b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f089571b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 0x7f089015c920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f089571b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f089015ff40 con 0x7f089006d800 2024-05-09T05:20:37.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f089571b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f08900cb450 con 0x7f08900c6870 2024-05-09T05:20:37.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f089571b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0890160b50 con 0x7f08900cb040 2024-05-09T05:20:37.187 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088effd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f089006d800 0x7f08900cc760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52532/0 (socket says 172.21.15.1:52532) 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088effd640 1 -- 172.21.15.1:0/3466147902 learned_addr learned my addr 172.21.15.1:0/3466147902 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088f7fe640 1 --2- 172.21.15.1:0/3466147902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 0x7f089015c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088e7fc640 1 --2- 172.21.15.1:0/3466147902 >> v2:172.21.15.196:3300/0 conn(0x7f08900c6870 0x7f0890156b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088dffb640 1 -- 172.21.15.1:0/3466147902 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3124235495 0 0) 0x7f089015ff40 con 0x7f089006d800 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088dffb640 1 -- 172.21.15.1:0/3466147902 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f0878003520 con 0x7f089006d800 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088f7fe640 1 -- 172.21.15.1:0/3466147902 >> v2:172.21.15.196:3300/0 conn(0x7f08900c6870 msgr2=0x7f0890156b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088f7fe640 1 --2- 172.21.15.1:0/3466147902 >> v2:172.21.15.196:3300/0 conn(0x7f08900c6870 0x7f0890156b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088f7fe640 1 -- 172.21.15.1:0/3466147902 >> v1:172.21.15.1:6789/0 conn(0x7f089006d800 legacy=0x7f08900cc760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.188 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.190+0000 7f088f7fe640 1 -- 172.21.15.1:0/3466147902 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0890160220 con 0x7f08900cb040 2024-05-09T05:20:37.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.191+0000 7f088f7fe640 1 --2- 172.21.15.1:0/3466147902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 0x7f089015c920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0888009d90 tx=0x7f088802a6f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dda4bfd35fdc496 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.192+0000 7f088dffb640 1 -- 172.21.15.1:0/3466147902 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0888007740 con 0x7f08900cb040 2024-05-09T05:20:37.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.192+0000 7f088dffb640 1 -- 172.21.15.1:0/3466147902 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08880078e0 con 0x7f08900cb040 2024-05-09T05:20:37.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.192+0000 7f088dffb640 1 -- 172.21.15.1:0/3466147902 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0888007cb0 con 0x7f08900cb040 2024-05-09T05:20:37.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.193+0000 7f089571b640 1 -- 172.21.15.1:0/3466147902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 msgr2=0x7f089015c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.193+0000 7f089571b640 1 --2- 172.21.15.1:0/3466147902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 0x7f089015c920 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f0888009d90 tx=0x7f088802a6f0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.193+0000 7f089571b640 1 -- 172.21.15.1:0/3466147902 shutdown_connections 2024-05-09T05:20:37.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.193+0000 7f089571b640 1 --2- 172.21.15.1:0/3466147902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 0x7f089015c920 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.193+0000 7f089571b640 1 --2- 172.21.15.1:0/3466147902 >> v2:172.21.15.196:3300/0 conn(0x7f08900c6870 0x7f0890156b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.193+0000 7f089571b640 1 -- 172.21.15.1:0/3466147902 >> 172.21.15.1:0/3466147902 conn(0x7f0890072e50 msgr2=0x7f0890073260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:37.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.193+0000 7f089571b640 1 -- 172.21.15.1:0/3466147902 shutdown_connections 2024-05-09T05:20:37.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.193+0000 7f089571b640 1 -- 172.21.15.1:0/3466147902 wait complete. 2024-05-09T05:20:37.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.197+0000 7f089571b640 1 Processor -- start 2024-05-09T05:20:37.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.197+0000 7f089571b640 1 -- start start 2024-05-09T05:20:37.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.198+0000 7f089571b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f089006d800 0x7f08900ceff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.198+0000 7f089571b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08900c6870 0x7f08900cffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f089571b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 0x7f0890154df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f089571b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f08900cce70 con 0x7f08900c6870 2024-05-09T05:20:37.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f089571b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f08901607e0 con 0x7f089006d800 2024-05-09T05:20:37.197 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f089571b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f08900ccff0 con 0x7f08900cb040 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088effd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f089006d800 0x7f08900ceff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088effd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f089006d800 0x7f08900ceff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:49472/0 (socket says 172.21.15.1:49472) 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088effd640 1 -- 172.21.15.1:0/2050180974 learned_addr learned my addr 172.21.15.1:0/2050180974 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088e7fc640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08900c6870 0x7f08900cffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088effd640 1 -- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 msgr2=0x7f0890154df0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088effd640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08900cb040 0x7f0890154df0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088effd640 1 -- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08900c6870 msgr2=0x7f08900cffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088effd640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08900c6870 0x7f08900cffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.199+0000 7f088effd640 1 -- 172.21.15.1:0/2050180974 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08900cf530 con 0x7f089006d800 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.200+0000 7f088effd640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f089006d800 0x7f08900ceff0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f088000c410 tx=0x7f088000c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.200+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0880017070 con 0x7f089006d800 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.200+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08900c9d00 con 0x7f089006d800 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.200+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08901f2db0 con 0x7f089006d800 2024-05-09T05:20:37.198 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.200+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0880005da0 con 0x7f089006d800 2024-05-09T05:20:37.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.200+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08800072d0 con 0x7f089006d800 2024-05-09T05:20:37.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.202+0000 7f089571b640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870002a70 0x7f0870004f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.199 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.202+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0870007f10 con 0x7f0870002a70 2024-05-09T05:20:37.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.202+0000 7f088e7fc640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870002a70 0x7f0870004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.202+0000 7f088e7fc640 1 -- 172.21.15.1:0/2050180974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0870008110 con 0x7f0870002a70 2024-05-09T05:20:37.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.203+0000 7f088e7fc640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870002a70 0x7f0870004f80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08900762d0 tx=0x7f0884011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.203+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f088001a070 con 0x7f089006d800 2024-05-09T05:20:37.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.204+0000 7f08777fe640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f087c0762d0 0x7f087c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.204+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f08800968c0 con 0x7f089006d800 2024-05-09T05:20:37.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.205+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884019070 con 0x7f0870002a70 2024-05-09T05:20:37.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.205+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0884004240 con 0x7f0870002a70 2024-05-09T05:20:37.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.205+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870002a70 msgr2=0x7f0870004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.205+0000 7f08777fe640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870002a70 0x7f0870004f80 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08900762d0 tx=0x7f0884011cf0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.205+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870002a70 msgr2=0x7f0870004f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:37.202 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.205+0000 7f08777fe640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870002a70 0x7f0870004f80 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f08900762d0 tx=0x7f0884011cf0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.207+0000 7f088f7fe640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f087c0762d0 0x7f087c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.208+0000 7f088f7fe640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f087c0762d0 0x7f087c0787e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0888034000 tx=0x7f088802f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.219+0000 7f089571b640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870006f80 0x7f0870007440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.215 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.219+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f087000a4f0 con 0x7f0870006f80 2024-05-09T05:20:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.219+0000 7f088e7fc640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870006f80 0x7f0870007440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.220+0000 7f088e7fc640 1 -- 172.21.15.1:0/2050180974 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0870007980 con 0x7f0870006f80 2024-05-09T05:20:37.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.220+0000 7f088e7fc640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870006f80 0x7f0870007440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f08900762d0 tx=0x7f08840046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.220+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0884019040 con 0x7f0870006f80 2024-05-09T05:20:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4553 (secure 0 0 0) 0x7f08840044d0 con 0x7f0870006f80 2024-05-09T05:20:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870006f80 msgr2=0x7f0870007440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f08777fe640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870006f80 0x7f0870007440 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f08900762d0 tx=0x7f08840046c0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f08777fe640 1 -- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870006f80 msgr2=0x7f0870007440 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:37.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f08777fe640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0870006f80 0x7f0870007440 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.218 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:20:37.218 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:20:37.218 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 114, 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 25779, 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.219 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.220 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:20:37.221 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.222 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:20:37.223 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.224 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.225 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.226 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.227 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.228 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.229 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.230 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.230 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:20:37.230 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f087c0762d0 msgr2=0x7f087c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f089571b640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f087c0762d0 0x7f087c0787e0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0888034000 tx=0x7f088802f000 comp rx=0 tx=0).stop 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f089006d800 msgr2=0x7f08900ceff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f089571b640 1 --2- 172.21.15.1:0/2050180974 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f089006d800 0x7f08900ceff0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f088000c410 tx=0x7f088000c8d0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f088f7fe640 1 -- 172.21.15.1:0/2050180974 reap_dead start 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f088f7fe640 1 -- 172.21.15.1:0/2050180974 reap_dead start 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 shutdown_connections 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.221+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 >> 172.21.15.1:0/2050180974 conn(0x7f0890072e50 msgr2=0x7f089006f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.222+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 shutdown_connections 2024-05-09T05:20:37.231 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.222+0000 7f089571b640 1 -- 172.21.15.1:0/2050180974 wait complete. 2024-05-09T05:20:37.236 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:20:37.236 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:20:37.236 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:20:37.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.418+0000 7f00dcb86640 1 Processor -- start 2024-05-09T05:20:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.418+0000 7f00dcb86640 1 -- start start 2024-05-09T05:20:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.420+0000 7f00dcb86640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f00d815aee0 0x7f00d8073a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.420+0000 7f00dcb86640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d8160c20 con 0x7f00d80c8300 2024-05-09T05:20:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.420+0000 7f00dcb86640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f00d80c4f10 con 0x7f00d81619c0 2024-05-09T05:20:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.420+0000 7f00dcb86640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f00d8073fc0 con 0x7f00d815aee0 2024-05-09T05:20:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.421+0000 7f00d6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f00d80c8300 0x7f00d80c8710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52536/0 (socket says 172.21.15.1:52536) 2024-05-09T05:20:37.418 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.421+0000 7f00d6d9d640 1 -- 172.21.15.1:0/1686113929 learned_addr learned my addr 172.21.15.1:0/1686113929 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.421+0000 7f00cffff640 1 --2- 172.21.15.1:0/1686113929 >> v2:172.21.15.196:3300/0 conn(0x7f00d815aee0 0x7f00d8073a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.422+0000 7f00cffff640 1 -- 172.21.15.1:0/1686113929 >> v1:172.21.15.196:6789/0 conn(0x7f00d81619c0 legacy=0x7f00d80c6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.422+0000 7f00cffff640 1 -- 172.21.15.1:0/1686113929 >> v1:172.21.15.1:6789/0 conn(0x7f00d80c8300 legacy=0x7f00d80c8710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.422+0000 7f00cffff640 1 -- 172.21.15.1:0/1686113929 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d80c51f0 con 0x7f00d815aee0 2024-05-09T05:20:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.423+0000 7f00cffff640 1 --2- 172.21.15.1:0/1686113929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d815aee0 0x7f00d8073a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f00c00097a0 tx=0x7f00c002a110 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c54a1b1ff38460 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.424+0000 7f00d5d9b640 1 -- 172.21.15.1:0/1686113929 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0037070 con 0x7f00d815aee0 2024-05-09T05:20:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.424+0000 7f00d5d9b640 1 -- 172.21.15.1:0/1686113929 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c0032440 con 0x7f00d815aee0 2024-05-09T05:20:37.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.424+0000 7f00d5d9b640 1 -- 172.21.15.1:0/1686113929 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c002ae60 con 0x7f00d815aee0 2024-05-09T05:20:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.426+0000 7f00dcb86640 1 -- 172.21.15.1:0/1686113929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d815aee0 msgr2=0x7f00d8073a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.426+0000 7f00dcb86640 1 --2- 172.21.15.1:0/1686113929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d815aee0 0x7f00d8073a80 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f00c00097a0 tx=0x7f00c002a110 comp rx=0 tx=0).stop 2024-05-09T05:20:37.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.427+0000 7f00dcb86640 1 -- 172.21.15.1:0/1686113929 shutdown_connections 2024-05-09T05:20:37.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.427+0000 7f00dcb86640 1 --2- 172.21.15.1:0/1686113929 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d815aee0 0x7f00d8073a80 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.427+0000 7f00dcb86640 1 -- 172.21.15.1:0/1686113929 >> 172.21.15.1:0/1686113929 conn(0x7f00d8070f40 msgr2=0x7f00d806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:37.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.427+0000 7f00dcb86640 1 -- 172.21.15.1:0/1686113929 shutdown_connections 2024-05-09T05:20:37.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.427+0000 7f00dcb86640 1 -- 172.21.15.1:0/1686113929 wait complete. 2024-05-09T05:20:37.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.430+0000 7f00dcb86640 1 Processor -- start 2024-05-09T05:20:37.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.430+0000 7f00dcb86640 1 -- start start 2024-05-09T05:20:37.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.431+0000 7f00dcb86640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d80c8300 0x7f00d81f9ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.431+0000 7f00dcb86640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f00d815aee0 0x7f00d806c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.431+0000 7f00dcb86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8164df0 0x7f00d81531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.431+0000 7f00d659c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d80c8300 0x7f00d81f9ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.431+0000 7f00d659c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d80c8300 0x7f00d81f9ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:49486/0 (socket says 172.21.15.1:49486) 2024-05-09T05:20:37.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.431+0000 7f00cffff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f00d815aee0 0x7f00d806c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8164df0 0x7f00d81531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00cffff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f00d815aee0 0x7f00d806c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:48194/0 (socket says 172.21.15.1:48194) 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8164df0 0x7f00d81531d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47422/0 (socket says 172.21.15.1:47422) 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d659c640 1 -- 172.21.15.1:0/2198003548 learned_addr learned my addr 172.21.15.1:0/2198003548 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00dcb86640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f00d81611c0 con 0x7f00d8164df0 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00dcb86640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f00d8161530 con 0x7f00d80c8300 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00dcb86640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f00d815b2f0 con 0x7f00d815aee0 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d659c640 1 -- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f00d815aee0 msgr2=0x7f00d806c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d659c640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f00d815aee0 0x7f00d806c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d659c640 1 -- 172.21.15.1:0/2198003548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8164df0 msgr2=0x7f00d81531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d659c640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f00d8164df0 0x7f00d81531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d659c640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00d8162260 con 0x7f00d80c8300 2024-05-09T05:20:37.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.432+0000 7f00d659c640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d80c8300 0x7f00d81f9ff0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f00c800b420 tx=0x7f00c800b8e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.433+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c8011830 con 0x7f00d80c8300 2024-05-09T05:20:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.433+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f00c8016070 con 0x7f00d80c8300 2024-05-09T05:20:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.433+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c8011c10 con 0x7f00d80c8300 2024-05-09T05:20:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.433+0000 7f00dcb86640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00d815c580 con 0x7f00d80c8300 2024-05-09T05:20:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.433+0000 7f00dcb86640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00d815e2b0 con 0x7f00d80c8300 2024-05-09T05:20:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.434+0000 7f00ccff9640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0002a70 0x7f00b0004f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.435+0000 7f00ccff9640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f00b0007f10 con 0x7f00b0002a70 2024-05-09T05:20:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.435+0000 7f00cffff640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0002a70 0x7f00b0004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.435+0000 7f00cffff640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f00b0008110 con 0x7f00b0002a70 2024-05-09T05:20:37.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.436+0000 7f00cffff640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0002a70 0x7f00b0004f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f00d80c4cb0 tx=0x7f00c00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.437+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f00c801b460 con 0x7f00d80c8300 2024-05-09T05:20:37.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.438+0000 7f00ceffd640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f00b80762d0 0x7f00b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.438+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f00c8097240 con 0x7f00d80c8300 2024-05-09T05:20:37.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.438+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0037040 con 0x7f00b0002a70 2024-05-09T05:20:37.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.439+0000 7f00d6d9d640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f00b80762d0 0x7f00b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.439+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f00c00093a0 con 0x7f00b0002a70 2024-05-09T05:20:37.436 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.439+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0002a70 msgr2=0x7f00b0004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.439+0000 7f00ceffd640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0002a70 0x7f00b0004f80 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f00d80c4cb0 tx=0x7f00c00046c0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.439+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0002a70 msgr2=0x7f00b0004f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:37.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.439+0000 7f00ceffd640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0002a70 0x7f00b0004f80 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.440+0000 7f00d6d9d640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f00b80762d0 0x7f00b80787e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f00d8165200 tx=0x7f00d0009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.451+0000 7f00ccff9640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0006f30 0x7f00b0007410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.451+0000 7f00ccff9640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f00b000a4f0 con 0x7f00b0006f30 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.454+0000 7f00cffff640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0006f30 0x7f00b0007410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.455+0000 7f00cffff640 1 -- 172.21.15.1:0/2198003548 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f00b0007950 con 0x7f00b0006f30 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.455+0000 7f00cffff640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0006f30 0x7f00b0007410 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00c0000c00 tx=0x7f00c0008030 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.456+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00c0037040 con 0x7f00b0006f30 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.456+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f00c0049a50 con 0x7f00b0006f30 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.456+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0006f30 msgr2=0x7f00b0007410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.456+0000 7f00ceffd640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0006f30 0x7f00b0007410 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00c0000c00 tx=0x7f00c0008030 comp rx=0 tx=0).stop 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.456+0000 7f00ceffd640 1 -- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0006f30 msgr2=0x7f00b0007410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:37.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.456+0000 7f00ceffd640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00b0006f30 0x7f00b0007410 secure :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f00c0000c00 tx=0x7f00c0008030 comp rx=0 tx=0).stop 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 114, 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 187861, 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:20:37.457 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.458 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.459 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.460 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:20:37.461 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.462 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:20:37.463 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.464 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.465 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:20:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:20:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.467 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:20:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:20:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.468 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:20:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.469 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.471 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.472 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:37.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.474 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:20:37.474 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:20:37.475 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 -- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f00b80762d0 msgr2=0x7f00b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f00b80762d0 0x7f00b80787e0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f00d8165200 tx=0x7f00d0009040 comp rx=0 tx=0).stop 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00d6d9d640 1 -- 172.21.15.1:0/2198003548 reap_dead start 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 -- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d80c8300 msgr2=0x7f00d81f9ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d80c8300 0x7f00d81f9ff0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f00c800b420 tx=0x7f00c800b8e0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 -- 172.21.15.1:0/2198003548 shutdown_connections 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 --2- 172.21.15.1:0/2198003548 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f00d80c8300 0x7f00d81f9ff0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 -- 172.21.15.1:0/2198003548 >> 172.21.15.1:0/2198003548 conn(0x7f00d8070f40 msgr2=0x7f00d815f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 -- 172.21.15.1:0/2198003548 shutdown_connections 2024-05-09T05:20:37.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.457+0000 7f00ccff9640 1 -- 172.21.15.1:0/2198003548 wait complete. 2024-05-09T05:20:37.477 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:20:37.477 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:20:37.477 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:20:37.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.661+0000 7f626bb57640 1 Processor -- start 2024-05-09T05:20:37.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.661+0000 7f626bb57640 1 -- start start 2024-05-09T05:20:37.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f626bb57640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c4ba0 0x7f62641428d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f626bb57640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f6264073e40 0x7f6264074250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f626bb57640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6264142e10 con 0x7f6264159f60 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f626bb57640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f62640739b0 con 0x7f6264073e40 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f626bb57640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f626406cb10 con 0x7f62640c4ba0 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f626a0f4640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f6264073e40 0x7f6264074250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f62698f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c4ba0 0x7f62641428d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f62698f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c4ba0 0x7f62641428d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:48210/0 (socket says 172.21.15.1:48210) 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.663+0000 7f62698f3640 1 -- 172.21.15.1:0/3453864864 learned_addr learned my addr 172.21.15.1:0/3453864864 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.664+0000 7f62698f3640 1 -- 172.21.15.1:0/3453864864 >> v2:172.21.15.196:3300/0 conn(0x7f6264073e40 msgr2=0x7f6264074250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:37.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.664+0000 7f62698f3640 1 --2- 172.21.15.1:0/3453864864 >> v2:172.21.15.196:3300/0 conn(0x7f6264073e40 0x7f6264074250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.664+0000 7f62698f3640 1 -- 172.21.15.1:0/3453864864 >> v1:172.21.15.196:6789/0 conn(0x7f6264159f60 legacy=0x7f62640c2cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.664+0000 7f62698f3640 1 -- 172.21.15.1:0/3453864864 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f626406cdf0 con 0x7f62640c4ba0 2024-05-09T05:20:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.665+0000 7f62698f3640 1 --2- 172.21.15.1:0/3453864864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c4ba0 0x7f62641428d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6260009870 tx=0x7f626002a300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4795eb1268a1bae0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.665+0000 7f62688f1640 1 -- 172.21.15.1:0/3453864864 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260037070 con 0x7f62640c4ba0 2024-05-09T05:20:37.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.665+0000 7f62688f1640 1 -- 172.21.15.1:0/3453864864 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f626002ae50 con 0x7f62640c4ba0 2024-05-09T05:20:37.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.665+0000 7f62688f1640 1 -- 172.21.15.1:0/3453864864 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260032860 con 0x7f62640c4ba0 2024-05-09T05:20:37.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.667+0000 7f626bb57640 1 -- 172.21.15.1:0/3453864864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c4ba0 msgr2=0x7f62641428d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.667+0000 7f626bb57640 1 --2- 172.21.15.1:0/3453864864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c4ba0 0x7f62641428d0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f6260009870 tx=0x7f626002a300 comp rx=0 tx=0).stop 2024-05-09T05:20:37.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.667+0000 7f626bb57640 1 -- 172.21.15.1:0/3453864864 shutdown_connections 2024-05-09T05:20:37.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.667+0000 7f626bb57640 1 --2- 172.21.15.1:0/3453864864 >> v2:172.21.15.196:3300/0 conn(0x7f6264073e40 0x7f6264074250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.667+0000 7f626bb57640 1 --2- 172.21.15.1:0/3453864864 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c4ba0 0x7f62641428d0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.667+0000 7f626bb57640 1 -- 172.21.15.1:0/3453864864 >> 172.21.15.1:0/3453864864 conn(0x7f62640cdcd0 msgr2=0x7f62640ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:37.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.667+0000 7f626bb57640 1 -- 172.21.15.1:0/3453864864 shutdown_connections 2024-05-09T05:20:37.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.667+0000 7f626bb57640 1 -- 172.21.15.1:0/3453864864 wait complete. 2024-05-09T05:20:37.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.670+0000 7f626bb57640 1 Processor -- start 2024-05-09T05:20:37.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.670+0000 7f626bb57640 1 -- start start 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.672+0000 7f626bb57640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6264073e40 0x7f62641f6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.672+0000 7f626bb57640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c4ba0 0x7f62640c5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.672+0000 7f626bb57640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6264159f60 0x7f62641378f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.672+0000 7f626bb57640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6264073640 con 0x7f62640c4ba0 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.672+0000 7f626bb57640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6264073360 con 0x7f6264159f60 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.672+0000 7f626bb57640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6264074790 con 0x7f6264073e40 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62690f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c4ba0 0x7f62640c5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62698f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6264073e40 0x7f62641f6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62698f3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6264073e40 0x7f62641f6190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:48220/0 (socket says 172.21.15.1:48220) 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62690f2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c4ba0 0x7f62640c5490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47424/0 (socket says 172.21.15.1:47424) 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62698f3640 1 -- 172.21.15.1:0/3847505005 learned_addr learned my addr 172.21.15.1:0/3847505005 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f626a0f4640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6264159f60 0x7f62641378f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62698f3640 1 -- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6264159f60 msgr2=0x7f62641378f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62698f3640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6264159f60 0x7f62641378f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62698f3640 1 -- 172.21.15.1:0/3847505005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c4ba0 msgr2=0x7f62640c5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62698f3640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f62640c4ba0 0x7f62640c5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.673+0000 7f62698f3640 1 -- 172.21.15.1:0/3847505005 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f62640c59d0 con 0x7f6264073e40 2024-05-09T05:20:37.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.674+0000 7f62698f3640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6264073e40 0x7f62641f6190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6260009870 tx=0x7f62600046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.674+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260041070 con 0x7f6264073e40 2024-05-09T05:20:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.674+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f626003c890 con 0x7f6264073e40 2024-05-09T05:20:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.674+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6260037040 con 0x7f6264073e40 2024-05-09T05:20:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.674+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f62641f26c0 con 0x7f6264073e40 2024-05-09T05:20:37.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.674+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f62641f2c80 con 0x7f6264073e40 2024-05-09T05:20:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.676+0000 7f626bb57640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c5ba0 0x7f62641525e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.676+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f62641f66d0 con 0x7f62640c5ba0 2024-05-09T05:20:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.677+0000 7f62690f2640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c5ba0 0x7f62641525e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.677+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6260044070 con 0x7f6264073e40 2024-05-09T05:20:37.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.678+0000 7f625a7fc640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6248076350 0x7f6248078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.678+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f62600b7df0 con 0x7f6264073e40 2024-05-09T05:20:37.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.679+0000 7f62690f2640 1 -- 172.21.15.1:0/3847505005 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f6264050870 con 0x7f62640c5ba0 2024-05-09T05:20:37.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.679+0000 7f626a0f4640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6248076350 0x7f6248078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.679+0000 7f62690f2640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c5ba0 0x7f62641525e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6264142de0 tx=0x7f625400f9d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.679+0000 7f626a0f4640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6248076350 0x7f6248078860 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f625c011340 tx=0x7f625c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.679+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254019070 con 0x7f62640c5ba0 2024-05-09T05:20:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.680+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f6254005320 con 0x7f62640c5ba0 2024-05-09T05:20:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.680+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c5ba0 msgr2=0x7f62641525e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.680+0000 7f625a7fc640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c5ba0 0x7f62641525e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6264142de0 tx=0x7f625400f9d0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.680+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c5ba0 msgr2=0x7f62641525e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:37.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.680+0000 7f625a7fc640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c5ba0 0x7f62641525e0 secure :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f6264142de0 tx=0x7f625400f9d0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.693+0000 7f626bb57640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c6af0 0x7f6264152b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.693+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f62640ce330 con 0x7f62640c6af0 2024-05-09T05:20:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.694+0000 7f62690f2640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c6af0 0x7f6264152b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.694+0000 7f62690f2640 1 -- 172.21.15.1:0/3847505005 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f62640c5050 con 0x7f62640c6af0 2024-05-09T05:20:37.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.694+0000 7f62690f2640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c6af0 0x7f6264152b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6254016000 tx=0x7f6254000ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.693 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.695+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6254019070 con 0x7f62640c6af0 2024-05-09T05:20:37.693 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:20:37.694 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:20:37.694 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:20:37.694 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:20:37.694 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 114, 2024-05-09T05:20:37.694 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 79585, 2024-05-09T05:20:37.694 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:20:37.695 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:20:37.695 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:20:37.695 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:20:37.695 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.695 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:20:37.695 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:37.695 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:20:37.696 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:20:37.696 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.696 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.696 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.698 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.698 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.698 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.698 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.699 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.699 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.699 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.699 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.699 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.700 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:20:37.700 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:20:37.700 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.700 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.700 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.700 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.701 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.701 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.701 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.701 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.701 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.701 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.702 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.702 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.702 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.702 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:20:37.702 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:20:37.703 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:20:37.703 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:20:37.703 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:20:37.703 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:20:37.703 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:37.703 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:37.704 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:20:37.704 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:20:37.704 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:20:37.704 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:20:37.704 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:20:37.704 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:20:37.705 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:20:37.705 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:37.705 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:20:37.705 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:37.705 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:37.705 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:37.706 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:37.706 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:37.706 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:37.706 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:37.706 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:37.706 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:37.707 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:37.707 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:37.707 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.707 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:20:37.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.707 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:20:37.708 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:20:37.710 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:20:37.710 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.710 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.710 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.710 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:20:37.711 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:20:37.711 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.711 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:20:37.711 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:20:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:20:37.713 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:20:37.713 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.713 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.713 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.714 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.714 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:20:37.714 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:20:37.714 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:37.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:37.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.715 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:37.715 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:20:37.715 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.715 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.716 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:37.716 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:20:37.716 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:37.716 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.716 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.716 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.716 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:20:37.716 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:20:37.717 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:37.717 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:37.717 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:37.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.717 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:20:37.717 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:20:37.718 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.718 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.718 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.718 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:37.718 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.718 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.718 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:20:37.719 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.719 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.719 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.719 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:20:37.719 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.719 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.720 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:20:37.720 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.720 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.720 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.720 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:20:37.720 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.720 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.721 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:20:37.721 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.721 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.721 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.721 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:20:37.721 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.722 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:37.722 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:20:37.722 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:37.722 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:37.722 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:37.722 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:37.723 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:37.723 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:37.723 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:37.723 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:20:37.723 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:20:37.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.695+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5061 (secure 0 0 0) 0x7f62540069e0 con 0x7f62640c6af0 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.695+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c6af0 msgr2=0x7f6264152b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.695+0000 7f625a7fc640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c6af0 0x7f6264152b20 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6254016000 tx=0x7f6254000ed0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f625a7fc640 1 -- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c6af0 msgr2=0x7f6264152b20 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f625a7fc640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f62640c6af0 0x7f6264152b20 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6248076350 msgr2=0x7f6248078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626bb57640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6248076350 0x7f6248078860 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f625c011340 tx=0x7f625c00b040 comp rx=0 tx=0).stop 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6264073e40 msgr2=0x7f62641f6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626bb57640 1 --2- 172.21.15.1:0/3847505005 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6264073e40 0x7f62641f6190 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f6260009870 tx=0x7f62600046c0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626a0f4640 1 -- 172.21.15.1:0/3847505005 reap_dead start 2024-05-09T05:20:37.726 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626a0f4640 1 -- 172.21.15.1:0/3847505005 reap_dead start 2024-05-09T05:20:37.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 shutdown_connections 2024-05-09T05:20:37.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 >> 172.21.15.1:0/3847505005 conn(0x7f62640cdcd0 msgr2=0x7f62640ccd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:37.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 shutdown_connections 2024-05-09T05:20:37.727 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.696+0000 7f626bb57640 1 -- 172.21.15.1:0/3847505005 wait complete. 2024-05-09T05:20:37.727 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:20:37.727 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:20:37.727 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:20:37.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.911+0000 7fcf157b1640 1 Processor -- start 2024-05-09T05:20:37.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.911+0000 7fcf157b1640 1 -- start start 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.914+0000 7fcf157b1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf08049c60 0x7fcf080d9690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.914+0000 7fcf157b1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08007800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.914+0000 7fcf157b1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcf0804fff0 con 0x7fcf080e2910 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.914+0000 7fcf157b1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcf08007d40 con 0x7fcf08049c60 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.914+0000 7fcf157b1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcf080502d0 con 0x7fcf08050960 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf0ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcf08049c60 0x7fcf080d9690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf0f7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08007800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf14fb0640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fcf080e2910 0x7fcf080e2d20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:44758/0 (socket says 172.21.15.1:44758) 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf14fb0640 1 -- 172.21.15.1:0/2238228125 learned_addr learned my addr 172.21.15.1:0/2238228125 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:37.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf0effd640 1 -- 172.21.15.1:0/2238228125 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3936039696 0 0) 0x7fcf0804fff0 con 0x7fcf080e2910 2024-05-09T05:20:37.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf0f7fe640 1 -- 172.21.15.1:0/2238228125 >> v2:172.21.15.1:3300/0 conn(0x7fcf08049c60 msgr2=0x7fcf080d9690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:37.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf0f7fe640 1 --2- 172.21.15.1:0/2238228125 >> v2:172.21.15.1:3300/0 conn(0x7fcf08049c60 0x7fcf080d9690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf0f7fe640 1 -- 172.21.15.1:0/2238228125 >> v1:172.21.15.196:6789/0 conn(0x7fcf080e2910 legacy=0x7fcf080e2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.915+0000 7fcf0f7fe640 1 -- 172.21.15.1:0/2238228125 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf08007000 con 0x7fcf08050960 2024-05-09T05:20:37.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.916+0000 7fcf0f7fe640 1 --2- 172.21.15.1:0/2238228125 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08007800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0097d0 tx=0x7fcefc02a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2b7b42b324c390a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.916+0000 7fcf0effd640 1 -- 172.21.15.1:0/2238228125 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc037070 con 0x7fcf08050960 2024-05-09T05:20:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.916+0000 7fcf0effd640 1 -- 172.21.15.1:0/2238228125 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcefc032440 con 0x7fcf08050960 2024-05-09T05:20:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.916+0000 7fcf0effd640 1 -- 172.21.15.1:0/2238228125 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcefc032810 con 0x7fcf08050960 2024-05-09T05:20:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.917+0000 7fcf157b1640 1 -- 172.21.15.1:0/2238228125 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 msgr2=0x7fcf08007800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.917+0000 7fcf157b1640 1 --2- 172.21.15.1:0/2238228125 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08007800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fcefc0097d0 tx=0x7fcefc02a1e0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.917+0000 7fcf157b1640 1 -- 172.21.15.1:0/2238228125 shutdown_connections 2024-05-09T05:20:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.917+0000 7fcf157b1640 1 --2- 172.21.15.1:0/2238228125 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08007800 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.917+0000 7fcf157b1640 1 --2- 172.21.15.1:0/2238228125 >> v2:172.21.15.1:3300/0 conn(0x7fcf08049c60 0x7fcf080d9690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.917+0000 7fcf157b1640 1 -- 172.21.15.1:0/2238228125 >> 172.21.15.1:0/2238228125 conn(0x7fcf08051220 msgr2=0x7fcf08051630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.918+0000 7fcf157b1640 1 -- 172.21.15.1:0/2238228125 shutdown_connections 2024-05-09T05:20:37.915 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.918+0000 7fcf157b1640 1 -- 172.21.15.1:0/2238228125 wait complete. 2024-05-09T05:20:37.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.921+0000 7fcf157b1640 1 Processor -- start 2024-05-09T05:20:37.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.921+0000 7fcf157b1640 1 -- start start 2024-05-09T05:20:37.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.923+0000 7fcf157b1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf08049c60 0x7fcf0816a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.923+0000 7fcf157b1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08178db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.923+0000 7fcf157b1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf080e2910 0x7fcf081792f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.924+0000 7fcf0f7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08178db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.924+0000 7fcf0f7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08178db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:48248/0 (socket says 172.21.15.1:48248) 2024-05-09T05:20:37.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.924+0000 7fcf0f7fe640 1 -- 172.21.15.1:0/54544938 learned_addr learned my addr 172.21.15.1:0/54544938 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:37.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.924+0000 7fcf14fb0640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf080e2910 0x7fcf081792f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.924+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcf080384e0 con 0x7fcf08049c60 2024-05-09T05:20:37.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.924+0000 7fcf0ffff640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf08049c60 0x7fcf0816a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.924+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcf080496f0 con 0x7fcf080e2910 2024-05-09T05:20:37.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.924+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcf08038660 con 0x7fcf08050960 2024-05-09T05:20:37.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.925+0000 7fcf0ffff640 1 -- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 msgr2=0x7fcf08178db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.925+0000 7fcf0ffff640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08178db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.925+0000 7fcf0ffff640 1 -- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf080e2910 msgr2=0x7fcf081792f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:37.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.925+0000 7fcf0ffff640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf080e2910 0x7fcf081792f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:37.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.925+0000 7fcf0ffff640 1 -- 172.21.15.1:0/54544938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcf0816a590 con 0x7fcf08049c60 2024-05-09T05:20:37.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.926+0000 7fcf0ffff640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf08049c60 0x7fcf0816a050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400e870 tx=0x7fcf0400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.926+0000 7fcf0cff9640 1 -- 172.21.15.1:0/54544938 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf04017070 con 0x7fcf08049c60 2024-05-09T05:20:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.927+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcf08053510 con 0x7fcf08049c60 2024-05-09T05:20:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.927+0000 7fcf0cff9640 1 -- 172.21.15.1:0/54544938 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcf0400ee60 con 0x7fcf08049c60 2024-05-09T05:20:37.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.927+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcf080537d0 con 0x7fcf08049c60 2024-05-09T05:20:37.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.928+0000 7fcf0cff9640 1 -- 172.21.15.1:0/54544938 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf0400cb60 con 0x7fcf08049c60 2024-05-09T05:20:37.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.930+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef0005290 con 0x7fcf08049c60 2024-05-09T05:20:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.930+0000 7fcf0cff9640 1 -- 172.21.15.1:0/54544938 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcf0401a070 con 0x7fcf08049c60 2024-05-09T05:20:37.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.931+0000 7fcf0cff9640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcef4076300 0x7fcef4078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:37.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.932+0000 7fcf0f7fe640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcef4076300 0x7fcef4078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:37.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.932+0000 7fcf0cff9640 1 -- 172.21.15.1:0/54544938 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcf040978a0 con 0x7fcf08049c60 2024-05-09T05:20:37.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.932+0000 7fcf0f7fe640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcef4076300 0x7fcef4078810 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcefc02a220 tx=0x7fcefc02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:37.937 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:37.940+0000 7fcf0cff9640 1 -- 172.21.15.1:0/54544938 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcf04061050 con 0x7fcf08049c60 2024-05-09T05:20:38.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.446+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fcef0005d90 con 0x7fcf08049c60 2024-05-09T05:20:38.445 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.448+0000 7fcf0cff9640 1 -- 172.21.15.1:0/54544938 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fcf04060950 con 0x7fcf08049c60 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcef4076300 msgr2=0x7fcef4078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcef4076300 0x7fcef4078810 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fcefc02a220 tx=0x7fcefc02e000 comp rx=0 tx=0).stop 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf08049c60 msgr2=0x7fcf0816a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf08049c60 0x7fcf0816a050 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400e870 tx=0x7fcf0400ed30 comp rx=0 tx=0).stop 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 shutdown_connections 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcef4076300 0x7fcef4078810 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcf080e2910 0x7fcf081792f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcf08050960 0x7fcf08178db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 --2- 172.21.15.1:0/54544938 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcf08049c60 0x7fcf0816a050 secure :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fcf0400e870 tx=0x7fcf0400ed30 comp rx=0 tx=0).stop 2024-05-09T05:20:38.451 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.453+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 >> 172.21.15.1:0/54544938 conn(0x7fcf08051220 msgr2=0x7fcf0804a800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:38.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.454+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 shutdown_connections 2024-05-09T05:20:38.452 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:38.454+0000 7fcf157b1640 1 -- 172.21.15.1:0/54544938 wait complete. 2024-05-09T05:20:38.470 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:20:48.471 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:20:48.471 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:20:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.658+0000 7f782bddc640 1 Processor -- start 2024-05-09T05:20:48.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.659+0000 7f782bddc640 1 -- start start 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.663+0000 7f782bddc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7824073f60 0x7f7824153910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.663+0000 7f782bddc640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78240737c0 con 0x7f7824072fe0 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.663+0000 7f782bddc640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78240c4260 con 0x7f78240c2b30 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.663+0000 7f782bddc640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f78240c4540 con 0x7f7824073f60 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.664+0000 7f782a379640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f78240c2b30 0x7f78240ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:57110/0 (socket says 172.21.15.1:57110) 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.664+0000 7f782a379640 1 -- 172.21.15.1:0/2665355708 learned_addr learned my addr 172.21.15.1:0/2665355708 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.664+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1895078431 0 0) 0x7f78240c4260 con 0x7f78240c2b30 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.664+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7810003520 con 0x7f78240c2b30 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.664+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3859184592 0 0) 0x7f7814001040 con 0x7f7824072fe0 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.664+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f78240c4260 con 0x7f7824072fe0 2024-05-09T05:20:48.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2435490380 0 0) 0x7f7810003520 con 0x7f78240c2b30 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7829b78640 1 --2- 172.21.15.1:0/2665355708 >> v2:172.21.15.1:3300/0 conn(0x7f7824073f60 0x7f7824153910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 >> v2:172.21.15.1:3300/0 conn(0x7f7824073f60 msgr2=0x7f7824153910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 --2- 172.21.15.1:0/2665355708 >> v2:172.21.15.1:3300/0 conn(0x7f7824073f60 0x7f7824153910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 >> v1:172.21.15.1:6789/0 conn(0x7f7824072fe0 legacy=0x7f78241594d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78240c4820 con 0x7f78240c2b30 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f781c001a70 con 0x7f78240c2b30 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 >> v1:172.21.15.196:6789/0 conn(0x7f78240c2b30 legacy=0x7f78240ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810004ce0 0x7f7810005110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f7810003bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7810009a30 0x7f781000be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f78240c4540 con 0x7f7810004ce0 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f781000c3c0 con 0x7f78100037a0 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f781000c5c0 con 0x7f7810009a30 2024-05-09T05:20:48.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.665+0000 7f7829b78640 1 --2- 172.21.15.1:0/2665355708 >> v2:172.21.15.1:3300/0 conn(0x7f7824073f60 0x7f7824153910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:20:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.666+0000 7f7829b78640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810004ce0 0x7f7810005110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:48.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.666+0000 7f7829377640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f7810003bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.666+0000 7f7829377640 1 -- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7810009a30 msgr2=0x7f781000be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.666+0000 7f7829377640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7810009a30 0x7f781000be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.666+0000 7f7829377640 1 -- 172.21.15.1:0/2665355708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810004ce0 msgr2=0x7f7810005110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.666+0000 7f7829377640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810004ce0 0x7f7810005110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.666+0000 7f7829377640 1 -- 172.21.15.1:0/2665355708 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7810003520 con 0x7f78100037a0 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.666+0000 7f782a379640 1 -- 172.21.15.1:0/2665355708 reap_dead start 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.667+0000 7f7829b78640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7810004ce0 0x7f7810005110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.668+0000 7f7829377640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f7810003bf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7814007b50 tx=0x7f7810005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b4ea03dc0bb3f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:48.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.669+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814036020 con 0x7f78100037a0 2024-05-09T05:20:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.670+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7814005270 con 0x7f78100037a0 2024-05-09T05:20:48.667 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.670+0000 7f7828b76640 1 -- 172.21.15.1:0/2665355708 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814007b90 con 0x7f78100037a0 2024-05-09T05:20:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.672+0000 7f782bddc640 1 -- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 msgr2=0x7f7810003bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:48.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.672+0000 7f782bddc640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f7810003bf0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f7814007b50 tx=0x7f7810005830 comp rx=0 tx=0).stop 2024-05-09T05:20:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.673+0000 7f782bddc640 1 -- 172.21.15.1:0/2665355708 shutdown_connections 2024-05-09T05:20:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.673+0000 7f782bddc640 1 --2- 172.21.15.1:0/2665355708 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f7810003bf0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.673+0000 7f782bddc640 1 -- 172.21.15.1:0/2665355708 >> 172.21.15.1:0/2665355708 conn(0x7f7824070dd0 msgr2=0x7f78240711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.673+0000 7f782bddc640 1 -- 172.21.15.1:0/2665355708 shutdown_connections 2024-05-09T05:20:48.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.674+0000 7f782bddc640 1 -- 172.21.15.1:0/2665355708 wait complete. 2024-05-09T05:20:48.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.677+0000 7f782bddc640 1 Processor -- start 2024-05-09T05:20:48.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.677+0000 7f782bddc640 1 -- start start 2024-05-09T05:20:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.680+0000 7f782bddc640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f78241f1f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:48.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.680+0000 7f782bddc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7824159510 0x7f782414c790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:48.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.680+0000 7f782bddc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f782414faf0 0x7f7824142170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.681+0000 7f7829377640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7824159510 0x7f782414c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.681+0000 7f7829377640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7824159510 0x7f782414c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51574/0 (socket says 172.21.15.1:51574) 2024-05-09T05:20:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.681+0000 7f7829b78640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f78241f1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.681+0000 7f7829b78640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f78241f1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44044/0 (socket says 172.21.15.1:44044) 2024-05-09T05:20:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.681+0000 7f7829377640 1 -- 172.21.15.1:0/3893763788 learned_addr learned my addr 172.21.15.1:0/3893763788 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.681+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f78240c2f40 con 0x7f7824159510 2024-05-09T05:20:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.681+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f78240733f0 con 0x7f78100037a0 2024-05-09T05:20:48.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.681+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f78240c30c0 con 0x7f782414faf0 2024-05-09T05:20:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.682+0000 7f7829377640 1 -- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f782414faf0 msgr2=0x7f7824142170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.682+0000 7f7829377640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f782414faf0 0x7f7824142170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.682+0000 7f7829377640 1 -- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 msgr2=0x7f78241f1f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.682+0000 7f7829377640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f78241f1f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:48.679 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.682+0000 7f7829377640 1 -- 172.21.15.1:0/3893763788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78241431f0 con 0x7f7824159510 2024-05-09T05:20:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.683+0000 7f7829377640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7824159510 0x7f782414c790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f781400c1d0 tx=0x7f7814001870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.683+0000 7f781a7fc640 1 -- 172.21.15.1:0/3893763788 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7814036030 con 0x7f7824159510 2024-05-09T05:20:48.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.683+0000 7f781a7fc640 1 -- 172.21.15.1:0/3893763788 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7814001040 con 0x7f7824159510 2024-05-09T05:20:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.683+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78240c6d00 con 0x7f7824159510 2024-05-09T05:20:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.684+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78240c7280 con 0x7f7824159510 2024-05-09T05:20:48.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.684+0000 7f781a7fc640 1 -- 172.21.15.1:0/3893763788 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f781403b620 con 0x7f7824159510 2024-05-09T05:20:48.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.686+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f780c005290 con 0x7f7824159510 2024-05-09T05:20:48.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.687+0000 7f781a7fc640 1 -- 172.21.15.1:0/3893763788 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7814039020 con 0x7f7824159510 2024-05-09T05:20:48.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.688+0000 7f781a7fc640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080763d0 0x7f78080788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.688+0000 7f781a7fc640 1 -- 172.21.15.1:0/3893763788 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f78140bbb50 con 0x7f7824159510 2024-05-09T05:20:48.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.690+0000 7f7829b78640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080763d0 0x7f78080788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:48.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.691+0000 7f7829b78640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080763d0 0x7f78080788e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f782406b7d0 tx=0x7f7820008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:48.692 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:48.696+0000 7f781a7fc640 1 -- 172.21.15.1:0/3893763788 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f7814041070 con 0x7f7824159510 2024-05-09T05:20:49.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.192+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f780c005580 con 0x7f7824159510 2024-05-09T05:20:49.190 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.192+0000 7f781a7fc640 1 -- 172.21.15.1:0/3893763788 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f78240737c0 con 0x7f7824159510 2024-05-09T05:20:49.191 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:20:49.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080763d0 msgr2=0x7f78080788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080763d0 0x7f78080788e0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f782406b7d0 tx=0x7f7820008040 comp rx=0 tx=0).stop 2024-05-09T05:20:49.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7824159510 msgr2=0x7f782414c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7824159510 0x7f782414c790 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f781400c1d0 tx=0x7f7814001870 comp rx=0 tx=0).stop 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 shutdown_connections 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78080763d0 0x7f78080788e0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f782414faf0 0x7f7824142170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7824159510 0x7f782414c790 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 --2- 172.21.15.1:0/3893763788 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78100037a0 0x7f78241f1f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 >> 172.21.15.1:0/3893763788 conn(0x7f7824070dd0 msgr2=0x7f78240b3360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 shutdown_connections 2024-05-09T05:20:49.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.196+0000 7f782bddc640 1 -- 172.21.15.1:0/3893763788 wait complete. 2024-05-09T05:20:49.213 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:20:49.213 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:20:49.213 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:20:49.213 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:20:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.406+0000 7f7d15669640 1 Processor -- start 2024-05-09T05:20:49.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.406+0000 7f7d15669640 1 -- start start 2024-05-09T05:20:49.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d15669640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7d100b4ec0 0x7f7d100cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d15669640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d10160460 0x7f7d10160870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d15669640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7d1015c7b0 con 0x7f7d100ce010 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d15669640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7d1015d0c0 con 0x7f7d100b4ec0 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d15669640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d100cca50 con 0x7f7d10160460 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d0e7fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f7d100ce010 0x7f7d1015fd50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51798/0 (socket says 172.21.15.1:51798) 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d0e7fc640 1 -- 172.21.15.1:0/3242992120 learned_addr learned my addr 172.21.15.1:0/3242992120 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d0effd640 1 --2- 172.21.15.1:0/3242992120 >> v2:172.21.15.1:3300/0 conn(0x7f7d100b4ec0 0x7f7d100cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d0f7fe640 1 --2- 172.21.15.1:0/3242992120 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d10160460 0x7f7d10160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d0dffb640 1 -- 172.21.15.1:0/3242992120 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4050500933 0 0) 0x7f7d1015c7b0 con 0x7f7d100ce010 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.408+0000 7f7d0dffb640 1 -- 172.21.15.1:0/3242992120 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f7cf8003520 con 0x7f7d100ce010 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.409+0000 7f7d0effd640 1 -- 172.21.15.1:0/3242992120 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d10160460 msgr2=0x7f7d10160870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.409+0000 7f7d0effd640 1 --2- 172.21.15.1:0/3242992120 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d10160460 0x7f7d10160870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.409+0000 7f7d0effd640 1 -- 172.21.15.1:0/3242992120 >> v1:172.21.15.1:6789/0 conn(0x7f7d100ce010 legacy=0x7f7d1015fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.409+0000 7f7d0effd640 1 -- 172.21.15.1:0/3242992120 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d100ccd30 con 0x7f7d100b4ec0 2024-05-09T05:20:49.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.410+0000 7f7d0effd640 1 --2- 172.21.15.1:0/3242992120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d100cf9d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7d1015d240 tx=0x7f7d0002a160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe588aa5548e9b5f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.410+0000 7f7d0dffb640 1 -- 172.21.15.1:0/3242992120 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00037070 con 0x7f7d100b4ec0 2024-05-09T05:20:49.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.410+0000 7f7d0dffb640 1 -- 172.21.15.1:0/3242992120 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d00032440 con 0x7f7d100b4ec0 2024-05-09T05:20:49.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.411+0000 7f7d0dffb640 1 -- 172.21.15.1:0/3242992120 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00032810 con 0x7f7d100b4ec0 2024-05-09T05:20:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.411+0000 7f7d15669640 1 -- 172.21.15.1:0/3242992120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 msgr2=0x7f7d100cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.411+0000 7f7d15669640 1 --2- 172.21.15.1:0/3242992120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d100cf9d0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f7d1015d240 tx=0x7f7d0002a160 comp rx=0 tx=0).stop 2024-05-09T05:20:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.412+0000 7f7d15669640 1 -- 172.21.15.1:0/3242992120 shutdown_connections 2024-05-09T05:20:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.412+0000 7f7d15669640 1 --2- 172.21.15.1:0/3242992120 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d10160460 0x7f7d10160870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.412+0000 7f7d15669640 1 --2- 172.21.15.1:0/3242992120 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d100cf9d0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.412+0000 7f7d15669640 1 -- 172.21.15.1:0/3242992120 >> 172.21.15.1:0/3242992120 conn(0x7f7d100d2360 msgr2=0x7f7d100ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.412+0000 7f7d15669640 1 -- 172.21.15.1:0/3242992120 shutdown_connections 2024-05-09T05:20:49.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.412+0000 7f7d15669640 1 -- 172.21.15.1:0/3242992120 wait complete. 2024-05-09T05:20:49.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.415+0000 7f7d15669640 1 Processor -- start 2024-05-09T05:20:49.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.415+0000 7f7d15669640 1 -- start start 2024-05-09T05:20:49.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.417+0000 7f7d15669640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d101f3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.417+0000 7f7d15669640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d100ce010 0x7f7d1014f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d15669640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d10160460 0x7f7d100d0180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d0effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d101f3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d0effd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d101f3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:51582/0 (socket says 172.21.15.1:51582) 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d0e7fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d100ce010 0x7f7d1014f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d0e7fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d100ce010 0x7f7d1014f230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:43486/0 (socket says 172.21.15.1:43486) 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d0effd640 1 -- 172.21.15.1:0/327492635 learned_addr learned my addr 172.21.15.1:0/327492635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d100cc750 con 0x7f7d100b4ec0 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7d1015cd50 con 0x7f7d10160460 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7d100cff10 con 0x7f7d100ce010 2024-05-09T05:20:49.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.418+0000 7f7d0f7fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d10160460 0x7f7d100d0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.419+0000 7f7d0effd640 1 -- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d100ce010 msgr2=0x7f7d1014f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.419+0000 7f7d0effd640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d100ce010 0x7f7d1014f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.419+0000 7f7d0effd640 1 -- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d10160460 msgr2=0x7f7d100d0180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.419+0000 7f7d0effd640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d10160460 0x7f7d100d0180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.419+0000 7f7d0effd640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d1006ee60 con 0x7f7d100b4ec0 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.419+0000 7f7d0e7fc640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7d100ce010 0x7f7d1014f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.419+0000 7f7d0f7fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7d10160460 0x7f7d100d0180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.420+0000 7f7d0effd640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d101f3640 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7d00009890 tx=0x7f7d000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.420+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d00037050 con 0x7f7d100b4ec0 2024-05-09T05:20:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.420+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7d00043070 con 0x7f7d100b4ec0 2024-05-09T05:20:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.420+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0003cb10 con 0x7f7d100b4ec0 2024-05-09T05:20:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.421+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d101f6130 con 0x7f7d100b4ec0 2024-05-09T05:20:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.421+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d101f63d0 con 0x7f7d100b4ec0 2024-05-09T05:20:49.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.423+0000 7f7d15669640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1015a5b0 0x7f7d100d06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.423+0000 7f7d0e7fc640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1015a5b0 0x7f7d100d06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.421 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.423+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d101f3b80 con 0x7f7d1015a5b0 2024-05-09T05:20:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.424+0000 7f7d0e7fc640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7d10050870 con 0x7f7d1015a5b0 2024-05-09T05:20:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.424+0000 7f7d0e7fc640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1015a5b0 0x7f7d100d06c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d04007fd0 tx=0x7f7d0400f5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.424+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7d0003ccb0 con 0x7f7d100b4ec0 2024-05-09T05:20:49.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.425+0000 7f7cf77fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cfc076350 0x7f7cfc078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.426+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7d000b8210 con 0x7f7d100b4ec0 2024-05-09T05:20:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.426+0000 7f7d0f7fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cfc076350 0x7f7cfc078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.426+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400fcf0 con 0x7f7d1015a5b0 2024-05-09T05:20:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.426+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7d04018070 con 0x7f7d1015a5b0 2024-05-09T05:20:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.426+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1015a5b0 msgr2=0x7f7d100d06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.426+0000 7f7cf77fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1015a5b0 0x7f7d100d06c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d04007fd0 tx=0x7f7d0400f5a0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.426+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1015a5b0 msgr2=0x7f7d100d06c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:49.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.426+0000 7f7cf77fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d1015a5b0 0x7f7d100d06c0 secure :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f7d04007fd0 tx=0x7f7d0400f5a0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.427+0000 7f7d0f7fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cfc076350 0x7f7cfc078860 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009fd0 tx=0x7f7d0800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.439+0000 7f7d15669640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d101f65a0 0x7f7d10075ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.439+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7d10150330 con 0x7f7d101f65a0 2024-05-09T05:20:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.439+0000 7f7d0e7fc640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d101f65a0 0x7f7d10075ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.437 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.440+0000 7f7d0e7fc640 1 -- 172.21.15.1:0/327492635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7d100cd2b0 con 0x7f7d101f65a0 2024-05-09T05:20:49.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.440+0000 7f7d0e7fc640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d101f65a0 0x7f7d10075ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d04017000 tx=0x7f7d04001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.441+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0401de00 con 0x7f7d101f65a0 2024-05-09T05:20:49.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.441+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4554 (secure 0 0 0) 0x7f7d04013600 con 0x7f7d101f65a0 2024-05-09T05:20:49.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.441+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d101f65a0 msgr2=0x7f7d10075ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.441+0000 7f7cf77fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d101f65a0 0x7f7d10075ca0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d04017000 tx=0x7f7d04001080 comp rx=0 tx=0).stop 2024-05-09T05:20:49.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.441+0000 7f7cf77fe640 1 -- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d101f65a0 msgr2=0x7f7d10075ca0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:49.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.441+0000 7f7cf77fe640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d101f65a0 0x7f7d10075ca0 secure :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7d04017000 tx=0x7f7d04001080 comp rx=0 tx=0).stop 2024-05-09T05:20:49.440 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:20:49.440 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:20:49.440 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:20:49.440 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 114, 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 38001, 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:20:49.441 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.442 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.443 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:20:49.444 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.445 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.446 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.447 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.448 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.450 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.453 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.454 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:20:49.455 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:20:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.442+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cfc076350 msgr2=0x7f7cfc078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.442+0000 7f7d15669640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cfc076350 0x7f7cfc078860 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f7d08009fd0 tx=0x7f7d0800b040 comp rx=0 tx=0).stop 2024-05-09T05:20:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.442+0000 7f7d0f7fe640 1 -- 172.21.15.1:0/327492635 reap_dead start 2024-05-09T05:20:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.442+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 msgr2=0x7f7d101f3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.442+0000 7f7d15669640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d101f3640 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7d00009890 tx=0x7f7d000046c0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.442+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 shutdown_connections 2024-05-09T05:20:49.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.442+0000 7f7d15669640 1 --2- 172.21.15.1:0/327492635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7d100b4ec0 0x7f7d101f3640 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.443+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 >> 172.21.15.1:0/327492635 conn(0x7f7d100d2360 msgr2=0x7f7d100b6860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:49.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.443+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 shutdown_connections 2024-05-09T05:20:49.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.443+0000 7f7d15669640 1 -- 172.21.15.1:0/327492635 wait complete. 2024-05-09T05:20:49.458 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:20:49.458 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:20:49.458 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:20:49.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.650+0000 7fc39ed84640 1 Processor -- start 2024-05-09T05:20:49.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.650+0000 7fc39ed84640 1 -- start start 2024-05-09T05:20:49.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.653+0000 7fc39ed84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3980c4e00 0x7fc3980c5210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.653+0000 7fc39ed84640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3980c6eb0 0x7fc3980c72c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.653+0000 7fc39ed84640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc398158970 con 0x7fc3980c8870 2024-05-09T05:20:49.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.653+0000 7fc39ed84640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc39815c560 con 0x7fc3980c6eb0 2024-05-09T05:20:49.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.653+0000 7fc39ed84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc3980c8c80 con 0x7fc3980c4e00 2024-05-09T05:20:49.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.654+0000 7fc39cb20640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc3980c8870 0x7fc39815be50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:51802/0 (socket says 172.21.15.1:51802) 2024-05-09T05:20:49.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.654+0000 7fc39cb20640 1 -- 172.21.15.1:0/1762406682 learned_addr learned my addr 172.21.15.1:0/1762406682 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:49.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.654+0000 7fc39d321640 1 --2- 172.21.15.1:0/1762406682 >> v2:172.21.15.1:3300/0 conn(0x7fc3980c6eb0 0x7fc3980c72c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.654+0000 7fc39d321640 1 -- 172.21.15.1:0/1762406682 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3980c4e00 msgr2=0x7fc3980c5210 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:49.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.654+0000 7fc39d321640 1 --2- 172.21.15.1:0/1762406682 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3980c4e00 0x7fc3980c5210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.654+0000 7fc39d321640 1 -- 172.21.15.1:0/1762406682 >> v1:172.21.15.1:6789/0 conn(0x7fc3980c8870 legacy=0x7fc39815be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.654+0000 7fc39d321640 1 -- 172.21.15.1:0/1762406682 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3980c7a60 con 0x7fc3980c6eb0 2024-05-09T05:20:49.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.655+0000 7fc39d321640 1 --2- 172.21.15.1:0/1762406682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 0x7fc3980c72c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc390009dc0 tx=0x7fc39002a5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f19f15b2401a6714 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.655+0000 7fc3977fe640 1 -- 172.21.15.1:0/1762406682 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390007740 con 0x7fc3980c6eb0 2024-05-09T05:20:49.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.655+0000 7fc3977fe640 1 -- 172.21.15.1:0/1762406682 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc3900078e0 con 0x7fc3980c6eb0 2024-05-09T05:20:49.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.656+0000 7fc3977fe640 1 -- 172.21.15.1:0/1762406682 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390007cb0 con 0x7fc3980c6eb0 2024-05-09T05:20:49.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.657+0000 7fc39ed84640 1 -- 172.21.15.1:0/1762406682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 msgr2=0x7fc3980c72c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.657+0000 7fc39ed84640 1 --2- 172.21.15.1:0/1762406682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 0x7fc3980c72c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc390009dc0 tx=0x7fc39002a5b0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.657+0000 7fc39ed84640 1 -- 172.21.15.1:0/1762406682 shutdown_connections 2024-05-09T05:20:49.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.657+0000 7fc39ed84640 1 --2- 172.21.15.1:0/1762406682 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3980c4e00 0x7fc3980c5210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.657+0000 7fc39ed84640 1 --2- 172.21.15.1:0/1762406682 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 0x7fc3980c72c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.657+0000 7fc39ed84640 1 -- 172.21.15.1:0/1762406682 >> 172.21.15.1:0/1762406682 conn(0x7fc398070f40 msgr2=0x7fc39806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:49.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.657+0000 7fc39ed84640 1 -- 172.21.15.1:0/1762406682 shutdown_connections 2024-05-09T05:20:49.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.657+0000 7fc39ed84640 1 -- 172.21.15.1:0/1762406682 wait complete. 2024-05-09T05:20:49.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.660+0000 7fc39ed84640 1 Processor -- start 2024-05-09T05:20:49.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.660+0000 7fc39ed84640 1 -- start start 2024-05-09T05:20:49.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.661+0000 7fc39ed84640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3980c4e00 0x7fc398151e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39ed84640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 0x7fc3981f7140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39ed84640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3980c8870 0x7fc3980c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39ed84640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3980c7800 con 0x7fc3980c6eb0 2024-05-09T05:20:49.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39ed84640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc398159230 con 0x7fc3980c4e00 2024-05-09T05:20:49.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39ed84640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc398158ec0 con 0x7fc3980c8870 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39cb20640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3980c4e00 0x7fc398151e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39cb20640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3980c4e00 0x7fc398151e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:44066/0 (socket says 172.21.15.1:44066) 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39cb20640 1 -- 172.21.15.1:0/2081521550 learned_addr learned my addr 172.21.15.1:0/2081521550 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc397fff640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 0x7fc3981f7140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39d321640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3980c8870 0x7fc3980c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39cb20640 1 -- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3980c8870 msgr2=0x7fc3980c9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39cb20640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3980c8870 0x7fc3980c9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39cb20640 1 -- 172.21.15.1:0/2081521550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 msgr2=0x7fc3981f7140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39cb20640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 0x7fc3981f7140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc39cb20640 1 -- 172.21.15.1:0/2081521550 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc398143500 con 0x7fc3980c4e00 2024-05-09T05:20:49.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.662+0000 7fc397fff640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3980c6eb0 0x7fc3981f7140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:20:49.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.663+0000 7fc39cb20640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3980c4e00 0x7fc398151e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc38800c580 tx=0x7fc38800ca40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.663+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc388018070 con 0x7fc3980c4e00 2024-05-09T05:20:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.663+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc388007560 con 0x7fc3980c4e00 2024-05-09T05:20:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.663+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc388004a20 con 0x7fc3980c4e00 2024-05-09T05:20:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.663+0000 7fc39ed84640 1 -- 172.21.15.1:0/2081521550 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3981f7680 con 0x7fc3980c4e00 2024-05-09T05:20:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.664+0000 7fc39ed84640 1 -- 172.21.15.1:0/2081521550 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc398142980 con 0x7fc3980c4e00 2024-05-09T05:20:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.664+0000 7fc39ed84640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981f7850 0x7fc398142c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.664+0000 7fc39ed84640 1 -- 172.21.15.1:0/2081521550 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc398050870 con 0x7fc3981f7850 2024-05-09T05:20:49.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.665+0000 7fc397fff640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981f7850 0x7fc398142c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.665+0000 7fc397fff640 1 -- 172.21.15.1:0/2081521550 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3980c8440 con 0x7fc3981f7850 2024-05-09T05:20:49.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.666+0000 7fc397fff640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981f7850 0x7fc398142c60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00eae0 tx=0x7fc38c009ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.666+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c017070 con 0x7fc3981f7850 2024-05-09T05:20:49.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.666+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc388004bc0 con 0x7fc3980c4e00 2024-05-09T05:20:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.667+0000 7fc3957fa640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc384076200 0x7fc384078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.667+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc388096980 con 0x7fc3980c4e00 2024-05-09T05:20:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.667+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc38c00ca10 con 0x7fc3981f7850 2024-05-09T05:20:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.667+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981f7850 msgr2=0x7fc398142c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.667+0000 7fc3957fa640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981f7850 0x7fc398142c60 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00eae0 tx=0x7fc38c009ec0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.667+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981f7850 msgr2=0x7fc398142c60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:49.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.667+0000 7fc3957fa640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981f7850 0x7fc398142c60 secure :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00eae0 tx=0x7fc38c009ec0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.668+0000 7fc39d321640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc384076200 0x7fc384078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.674+0000 7fc39d321640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc384076200 0x7fc384078710 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc390034000 tx=0x7fc39002f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.680+0000 7fc39ed84640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981410b0 0x7fc3981414c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.680+0000 7fc39ed84640 1 -- 172.21.15.1:0/2081521550 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc39806d4e0 con 0x7fc3981410b0 2024-05-09T05:20:49.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.681+0000 7fc397fff640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981410b0 0x7fc3981414c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.680 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.682+0000 7fc397fff640 1 -- 172.21.15.1:0/2081521550 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc3981583e0 con 0x7fc3981410b0 2024-05-09T05:20:49.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.684+0000 7fc397fff640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981410b0 0x7fc3981414c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00eae0 tx=0x7fc38c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.684+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc38c02a020 con 0x7fc3981410b0 2024-05-09T05:20:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.684+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7fc38c00ce80 con 0x7fc3981410b0 2024-05-09T05:20:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.684+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981410b0 msgr2=0x7fc3981414c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.684+0000 7fc3957fa640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981410b0 0x7fc3981414c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00eae0 tx=0x7fc38c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.684+0000 7fc3957fa640 1 -- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981410b0 msgr2=0x7fc3981414c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:49.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.684+0000 7fc3957fa640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3981410b0 0x7fc3981414c0 secure :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fc38c00eae0 tx=0x7fc38c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 114, 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 200089, 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:20:49.683 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 12, 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.684 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.685 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:20:49.686 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.687 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.688 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.689 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.690 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.691 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:20:49.692 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.693 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.694 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.695 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.696 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:20:49.696 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:20:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 -- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc384076200 msgr2=0x7fc384078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc384076200 0x7fc384078710 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fc390034000 tx=0x7fc39002f000 comp rx=0 tx=0).stop 2024-05-09T05:20:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 -- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3980c4e00 msgr2=0x7fc398151e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.696 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3980c4e00 0x7fc398151e20 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fc38800c580 tx=0x7fc38800ca40 comp rx=0 tx=0).stop 2024-05-09T05:20:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc39d321640 1 -- 172.21.15.1:0/2081521550 reap_dead start 2024-05-09T05:20:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 -- 172.21.15.1:0/2081521550 shutdown_connections 2024-05-09T05:20:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 --2- 172.21.15.1:0/2081521550 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc3980c4e00 0x7fc398151e20 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 -- 172.21.15.1:0/2081521550 >> 172.21.15.1:0/2081521550 conn(0x7fc398070f40 msgr2=0x7fc3980745f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 -- 172.21.15.1:0/2081521550 shutdown_connections 2024-05-09T05:20:49.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.685+0000 7fc37effd640 1 -- 172.21.15.1:0/2081521550 wait complete. 2024-05-09T05:20:49.699 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:20:49.699 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:20:49.699 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:20:49.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.885+0000 7fb40e77d640 1 Processor -- start 2024-05-09T05:20:49.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.885+0000 7fb40e77d640 1 -- start start 2024-05-09T05:20:49.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.888+0000 7fb40e77d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081592a0 0x7fb40806c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.888+0000 7fb40e77d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb40806cc30 0x7fb4081631a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.888+0000 7fb40e77d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb408152f00 con 0x7fb408152af0 2024-05-09T05:20:49.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.888+0000 7fb40e77d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb40806f010 con 0x7fb40806cc30 2024-05-09T05:20:49.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.888+0000 7fb40e77d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb40806fb90 con 0x7fb4081592a0 2024-05-09T05:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.890+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081592a0 0x7fb40806c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.890+0000 7fb4077fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081592a0 0x7fb40806c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:53330/0 (socket says 172.21.15.1:53330) 2024-05-09T05:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.890+0000 7fb4077fe640 1 -- 172.21.15.1:0/2172399434 learned_addr learned my addr 172.21.15.1:0/2172399434 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.890+0000 7fb40cd1a640 1 --2- 172.21.15.1:0/2172399434 >> v2:172.21.15.196:3300/0 conn(0x7fb40806cc30 0x7fb4081631a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.891+0000 7fb40cd1a640 1 -- 172.21.15.1:0/2172399434 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081592a0 msgr2=0x7fb40806c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.891+0000 7fb40cd1a640 1 --2- 172.21.15.1:0/2172399434 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081592a0 0x7fb40806c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.891+0000 7fb40cd1a640 1 -- 172.21.15.1:0/2172399434 >> v1:172.21.15.1:6789/0 conn(0x7fb408152af0 legacy=0x7fb4080c3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.891+0000 7fb40cd1a640 1 -- 172.21.15.1:0/2172399434 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4081596b0 con 0x7fb40806cc30 2024-05-09T05:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.892+0000 7fb40cd1a640 1 --2- 172.21.15.1:0/2172399434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40806cc30 0x7fb4081631a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb40000f370 tx=0x7fb40002ea90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f043b3489853b50b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.892+0000 7fb406ffd640 1 -- 172.21.15.1:0/2172399434 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb400036070 con 0x7fb40806cc30 2024-05-09T05:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.893+0000 7fb406ffd640 1 -- 172.21.15.1:0/2172399434 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb40000d4e0 con 0x7fb40806cc30 2024-05-09T05:20:49.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.893+0000 7fb406ffd640 1 -- 172.21.15.1:0/2172399434 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb40000b960 con 0x7fb40806cc30 2024-05-09T05:20:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.895+0000 7fb40e77d640 1 -- 172.21.15.1:0/2172399434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40806cc30 msgr2=0x7fb4081631a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.895+0000 7fb40e77d640 1 --2- 172.21.15.1:0/2172399434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40806cc30 0x7fb4081631a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fb40000f370 tx=0x7fb40002ea90 comp rx=0 tx=0).stop 2024-05-09T05:20:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.895+0000 7fb40e77d640 1 -- 172.21.15.1:0/2172399434 shutdown_connections 2024-05-09T05:20:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.895+0000 7fb40e77d640 1 --2- 172.21.15.1:0/2172399434 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb40806cc30 0x7fb4081631a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.895+0000 7fb40e77d640 1 --2- 172.21.15.1:0/2172399434 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081592a0 0x7fb40806c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.895+0000 7fb40e77d640 1 -- 172.21.15.1:0/2172399434 >> 172.21.15.1:0/2172399434 conn(0x7fb4080cdcd0 msgr2=0x7fb4080ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:49.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.895+0000 7fb40e77d640 1 -- 172.21.15.1:0/2172399434 shutdown_connections 2024-05-09T05:20:49.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.895+0000 7fb40e77d640 1 -- 172.21.15.1:0/2172399434 wait complete. 2024-05-09T05:20:49.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.899+0000 7fb40e77d640 1 Processor -- start 2024-05-09T05:20:49.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.899+0000 7fb40e77d640 1 -- start start 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.901+0000 7fb40e77d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40806cc30 0x7fb4080c7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.901+0000 7fb40e77d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408152af0 0x7fb408156bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.902+0000 7fb40e77d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4081592a0 0x7fb408149070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.902+0000 7fb40e77d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb40806f540 con 0x7fb40806cc30 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.902+0000 7fb40e77d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb40806f260 con 0x7fb4081592a0 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.902+0000 7fb40cd1a640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4081592a0 0x7fb408149070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.902+0000 7fb40cd1a640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4081592a0 0x7fb408149070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:56524/0 (socket says 172.21.15.1:56524) 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.902+0000 7fb40cd1a640 1 -- 172.21.15.1:0/2141174033 learned_addr learned my addr 172.21.15.1:0/2141174033 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:49.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.902+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb40806f820 con 0x7fb408152af0 2024-05-09T05:20:49.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.902+0000 7fb4077fe640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408152af0 0x7fb408156bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.903+0000 7fb4077fe640 1 -- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4081592a0 msgr2=0x7fb408149070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.903+0000 7fb4077fe640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4081592a0 0x7fb408149070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.903+0000 7fb4077fe640 1 -- 172.21.15.1:0/2141174033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40806cc30 msgr2=0x7fb4080c7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.903+0000 7fb4077fe640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb40806cc30 0x7fb4080c7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.903+0000 7fb4077fe640 1 -- 172.21.15.1:0/2141174033 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb408157100 con 0x7fb408152af0 2024-05-09T05:20:49.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.904+0000 7fb4077fe640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408152af0 0x7fb408156bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800b710 tx=0x7fb3f800ae90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.904+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f800ec00 con 0x7fb408152af0 2024-05-09T05:20:49.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.904+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3f801b070 con 0x7fb408152af0 2024-05-09T05:20:49.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.904+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3f8016470 con 0x7fb408152af0 2024-05-09T05:20:49.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.904+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4081581e0 con 0x7fb408152af0 2024-05-09T05:20:49.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.905+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4081587a0 con 0x7fb408152af0 2024-05-09T05:20:49.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.906+0000 7fb40e77d640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081495b0 0x7fb40814f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.907+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb408050870 con 0x7fb4081495b0 2024-05-09T05:20:49.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.907+0000 7fb407fff640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081495b0 0x7fb40814f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.907+0000 7fb407fff640 1 -- 172.21.15.1:0/2141174033 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb408074c90 con 0x7fb4081495b0 2024-05-09T05:20:49.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.907+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb3f8016610 con 0x7fb408152af0 2024-05-09T05:20:49.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.907+0000 7fb407fff640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081495b0 0x7fb40814f0e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c340 tx=0x7fb3fc00c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.909+0000 7fb404ff9640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3f40763d0 0x7fb3f40788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.909+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb3f8098140 con 0x7fb408152af0 2024-05-09T05:20:49.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.909+0000 7fb40cd1a640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3f40763d0 0x7fb3f40788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.909+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc017070 con 0x7fb4081495b0 2024-05-09T05:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.909+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb3fc0052d0 con 0x7fb4081495b0 2024-05-09T05:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.909+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081495b0 msgr2=0x7fb40814f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.910+0000 7fb404ff9640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081495b0 0x7fb40814f0e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc00c340 tx=0x7fb3fc00c800 comp rx=0 tx=0).stop 2024-05-09T05:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.910+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081495b0 msgr2=0x7fb40814f0e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.910+0000 7fb40cd1a640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3f40763d0 0x7fb3f40788e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb408151b40 tx=0x7fb40003d040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.910+0000 7fb404ff9640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb4081495b0 0x7fb40814f0e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.924+0000 7fb40e77d640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408158e30 0x7fb40814f620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.924+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb408074140 con 0x7fb408158e30 2024-05-09T05:20:49.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.925+0000 7fb407fff640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408158e30 0x7fb40814f620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:49.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.925+0000 7fb407fff640 1 -- 172.21.15.1:0/2141174033 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb4081589e0 con 0x7fb408158e30 2024-05-09T05:20:49.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.926+0000 7fb407fff640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408158e30 0x7fb40814f620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc014000 tx=0x7fb3fc001080 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.927+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc017040 con 0x7fb408158e30 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.927+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7fb3fc0035f0 con 0x7fb408158e30 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.927+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408158e30 msgr2=0x7fb40814f620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.927+0000 7fb404ff9640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408158e30 0x7fb40814f620 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc014000 tx=0x7fb3fc001080 comp rx=0 tx=0).stop 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.927+0000 7fb404ff9640 1 -- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408158e30 msgr2=0x7fb40814f620 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.927+0000 7fb404ff9640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408158e30 0x7fb40814f620 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:20:49.925 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 114, 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 91817, 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 13, 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.926 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:20:49.927 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:20:49.929 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:20:49.930 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.932 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.934 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.935 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:20:49.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:20:49.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:20:49.939 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:20:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.928+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3f40763d0 msgr2=0x7fb3f40788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.928+0000 7fb40e77d640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3f40763d0 0x7fb3f40788e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fb408151b40 tx=0x7fb40003d040 comp rx=0 tx=0).stop 2024-05-09T05:20:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.928+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408152af0 msgr2=0x7fb408156bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.928+0000 7fb40e77d640 1 --2- 172.21.15.1:0/2141174033 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb408152af0 0x7fb408156bc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb3f800b710 tx=0x7fb3f800ae90 comp rx=0 tx=0).stop 2024-05-09T05:20:49.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.928+0000 7fb40cd1a640 1 -- 172.21.15.1:0/2141174033 reap_dead start 2024-05-09T05:20:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.928+0000 7fb40cd1a640 1 -- 172.21.15.1:0/2141174033 reap_dead start 2024-05-09T05:20:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.928+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 shutdown_connections 2024-05-09T05:20:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.928+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 >> 172.21.15.1:0/2141174033 conn(0x7fb4080cdcd0 msgr2=0x7fb4080caf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.929+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 shutdown_connections 2024-05-09T05:20:49.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:49.929+0000 7fb40e77d640 1 -- 172.21.15.1:0/2141174033 wait complete. 2024-05-09T05:20:49.945 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:20:49.945 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['c'] 2024-05-09T05:20:49.945 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:20:49.945 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.c 2024-05-09T05:20:49.945 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:20:49.945 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:20:49.946 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:20:49.964 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:20:49.964 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:20:49.964 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:20:50.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.146+0000 7fd3d3e02640 1 Processor -- start 2024-05-09T05:20:50.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.147+0000 7fd3d3e02640 1 -- start start 2024-05-09T05:20:50.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d3e02640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 0x7fd3cc0c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d3e02640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd3cc1538b0 0x7fd3cc0c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d3e02640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd3cc1594f0 con 0x7fd3cc0c9810 2024-05-09T05:20:50.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d3e02640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd3cc15a380 con 0x7fd3cc1538b0 2024-05-09T05:20:50.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d3e02640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc159770 con 0x7fd3cc0c87d0 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d1b9e640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 msgr2=0x7fd3cc0c5ca0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d1b9e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 0x7fd3cc0c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d239f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd3cc0c9810 0x7fd3cc15cc90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:34146/0 (socket says 172.21.15.1:34146) 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.149+0000 7fd3d239f640 1 -- 172.21.15.1:0/623149815 learned_addr learned my addr 172.21.15.1:0/623149815 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2175379862 0 0) 0x7fd3c4001110 con 0x7fd3cc0c9810 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd3b4003520 con 0x7fd3cc0c9810 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1172436664 0 0) 0x7fd3b4003520 con 0x7fd3cc0c9810 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 msgr2=0x7fd3cc0c5ca0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 0x7fd3cc0c5ca0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 >> v2:172.21.15.196:3300/0 conn(0x7fd3cc1538b0 msgr2=0x7fd3cc0c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 --2- 172.21.15.1:0/623149815 >> v2:172.21.15.196:3300/0 conn(0x7fd3cc1538b0 0x7fd3cc0c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc159a50 con 0x7fd3cc0c9810 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.150+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fd3c40018c0 con 0x7fd3cc0c9810 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 >> v1:172.21.15.1:6789/0 conn(0x7fd3cc0c9810 legacy=0x7fd3cc15cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d0b9c640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3b40039b0 0x7fd3b4003e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d0b9c640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b4004800 0x7fd3b4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d0b9c640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 0x7fd3b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc15a380 con 0x7fd3b4004800 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc159770 con 0x7fd3b40039b0 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd3b4005650 con 0x7fd3cc0c87d0 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d239f640 1 -- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 msgr2=0x7fd3b400b9c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d239f640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 0x7fd3b400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:50.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d139d640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b4004800 0x7fd3b4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:50.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d1b9e640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3b40039b0 0x7fd3b4003e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:50.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d139d640 1 -- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 msgr2=0x7fd3b400b9c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:50.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d139d640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc0c87d0 0x7fd3b400b9c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d139d640 1 -- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3b40039b0 msgr2=0x7fd3b4003e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:20:50.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d139d640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3b40039b0 0x7fd3b4003e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.151+0000 7fd3d139d640 1 -- 172.21.15.1:0/623149815 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3b4003520 con 0x7fd3b4004800 2024-05-09T05:20:50.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.153+0000 7fd3d139d640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b4004800 0x7fd3b4004c10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009e40 tx=0x7fd3b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc8f34d8c9732055 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:50.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.153+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc034020 con 0x7fd3b4004800 2024-05-09T05:20:50.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.153+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc006e50 con 0x7fd3b4004800 2024-05-09T05:20:50.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.153+0000 7fd3d0b9c640 1 -- 172.21.15.1:0/623149815 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc009a10 con 0x7fd3b4004800 2024-05-09T05:20:50.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.156+0000 7fd3d3e02640 1 -- 172.21.15.1:0/623149815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b4004800 msgr2=0x7fd3b4004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:50.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.156+0000 7fd3d3e02640 1 --2- 172.21.15.1:0/623149815 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3b4004800 0x7fd3b4004c10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009e40 tx=0x7fd3b4005830 comp rx=0 tx=0).stop 2024-05-09T05:20:50.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.156+0000 7fd3d239f640 1 -- 172.21.15.1:0/623149815 reap_dead start 2024-05-09T05:20:50.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.156+0000 7fd3d3e02640 1 -- 172.21.15.1:0/623149815 shutdown_connections 2024-05-09T05:20:50.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.156+0000 7fd3d3e02640 1 -- 172.21.15.1:0/623149815 >> 172.21.15.1:0/623149815 conn(0x7fd3cc0745a0 msgr2=0x7fd3cc071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:50.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.157+0000 7fd3d3e02640 1 -- 172.21.15.1:0/623149815 shutdown_connections 2024-05-09T05:20:50.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.157+0000 7fd3d3e02640 1 -- 172.21.15.1:0/623149815 wait complete. 2024-05-09T05:20:50.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.159+0000 7fd3d3e02640 1 Processor -- start 2024-05-09T05:20:50.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.159+0000 7fd3d3e02640 1 -- start start 2024-05-09T05:20:50.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d3e02640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc1f8120 0x7fd3cc15c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d3e02640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3cc147180 0x7fd3cc1fa9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d1b9e640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc1f8120 msgr2=0x7fd3cc15c500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:20:50.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d3e02640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3cc0ca420 0x7fd3cc0ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d1b9e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc1f8120 0x7fd3cc15c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:50.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d3e02640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc0c70c0 con 0x7fd3cc147180 2024-05-09T05:20:50.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d3e02640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd3cc15a010 con 0x7fd3cc0ca420 2024-05-09T05:20:50.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d3e02640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd3cc0c7240 con 0x7fd3cc1f8120 2024-05-09T05:20:50.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.161+0000 7fd3d139d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3cc147180 0x7fd3cc1fa9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:50.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.162+0000 7fd3d139d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3cc147180 0x7fd3cc1fa9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60968/0 (socket says 172.21.15.1:60968) 2024-05-09T05:20:50.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.162+0000 7fd3d139d640 1 -- 172.21.15.1:0/4210836928 learned_addr learned my addr 172.21.15.1:0/4210836928 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:50.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.162+0000 7fd3d239f640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3cc0ca420 0x7fd3cc0ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:50.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.162+0000 7fd3d139d640 1 -- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc1f8120 msgr2=0x7fd3cc15c500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:20:50.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.162+0000 7fd3d139d640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc1f8120 0x7fd3cc15c500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.162+0000 7fd3d139d640 1 -- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3cc0ca420 msgr2=0x7fd3cc0ca830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:50.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.162+0000 7fd3d139d640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3cc0ca420 0x7fd3cc0ca830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.162+0000 7fd3d139d640 1 -- 172.21.15.1:0/4210836928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3cc147590 con 0x7fd3cc147180 2024-05-09T05:20:50.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.163+0000 7fd3d139d640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3cc147180 0x7fd3cc1fa9e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009f60 tx=0x7fd3bc004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:50.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.163+0000 7fd3ca7fc640 1 -- 172.21.15.1:0/4210836928 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc034030 con 0x7fd3cc147180 2024-05-09T05:20:50.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.163+0000 7fd3ca7fc640 1 -- 172.21.15.1:0/4210836928 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd3bc030460 con 0x7fd3cc147180 2024-05-09T05:20:50.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.163+0000 7fd3ca7fc640 1 -- 172.21.15.1:0/4210836928 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3bc030850 con 0x7fd3cc147180 2024-05-09T05:20:50.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.163+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3cc151290 con 0x7fd3cc147180 2024-05-09T05:20:50.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.164+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3cc1f7640 con 0x7fd3cc147180 2024-05-09T05:20:50.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.166+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3cc1f7430 con 0x7fd3cc147180 2024-05-09T05:20:50.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.166+0000 7fd3ca7fc640 1 -- 172.21.15.1:0/4210836928 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd3bc037020 con 0x7fd3cc147180 2024-05-09T05:20:50.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.167+0000 7fd3ca7fc640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3b80762d0 0x7fd3b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:50.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.167+0000 7fd3ca7fc640 1 -- 172.21.15.1:0/4210836928 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd3bc0b8500 con 0x7fd3cc147180 2024-05-09T05:20:50.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.168+0000 7fd3d1b9e640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3b80762d0 0x7fd3b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:50.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.169+0000 7fd3d1b9e640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3b80762d0 0x7fd3b80787e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0003900 tx=0x7fd3c00090e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:50.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.176+0000 7fd3ca7fc640 1 -- 172.21.15.1:0/4210836928 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd3bc008030 con 0x7fd3cc147180 2024-05-09T05:20:50.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.679+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd3cc050870 con 0x7fd3cc147180 2024-05-09T05:20:50.678 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.680+0000 7fd3ca7fc640 1 -- 172.21.15.1:0/4210836928 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fd3bc081e50 con 0x7fd3cc147180 2024-05-09T05:20:50.678 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":13,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:20:50.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.683+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3b80762d0 msgr2=0x7fd3b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.683+0000 7fd3d3e02640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3b80762d0 0x7fd3b80787e0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fd3c0003900 tx=0x7fd3c00090e0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3cc147180 msgr2=0x7fd3cc1fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3cc147180 0x7fd3cc1fa9e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fd3bc009f60 tx=0x7fd3bc004b50 comp rx=0 tx=0).stop 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 shutdown_connections 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd3b80762d0 0x7fd3b80787e0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd3cc0ca420 0x7fd3cc0ca830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd3cc147180 0x7fd3cc1fa9e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 --2- 172.21.15.1:0/4210836928 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd3cc1f8120 0x7fd3cc15c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 >> 172.21.15.1:0/4210836928 conn(0x7fd3cc0745a0 msgr2=0x7fd3cc0cd7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 shutdown_connections 2024-05-09T05:20:50.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:50.684+0000 7fd3d3e02640 1 -- 172.21.15.1:0/4210836928 wait complete. 2024-05-09T05:20:53.703 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:20:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.889+0000 7fc991b40640 1 Processor -- start 2024-05-09T05:20:53.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.889+0000 7fc991b40640 1 -- start start 2024-05-09T05:20:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc991b40640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc98c0c9d10 0x7fc98c0716c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:53.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc991b40640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 0x7fc98c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc991b40640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc98c1594d0 con 0x7fc98c070dd0 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc991b40640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc98c159da0 con 0x7fc98c0c9d10 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc991b40640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc98c070740 con 0x7fc98c0c5960 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc98b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc98c0c9d10 0x7fc98c0716c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc98b7fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc98c0c9d10 0x7fc98c0716c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60972/0 (socket says 172.21.15.1:60972) 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc98b7fe640 1 -- 172.21.15.1:0/2171131949 learned_addr learned my addr 172.21.15.1:0/2171131949 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc98affd640 1 -- 172.21.15.1:0/2171131949 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 msgr2=0x7fc98c06da90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.891+0000 7fc98affd640 1 --2- 172.21.15.1:0/2171131949 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 0x7fc98c06da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.892+0000 7fc98b7fe640 1 -- 172.21.15.1:0/2171131949 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 msgr2=0x7fc98c06da90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.892+0000 7fc98b7fe640 1 --2- 172.21.15.1:0/2171131949 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 0x7fc98c06da90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.892+0000 7fc98b7fe640 1 -- 172.21.15.1:0/2171131949 >> v1:172.21.15.1:6789/0 conn(0x7fc98c070dd0 legacy=0x7fc98c1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:53.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.892+0000 7fc98b7fe640 1 -- 172.21.15.1:0/2171131949 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98c070a20 con 0x7fc98c0c9d10 2024-05-09T05:20:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.893+0000 7fc98b7fe640 1 --2- 172.21.15.1:0/2171131949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0716c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc980009870 tx=0x7fc98002a310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b0e84ad4017800f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.894+0000 7fc98a7fc640 1 -- 172.21.15.1:0/2171131949 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc980037070 con 0x7fc98c0c9d10 2024-05-09T05:20:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.894+0000 7fc98a7fc640 1 -- 172.21.15.1:0/2171131949 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc980032640 con 0x7fc98c0c9d10 2024-05-09T05:20:53.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.894+0000 7fc98a7fc640 1 -- 172.21.15.1:0/2171131949 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc98002ae60 con 0x7fc98c0c9d10 2024-05-09T05:20:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.895+0000 7fc991b40640 1 -- 172.21.15.1:0/2171131949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 msgr2=0x7fc98c0716c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:53.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.895+0000 7fc991b40640 1 --2- 172.21.15.1:0/2171131949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0716c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc980009870 tx=0x7fc98002a310 comp rx=0 tx=0).stop 2024-05-09T05:20:53.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.895+0000 7fc991b40640 1 -- 172.21.15.1:0/2171131949 shutdown_connections 2024-05-09T05:20:53.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.895+0000 7fc991b40640 1 --2- 172.21.15.1:0/2171131949 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 0x7fc98c06da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:53.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.895+0000 7fc991b40640 1 --2- 172.21.15.1:0/2171131949 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0716c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:53.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.895+0000 7fc991b40640 1 -- 172.21.15.1:0/2171131949 >> 172.21.15.1:0/2171131949 conn(0x7fc98c0cec70 msgr2=0x7fc98c0cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:53.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.896+0000 7fc991b40640 1 -- 172.21.15.1:0/2171131949 shutdown_connections 2024-05-09T05:20:53.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.896+0000 7fc991b40640 1 -- 172.21.15.1:0/2171131949 wait complete. 2024-05-09T05:20:53.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.899+0000 7fc991b40640 1 Processor -- start 2024-05-09T05:20:53.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.899+0000 7fc991b40640 1 -- start start 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.900+0000 7fc991b40640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc98c070dd0 0x7fc98c1f3980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.900+0000 7fc991b40640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 0x7fc98c141500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.900+0000 7fc991b40640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0c5e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.900+0000 7fc98affd640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 msgr2=0x7fc98c141500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.900+0000 7fc98affd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 0x7fc98c141500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.900+0000 7fc991b40640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c071c00 con 0x7fc98c0c9d10 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.900+0000 7fc991b40640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc98c159a30 con 0x7fc98c070dd0 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.900+0000 7fc991b40640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc98c06dfd0 con 0x7fc98c0c5960 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0c5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0c5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60984/0 (socket says 172.21.15.1:60984) 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98bfff640 1 -- 172.21.15.1:0/3507033303 learned_addr learned my addr 172.21.15.1:0/3507033303 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:53.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98b7fe640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc98c070dd0 0x7fc98c1f3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98bfff640 1 -- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 msgr2=0x7fc98c141500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98bfff640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 0x7fc98c141500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98bfff640 1 -- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc98c070dd0 msgr2=0x7fc98c1f3980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98bfff640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc98c070dd0 0x7fc98c1f3980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.901+0000 7fc98bfff640 1 -- 172.21.15.1:0/3507033303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc98c1f3ec0 con 0x7fc98c0c9d10 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.902+0000 7fc98bfff640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0c5e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc98400c760 tx=0x7fc98400cb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.902+0000 7fc973fff640 1 -- 172.21.15.1:0/3507033303 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984003b70 con 0x7fc98c0c9d10 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.902+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc98c1f6ec0 con 0x7fc98c0c9d10 2024-05-09T05:20:53.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.903+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc98c1f7480 con 0x7fc98c0c9d10 2024-05-09T05:20:53.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.903+0000 7fc973fff640 1 -- 172.21.15.1:0/3507033303 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc984003d10 con 0x7fc98c0c9d10 2024-05-09T05:20:53.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.905+0000 7fc973fff640 1 -- 172.21.15.1:0/3507033303 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9840052d0 con 0x7fc98c0c9d10 2024-05-09T05:20:53.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.906+0000 7fc973fff640 1 -- 172.21.15.1:0/3507033303 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc9840054b0 con 0x7fc98c0c9d10 2024-05-09T05:20:53.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.907+0000 7fc973fff640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9780762d0 0x7fc9780787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:53.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.907+0000 7fc973fff640 1 -- 172.21.15.1:0/3507033303 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc9840974d0 con 0x7fc98c0c9d10 2024-05-09T05:20:53.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.907+0000 7fc98b7fe640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9780762d0 0x7fc9780787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:53.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.907+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc98c158380 con 0x7fc98c0c9d10 2024-05-09T05:20:53.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.908+0000 7fc98b7fe640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9780762d0 0x7fc9780787e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc980034000 tx=0x7fc98002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:53.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:53.918+0000 7fc973fff640 1 -- 172.21.15.1:0/3507033303 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc98c158380 con 0x7fc98c0c9d10 2024-05-09T05:20:54.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.409+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc98c1590c0 con 0x7fc98c0c9d10 2024-05-09T05:20:54.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.412+0000 7fc973fff640 1 -- 172.21.15.1:0/3507033303 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fc98c1590c0 con 0x7fc98c0c9d10 2024-05-09T05:20:54.410 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:20:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9780762d0 msgr2=0x7fc9780787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9780762d0 0x7fc9780787e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fc980034000 tx=0x7fc98002e000 comp rx=0 tx=0).stop 2024-05-09T05:20:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 msgr2=0x7fc98c0c5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0c5e90 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc98400c760 tx=0x7fc98400cb60 comp rx=0 tx=0).stop 2024-05-09T05:20:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 shutdown_connections 2024-05-09T05:20:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc9780762d0 0x7fc9780787e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc98c0c9d10 0x7fc98c0c5e90 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:54.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc98c0c5960 0x7fc98c141500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 --2- 172.21.15.1:0/3507033303 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc98c070dd0 0x7fc98c1f3980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 >> 172.21.15.1:0/3507033303 conn(0x7fc98c0cec70 msgr2=0x7fc98c157340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 shutdown_connections 2024-05-09T05:20:54.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:54.416+0000 7fc991b40640 1 -- 172.21.15.1:0/3507033303 wait complete. 2024-05-09T05:20:57.431 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:20:57.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.621+0000 7f4621270640 1 Processor -- start 2024-05-09T05:20:57.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.622+0000 7f4621270640 1 -- start start 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f4621270640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f461c06ffe0 0x7f461c0c4180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f4621270640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 0x7f461c0c46c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f4621270640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f461c06f7d0 con 0x7f461c070570 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f4621270640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f461c0c9230 con 0x7f461c06ffe0 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f4621270640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f461c159760 con 0x7f461c0c9e20 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f461b7fe640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 msgr2=0x7f461c0c46c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f461b7fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 0x7f461c0c46c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f461affd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f461c070570 0x7f461c0c9710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:36236/0 (socket says 172.21.15.1:36236) 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f461a7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f461c06ffe0 0x7f461c0c4180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:57.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.624+0000 7f461affd640 1 -- 172.21.15.1:0/1473080186 learned_addr learned my addr 172.21.15.1:0/1473080186 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:20:57.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.625+0000 7f461a7fc640 1 -- 172.21.15.1:0/1473080186 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 msgr2=0x7f461c0c46c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:20:57.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.625+0000 7f461a7fc640 1 --2- 172.21.15.1:0/1473080186 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 0x7f461c0c46c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:57.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.625+0000 7f461a7fc640 1 -- 172.21.15.1:0/1473080186 >> v1:172.21.15.196:6789/0 conn(0x7f461c070570 legacy=0x7f461c0c9710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:57.623 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.625+0000 7f461a7fc640 1 -- 172.21.15.1:0/1473080186 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c159a40 con 0x7f461c06ffe0 2024-05-09T05:20:57.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.626+0000 7f461a7fc640 1 --2- 172.21.15.1:0/1473080186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c06ffe0 0x7f461c0c4180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f461000a940 tx=0x7f461002c690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6260b42fec64894 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:57.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.626+0000 7f4619ffb640 1 -- 172.21.15.1:0/1473080186 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f461000b040 con 0x7f461c06ffe0 2024-05-09T05:20:57.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.627+0000 7f4619ffb640 1 -- 172.21.15.1:0/1473080186 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4610031710 con 0x7f461c06ffe0 2024-05-09T05:20:57.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.627+0000 7f4619ffb640 1 -- 172.21.15.1:0/1473080186 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4610031ac0 con 0x7f461c06ffe0 2024-05-09T05:20:57.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.628+0000 7f4621270640 1 -- 172.21.15.1:0/1473080186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c06ffe0 msgr2=0x7f461c0c4180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:57.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.628+0000 7f4621270640 1 --2- 172.21.15.1:0/1473080186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c06ffe0 0x7f461c0c4180 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f461000a940 tx=0x7f461002c690 comp rx=0 tx=0).stop 2024-05-09T05:20:57.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.628+0000 7f4621270640 1 -- 172.21.15.1:0/1473080186 shutdown_connections 2024-05-09T05:20:57.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.628+0000 7f4621270640 1 --2- 172.21.15.1:0/1473080186 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 0x7f461c0c46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:57.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.628+0000 7f4621270640 1 --2- 172.21.15.1:0/1473080186 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c06ffe0 0x7f461c0c4180 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:57.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.628+0000 7f4621270640 1 -- 172.21.15.1:0/1473080186 >> 172.21.15.1:0/1473080186 conn(0x7f461c0ce290 msgr2=0x7f461c0ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:57.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.628+0000 7f4621270640 1 -- 172.21.15.1:0/1473080186 shutdown_connections 2024-05-09T05:20:57.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.628+0000 7f4621270640 1 -- 172.21.15.1:0/1473080186 wait complete. 2024-05-09T05:20:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.634+0000 7f4621270640 1 Processor -- start 2024-05-09T05:20:57.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.634+0000 7f4621270640 1 -- start start 2024-05-09T05:20:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f4621270640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c070570 0x7f461c1f0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f4621270640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 0x7f461c1f1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f4621270640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f461c1f22b0 0x7f461c148540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:57.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f461a7fc640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 msgr2=0x7f461c1f1d70 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f461a7fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 0x7f461c1f1d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f461affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c070570 0x7f461c1f0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f461affd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c070570 0x7f461c1f0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60994/0 (socket says 172.21.15.1:60994) 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f461affd640 1 -- 172.21.15.1:0/2255800220 learned_addr learned my addr 172.21.15.1:0/2255800220 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f461c06fa40 con 0x7f461c070570 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f461c06f4f0 con 0x7f461c1f22b0 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.636+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f461c06fbc0 con 0x7f461c0c9e20 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.637+0000 7f461b7fe640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f461c1f22b0 0x7f461c148540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.637+0000 7f461affd640 1 -- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 msgr2=0x7f461c1f1d70 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.637+0000 7f461affd640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 0x7f461c1f1d70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:57.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.637+0000 7f461affd640 1 -- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f461c1f22b0 msgr2=0x7f461c148540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:57.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.637+0000 7f461affd640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f461c1f22b0 0x7f461c148540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:57.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.637+0000 7f461affd640 1 -- 172.21.15.1:0/2255800220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f461c14f960 con 0x7f461c070570 2024-05-09T05:20:57.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.637+0000 7f461affd640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c070570 0x7f461c1f0bd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f460c0016c0 tx=0x7f460c001b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:57.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.637+0000 7f46037fe640 1 -- 172.21.15.1:0/2255800220 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c00e280 con 0x7f461c070570 2024-05-09T05:20:57.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.638+0000 7f46037fe640 1 -- 172.21.15.1:0/2255800220 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f460c005870 con 0x7f461c070570 2024-05-09T05:20:57.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.638+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f461c1f1110 con 0x7f461c070570 2024-05-09T05:20:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.638+0000 7f46037fe640 1 -- 172.21.15.1:0/2255800220 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f460c005c40 con 0x7f461c070570 2024-05-09T05:20:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.638+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f461c1efe10 con 0x7f461c070570 2024-05-09T05:20:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.640+0000 7f46037fe640 1 -- 172.21.15.1:0/2255800220 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f460c007070 con 0x7f461c070570 2024-05-09T05:20:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.641+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f461c06ec60 con 0x7f461c070570 2024-05-09T05:20:57.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.641+0000 7f46037fe640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4608076280 0x7f4608078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:20:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.641+0000 7f46037fe640 1 -- 172.21.15.1:0/2255800220 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f460c09afb0 con 0x7f461c070570 2024-05-09T05:20:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.641+0000 7f461a7fc640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4608076280 0x7f4608078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:20:57.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.642+0000 7f461a7fc640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4608076280 0x7f4608078790 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4610036000 tx=0x7f4610031000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:20:57.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:57.649+0000 7f46037fe640 1 -- 172.21.15.1:0/2255800220 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f461c06ec60 con 0x7f461c070570 2024-05-09T05:20:58.143 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":114,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:20:58.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.143+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f461c0c8660 con 0x7f461c070570 2024-05-09T05:20:58.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.144+0000 7f46037fe640 1 -- 172.21.15.1:0/2255800220 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f460c064890 con 0x7f461c070570 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.148+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4608076280 msgr2=0x7f4608078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.148+0000 7f4621270640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4608076280 0x7f4608078790 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f4610036000 tx=0x7f4610031000 comp rx=0 tx=0).stop 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.148+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c070570 msgr2=0x7f461c1f0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.148+0000 7f4621270640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c070570 0x7f461c1f0bd0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f460c0016c0 tx=0x7f460c001b80 comp rx=0 tx=0).stop 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.148+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 shutdown_connections 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.149+0000 7f4621270640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f4608076280 0x7f4608078790 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.149+0000 7f4621270640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f461c1f22b0 0x7f461c148540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.149+0000 7f4621270640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f461c0c9e20 0x7f461c1f1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:58.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.149+0000 7f4621270640 1 --2- 172.21.15.1:0/2255800220 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f461c070570 0x7f461c1f0bd0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:20:58.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.149+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 >> 172.21.15.1:0/2255800220 conn(0x7f461c0ce290 msgr2=0x7f461c150830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:20:58.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.150+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 shutdown_connections 2024-05-09T05:20:58.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:20:58.150+0000 7f4621270640 1 -- 172.21.15.1:0/2255800220 wait complete. 2024-05-09T05:21:01.168 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:21:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.351+0000 7fd7e0984640 1 Processor -- start 2024-05-09T05:21:01.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.351+0000 7fd7e0984640 1 -- start start 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.353+0000 7fd7e0984640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.353+0000 7fd7e0984640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7dc0c7750 con 0x7fd7dc0c5ca0 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.353+0000 7fd7e0984640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd7dc160d70 con 0x7fd7dc06cc30 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.353+0000 7fd7e0984640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fd7dc0c8d40 con 0x7fd7dc15b270 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.354+0000 7fd7d9d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.354+0000 7fd7d9d9b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50030/0 (socket says 172.21.15.1:50030) 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.354+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 learned_addr learned my addr 172.21.15.1:0/813549232 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.354+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.354+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.354+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:01.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.354+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:01.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.555+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:01.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.555+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:21:01.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.555+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:01.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.555+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:01.552 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.555+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:21:01.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.956+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:01.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.956+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:21:01.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.956+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:01.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.956+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:01.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:01.956+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:21:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:02.757+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:02.754 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:02.758+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:21:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:02.758+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:02.758+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:02.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:02.758+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:21:03.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.994+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3749606722 0 0) 0x7fd7dc160d70 con 0x7fd7dc06cc30 2024-05-09T05:21:03.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.994+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd7c4003520 con 0x7fd7dc06cc30 2024-05-09T05:21:03.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.994+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2637889619 0 0) 0x7fd7c4003520 con 0x7fd7dc06cc30 2024-05-09T05:21:03.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.994+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 msgr2=0x7fd7dc06c6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.994+0000 7fd7d959a640 1 --2- 172.21.15.1:0/813549232 >> v2:172.21.15.1:3300/0 conn(0x7fd7dc15b270 0x7fd7dc06c6f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.994+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 >> v1:172.21.15.1:6789/0 conn(0x7fd7dc0c5ca0 legacy=0x7fd7dc0cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc0c79c0 con 0x7fd7dc06cc30 2024-05-09T05:21:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7fd7d0001620 con 0x7fd7dc06cc30 2024-05-09T05:21:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 >> v1:172.21.15.196:6789/0 conn(0x7fd7dc06cc30 legacy=0x7fd7dc161270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d959a640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4003a50 0x7fd7c4003e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:03.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d959a640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7c4004800 0x7fd7c4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d959a640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c4005350 0x7fd7c400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7da59c640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4003a50 0x7fd7c4003e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d9d9b640 1 -- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7c4004800 msgr2=0x7fd7c4004c10 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7c4004800 0x7fd7c4004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc0c8d40 con 0x7fd7c4005350 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.995+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd7c400c2a0 con 0x7fd7c4003a50 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.996+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd7c400c4a0 con 0x7fd7c4004800 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.996+0000 7fd7da59c640 1 -- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7c4004800 msgr2=0x7fd7c4004c10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:03.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.996+0000 7fd7da59c640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7c4004800 0x7fd7c4004c10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.996+0000 7fd7da59c640 1 -- 172.21.15.1:0/813549232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c4005350 msgr2=0x7fd7c400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.996+0000 7fd7da59c640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7c4005350 0x7fd7c400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.996+0000 7fd7dad9d640 1 -- 172.21.15.1:0/813549232 reap_dead start 2024-05-09T05:21:03.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.996+0000 7fd7da59c640 1 -- 172.21.15.1:0/813549232 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7c4003520 con 0x7fd7c4003a50 2024-05-09T05:21:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.998+0000 7fd7da59c640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4003a50 0x7fd7c4003e80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4001e90 tx=0x7fd7c4005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=812072ee6e3352ca server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.998+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4001ab0 con 0x7fd7c4003a50 2024-05-09T05:21:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.998+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d402ae10 con 0x7fd7c4003a50 2024-05-09T05:21:03.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.998+0000 7fd7d959a640 1 -- 172.21.15.1:0/813549232 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d4003b70 con 0x7fd7c4003a50 2024-05-09T05:21:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.999+0000 7fd7e0984640 1 -- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4003a50 msgr2=0x7fd7c4003e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:03.999+0000 7fd7e0984640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4003a50 0x7fd7c4003e80 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4001e90 tx=0x7fd7c4005830 comp rx=0 tx=0).stop 2024-05-09T05:21:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.000+0000 7fd7e0984640 1 -- 172.21.15.1:0/813549232 shutdown_connections 2024-05-09T05:21:03.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.000+0000 7fd7e0984640 1 --2- 172.21.15.1:0/813549232 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7c4003a50 0x7fd7c4003e80 secure :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4001e90 tx=0x7fd7c4005830 comp rx=0 tx=0).stop 2024-05-09T05:21:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.000+0000 7fd7e0984640 1 -- 172.21.15.1:0/813549232 >> 172.21.15.1:0/813549232 conn(0x7fd7dc0745a0 msgr2=0x7fd7dc071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.000+0000 7fd7e0984640 1 -- 172.21.15.1:0/813549232 shutdown_connections 2024-05-09T05:21:03.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.000+0000 7fd7e0984640 1 -- 172.21.15.1:0/813549232 wait complete. 2024-05-09T05:21:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.008+0000 7fd7e0984640 1 Processor -- start 2024-05-09T05:21:04.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.008+0000 7fd7e0984640 1 -- start start 2024-05-09T05:21:04.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.010+0000 7fd7e0984640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7dc0c5370 0x7fd7dc1eca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.010+0000 7fd7e0984640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7dc1f6f30 0x7fd7dc154bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.010+0000 7fd7e0984640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7dc1fae40 0x7fd7dc155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.010+0000 7fd7e0984640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc0c8260 con 0x7fd7dc1fae40 2024-05-09T05:21:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.010+0000 7fd7e0984640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd7dc0c7f80 con 0x7fd7dc0c5370 2024-05-09T05:21:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.010+0000 7fd7e0984640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd7dc06c6f0 con 0x7fd7dc1f6f30 2024-05-09T05:21:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7d9d9b640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7dc1f6f30 msgr2=0x7fd7dc154bf0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:21:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7d9d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7dc1f6f30 0x7fd7dc154bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:04.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7da59c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7dc0c5370 0x7fd7dc1eca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7da59c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7dc0c5370 0x7fd7dc1eca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:53256/0 (socket says 172.21.15.1:53256) 2024-05-09T05:21:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7da59c640 1 -- 172.21.15.1:0/1471800893 learned_addr learned my addr 172.21.15.1:0/1471800893 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7da59c640 1 -- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7dc1f6f30 msgr2=0x7fd7dc154bf0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:04.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7da59c640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7dc1f6f30 0x7fd7dc154bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7da59c640 1 -- 172.21.15.1:0/1471800893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7dc1fae40 msgr2=0x7fd7dc155d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7da59c640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7dc1fae40 0x7fd7dc155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.011+0000 7fd7da59c640 1 -- 172.21.15.1:0/1471800893 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7dc1553c0 con 0x7fd7dc0c5370 2024-05-09T05:21:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.012+0000 7fd7da59c640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7dc0c5370 0x7fd7dc1eca70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4001e90 tx=0x7fd7d40014a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:04.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.013+0000 7fd7caffd640 1 -- 172.21.15.1:0/1471800893 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d403f030 con 0x7fd7dc0c5370 2024-05-09T05:21:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.013+0000 7fd7caffd640 1 -- 172.21.15.1:0/1471800893 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd7d402ac10 con 0x7fd7dc0c5370 2024-05-09T05:21:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.013+0000 7fd7caffd640 1 -- 172.21.15.1:0/1471800893 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd7d40050d0 con 0x7fd7dc0c5370 2024-05-09T05:21:04.011 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.013+0000 7fd7e0984640 1 -- 172.21.15.1:0/1471800893 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7dc1556a0 con 0x7fd7dc0c5370 2024-05-09T05:21:04.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.015+0000 7fd7e0984640 1 -- 172.21.15.1:0/1471800893 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7dc1fb250 con 0x7fd7dc0c5370 2024-05-09T05:21:04.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.020+0000 7fd7caffd640 1 -- 172.21.15.1:0/1471800893 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd7d4042070 con 0x7fd7dc0c5370 2024-05-09T05:21:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.021+0000 7fd7caffd640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd7bc0762d0 0x7fd7bc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.021+0000 7fd7caffd640 1 -- 172.21.15.1:0/1471800893 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd7d40b6ea0 con 0x7fd7dc0c5370 2024-05-09T05:21:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.021+0000 7fd7c8ff9640 1 -- 172.21.15.1:0/1471800893 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7dc050870 con 0x7fd7dc0c5370 2024-05-09T05:21:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.022+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd7bc0762d0 0x7fd7bc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.019 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.022+0000 7fd7d9d9b640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd7bc0762d0 0x7fd7bc0787e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0722f0 tx=0x7fd7cc008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:04.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.029+0000 7fd7caffd640 1 -- 172.21.15.1:0/1471800893 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fd7d4080760 con 0x7fd7dc0c5370 2024-05-09T05:21:04.530 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.532+0000 7fd7c8ff9640 1 -- 172.21.15.1:0/1471800893 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fd7dc06e710 con 0x7fd7dc0c5370 2024-05-09T05:21:04.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.535+0000 7fd7caffd640 1 -- 172.21.15.1:0/1471800893 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fd7d4080060 con 0x7fd7dc0c5370 2024-05-09T05:21:04.531 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":118,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:21:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 -- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd7bc0762d0 msgr2=0x7fd7bc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd7bc0762d0 0x7fd7bc0787e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fd7dc0722f0 tx=0x7fd7cc008040 comp rx=0 tx=0).stop 2024-05-09T05:21:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 -- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7dc0c5370 msgr2=0x7fd7dc1eca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7dc0c5370 0x7fd7dc1eca70 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fd7d4001e90 tx=0x7fd7d40014a0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 -- 172.21.15.1:0/1471800893 shutdown_connections 2024-05-09T05:21:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd7bc0762d0 0x7fd7bc0787e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd7dc1fae40 0x7fd7dc155d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd7dc1f6f30 0x7fd7dc154bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 --2- 172.21.15.1:0/1471800893 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd7dc0c5370 0x7fd7dc1eca70 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 -- 172.21.15.1:0/1471800893 >> 172.21.15.1:0/1471800893 conn(0x7fd7dc0745a0 msgr2=0x7fd7dc0ca620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 -- 172.21.15.1:0/1471800893 shutdown_connections 2024-05-09T05:21:04.536 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.538+0000 7fd7c8ff9640 1 -- 172.21.15.1:0/1471800893 wait complete. 2024-05-09T05:21:04.554 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:21:04.554 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:21:04.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:21:04.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.745+0000 7fd1cd134640 1 Processor -- start 2024-05-09T05:21:04.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.745+0000 7fd1cd134640 1 -- start start 2024-05-09T05:21:04.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.748+0000 7fd1cd134640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fd1c813e620 0x7fd1c8159410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.748+0000 7fd1cd134640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80734e0 0x7fd1c80c7100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.748+0000 7fd1cd134640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fd1c813ea30 con 0x7fd1c80730d0 2024-05-09T05:21:04.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.748+0000 7fd1cd134640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fd1c80c9320 con 0x7fd1c813e620 2024-05-09T05:21:04.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.748+0000 7fd1cd134640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c80c84c0 con 0x7fd1c80734e0 2024-05-09T05:21:04.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.748+0000 7fd1c659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fd1c80730d0 0x7fd1c80c4e00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57422/0 (socket says 172.21.15.1:57422) 2024-05-09T05:21:04.745 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.748+0000 7fd1c659c640 1 -- 172.21.15.1:0/1710226639 learned_addr learned my addr 172.21.15.1:0/1710226639 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:21:04.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.749+0000 7fd1c759e640 1 -- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80734e0 msgr2=0x7fd1c80c7100 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:21:04.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.749+0000 7fd1c759e640 1 --2- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80734e0 0x7fd1c80c7100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:04.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.749+0000 7fd1c5d9b640 1 -- 172.21.15.1:0/1710226639 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2275846357 0 0) 0x7fd1c813ea30 con 0x7fd1c80730d0 2024-05-09T05:21:04.746 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.749+0000 7fd1c5d9b640 1 -- 172.21.15.1:0/1710226639 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fd1b0003520 con 0x7fd1c80730d0 2024-05-09T05:21:04.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.750+0000 7fd1c6d9d640 1 --2- 172.21.15.1:0/1710226639 >> v2:172.21.15.196:3300/0 conn(0x7fd1c813e620 0x7fd1c8159410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.750+0000 7fd1c6d9d640 1 -- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80734e0 msgr2=0x7fd1c80c7100 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:04.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.750+0000 7fd1c6d9d640 1 --2- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80734e0 0x7fd1c80c7100 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.750+0000 7fd1c6d9d640 1 -- 172.21.15.1:0/1710226639 >> v1:172.21.15.1:6789/0 conn(0x7fd1c80730d0 legacy=0x7fd1c80c4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.747 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.750+0000 7fd1c6d9d640 1 -- 172.21.15.1:0/1710226639 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c80c87a0 con 0x7fd1c813e620 2024-05-09T05:21:04.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.751+0000 7fd1c6d9d640 1 --2- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c813e620 0x7fd1c8159410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80097a0 tx=0x7fd1b802a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=12d4b255c8bde3d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:04.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.752+0000 7fd1c5d9b640 1 -- 172.21.15.1:0/1710226639 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8037070 con 0x7fd1c813e620 2024-05-09T05:21:04.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.752+0000 7fd1c5d9b640 1 -- 172.21.15.1:0/1710226639 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1b8032440 con 0x7fd1c813e620 2024-05-09T05:21:04.749 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.752+0000 7fd1c5d9b640 1 -- 172.21.15.1:0/1710226639 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8032810 con 0x7fd1c813e620 2024-05-09T05:21:04.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.753+0000 7fd1cd134640 1 -- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c813e620 msgr2=0x7fd1c8159410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.753+0000 7fd1cd134640 1 --2- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c813e620 0x7fd1c8159410 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fd1b80097a0 tx=0x7fd1b802a1a0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.751 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.753+0000 7fd1cd134640 1 -- 172.21.15.1:0/1710226639 shutdown_connections 2024-05-09T05:21:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.753+0000 7fd1cd134640 1 --2- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80734e0 0x7fd1c80c7100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.753+0000 7fd1cd134640 1 --2- 172.21.15.1:0/1710226639 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c813e620 0x7fd1c8159410 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.753+0000 7fd1cd134640 1 -- 172.21.15.1:0/1710226639 >> 172.21.15.1:0/1710226639 conn(0x7fd1c8070f40 msgr2=0x7fd1c806c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.753+0000 7fd1cd134640 1 -- 172.21.15.1:0/1710226639 shutdown_connections 2024-05-09T05:21:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.753+0000 7fd1cd134640 1 -- 172.21.15.1:0/1710226639 wait complete. 2024-05-09T05:21:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.756+0000 7fd1cd134640 1 Processor -- start 2024-05-09T05:21:04.752 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.756+0000 7fd1cd134640 1 -- start start 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1cd134640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80730d0 0x7fd1c815c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1cd134640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c813e620 0x7fd1c8155ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1cd134640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c81f1510 0x7fd1c81517c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1cd134640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c80c9040 con 0x7fd1c813e620 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1cd134640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c80c8d60 con 0x7fd1c81f1510 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1cd134640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fd1c80c5510 con 0x7fd1c80730d0 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1c6d9d640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80730d0 msgr2=0x7fd1c815c800 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1c759e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c81f1510 0x7fd1c81517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.755 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1c6d9d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80730d0 0x7fd1c815c800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:04.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1c759e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c81f1510 0x7fd1c81517c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:53272/0 (socket says 172.21.15.1:53272) 2024-05-09T05:21:04.756 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.758+0000 7fd1c759e640 1 -- 172.21.15.1:0/20721647 learned_addr learned my addr 172.21.15.1:0/20721647 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:04.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1c659c640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c813e620 0x7fd1c8155ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1c759e640 1 -- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80730d0 msgr2=0x7fd1c815c800 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:04.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1c759e640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fd1c80730d0 0x7fd1c815c800 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.757 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1c759e640 1 -- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c813e620 msgr2=0x7fd1c8155ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1c759e640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c813e620 0x7fd1c8155ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1c759e640 1 -- 172.21.15.1:0/20721647 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1c81560e0 con 0x7fd1c81f1510 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1c659c640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c813e620 0x7fd1c8155ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1c759e640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c81f1510 0x7fd1c81517c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000bde0 tx=0x7fd1c000bed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.759+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c000caa0 con 0x7fd1c81f1510 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.760+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fd1c0010070 con 0x7fd1c81f1510 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.760+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1c000caa0 con 0x7fd1c81f1510 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.760+0000 7fd1cd134640 1 -- 172.21.15.1:0/20721647 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1c815a070 con 0x7fd1c81f1510 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.760+0000 7fd1cd134640 1 -- 172.21.15.1:0/20721647 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1c81ee550 con 0x7fd1c81f1510 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.761+0000 7fd1cd134640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81ee750 0x7fd1c8073c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.758 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.761+0000 7fd1cd134640 1 -- 172.21.15.1:0/20721647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c8155220 con 0x7fd1c81ee750 2024-05-09T05:21:04.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.762+0000 7fd1c6d9d640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81ee750 0x7fd1c8073c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.759 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.762+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fd1c0008030 con 0x7fd1c81f1510 2024-05-09T05:21:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.763+0000 7fd1c6d9d640 1 -- 172.21.15.1:0/20721647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fd1c8159de0 con 0x7fd1c81ee750 2024-05-09T05:21:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.763+0000 7fd1af7fe640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd1b40766b0 0x7fd1b4078bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.763+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fd1c000c8a0 con 0x7fd1c81f1510 2024-05-09T05:21:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.764+0000 7fd1c659c640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd1b40766b0 0x7fd1b4078bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.761 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.764+0000 7fd1c6d9d640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81ee750 0x7fd1c8073c10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80c4cd0 tx=0x7fd1b80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:04.762 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.766+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8042020 con 0x7fd1c81ee750 2024-05-09T05:21:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.766+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fd1b80093c0 con 0x7fd1c81ee750 2024-05-09T05:21:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.766+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81ee750 msgr2=0x7fd1c8073c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.766+0000 7fd1af7fe640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81ee750 0x7fd1c8073c10 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80c4cd0 tx=0x7fd1b80046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.763 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.766+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81ee750 msgr2=0x7fd1c8073c10 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:04.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.766+0000 7fd1af7fe640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81ee750 0x7fd1c8073c10 secure :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd1c80c4cd0 tx=0x7fd1b80046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.764 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.767+0000 7fd1c659c640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd1b40766b0 0x7fd1b4078bc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8152b00 tx=0x7fd1bc009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:04.774 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.778+0000 7fd1cd134640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81f1930 0x7fd1c8074150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.778+0000 7fd1cd134640 1 -- 172.21.15.1:0/20721647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fd1c8050870 con 0x7fd1c81f1930 2024-05-09T05:21:04.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.778+0000 7fd1c6d9d640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81f1930 0x7fd1c8074150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.777 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.780+0000 7fd1c6d9d640 1 -- 172.21.15.1:0/20721647 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fd1c813c6f0 con 0x7fd1c81f1930 2024-05-09T05:21:04.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.781+0000 7fd1c6d9d640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81f1930 0x7fd1c8074150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd1b802a1e0 tx=0x7fd1b8009190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:04.778 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.782+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1b8042020 con 0x7fd1c81f1930 2024-05-09T05:21:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.782+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5052 (secure 0 0 0) 0x7fd1b8003610 con 0x7fd1c81f1930 2024-05-09T05:21:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.782+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81f1930 msgr2=0x7fd1c8074150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.782+0000 7fd1af7fe640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81f1930 0x7fd1c8074150 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fd1b802a1e0 tx=0x7fd1b8009190 comp rx=0 tx=0).stop 2024-05-09T05:21:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.782+0000 7fd1af7fe640 1 -- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81f1930 msgr2=0x7fd1c8074150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:04.779 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.782+0000 7fd1af7fe640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fd1c81f1930 0x7fd1c8074150 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.779 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:21:04.779 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:04.779 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 53342, 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:04.780 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:04.781 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:04.782 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:04.782 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:04.782 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:04.782 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:04.782 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:04.782 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:04.782 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:04.782 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:04.783 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:04.783 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:04.783 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.783 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:21:04.783 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:21:04.783 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:21:04.783 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:21:04.783 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:04.784 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:21:04.785 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:04.785 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:04.785 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:04.785 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:04.785 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:04.785 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:04.785 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:04.786 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.787 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:04.788 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.789 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:21:04.790 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:04.791 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:04.792 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.793 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:04.793 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.793 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:21:04.793 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:21:04.793 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:04.794 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:04.794 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:04.794 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.794 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:04.794 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.794 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:21:04.795 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:21:04.795 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.795 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:04.795 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:04.795 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:04.795 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.795 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:04.795 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:21:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:21:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:21:04.796 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:21:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:21:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.797 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:21:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:04.798 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:21:04.799 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:21:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.783+0000 7fd1ad7fa640 1 -- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd1b40766b0 msgr2=0x7fd1b4078bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.783+0000 7fd1ad7fa640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fd1b40766b0 0x7fd1b4078bc0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fd1c8152b00 tx=0x7fd1bc009040 comp rx=0 tx=0).stop 2024-05-09T05:21:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.783+0000 7fd1ad7fa640 1 -- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c81f1510 msgr2=0x7fd1c81517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.783+0000 7fd1ad7fa640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c81f1510 0x7fd1c81517c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fd1c000bde0 tx=0x7fd1c000bed0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.783+0000 7fd1c759e640 1 -- 172.21.15.1:0/20721647 reap_dead start 2024-05-09T05:21:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.783+0000 7fd1ad7fa640 1 -- 172.21.15.1:0/20721647 shutdown_connections 2024-05-09T05:21:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.783+0000 7fd1ad7fa640 1 --2- 172.21.15.1:0/20721647 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fd1c81f1510 0x7fd1c81517c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.783+0000 7fd1ad7fa640 1 -- 172.21.15.1:0/20721647 >> 172.21.15.1:0/20721647 conn(0x7fd1c8070f40 msgr2=0x7fd1c813b480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.784+0000 7fd1ad7fa640 1 -- 172.21.15.1:0/20721647 shutdown_connections 2024-05-09T05:21:04.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.784+0000 7fd1ad7fa640 1 -- 172.21.15.1:0/20721647 wait complete. 2024-05-09T05:21:04.802 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:21:04.803 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon b 2024-05-09T05:21:04.803 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:21:04.987 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.991+0000 7efd2ded8640 1 Processor -- start 2024-05-09T05:21:04.988 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.991+0000 7efd2ded8640 1 -- start start 2024-05-09T05:21:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.993+0000 7efd2ded8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd280c81a0 0x7efd280c5600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.993+0000 7efd2ded8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efd280cbf60 0x7efd2806c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.993+0000 7efd2ded8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7efd280746d0 con 0x7efd280742c0 2024-05-09T05:21:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.993+0000 7efd2ded8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7efd281617b0 con 0x7efd280c81a0 2024-05-09T05:21:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.993+0000 7efd2ded8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7efd28161ce0 con 0x7efd280cbf60 2024-05-09T05:21:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.994+0000 7efd1ffff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efd280cbf60 0x7efd2806c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.994+0000 7efd1ffff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efd280cbf60 0x7efd2806c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:53278/0 (socket says 172.21.15.1:53278) 2024-05-09T05:21:04.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.994+0000 7efd1ffff640 1 -- 172.21.15.1:0/395659526 learned_addr learned my addr 172.21.15.1:0/395659526 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.995+0000 7efd277fe640 1 --2- 172.21.15.1:0/395659526 >> v2:172.21.15.1:3300/0 conn(0x7efd280c81a0 0x7efd280c5600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.996+0000 7efd1ffff640 1 -- 172.21.15.1:0/395659526 >> v2:172.21.15.1:3300/0 conn(0x7efd280c81a0 msgr2=0x7efd280c5600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.997+0000 7efd26ffd640 1 -- 172.21.15.1:0/395659526 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2572047950 0 0) 0x7efd280746d0 con 0x7efd280742c0 2024-05-09T05:21:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.997+0000 7efd1ffff640 1 --2- 172.21.15.1:0/395659526 >> v2:172.21.15.1:3300/0 conn(0x7efd280c81a0 0x7efd280c5600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.997+0000 7efd1ffff640 1 -- 172.21.15.1:0/395659526 >> v1:172.21.15.1:6789/0 conn(0x7efd280742c0 legacy=0x7efd280c70b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.997+0000 7efd1ffff640 1 -- 172.21.15.1:0/395659526 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd28161a00 con 0x7efd280cbf60 2024-05-09T05:21:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.997+0000 7efd277fe640 1 --2- 172.21.15.1:0/395659526 >> v2:172.21.15.1:3300/0 conn(0x7efd280c81a0 0x7efd280c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:21:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:04.998+0000 7efd1ffff640 1 --2- 172.21.15.1:0/395659526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280cbf60 0x7efd2806c6f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efd28161930 tx=0x7efd1002a2a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=548479c3b89bdbf3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.001+0000 7efd26ffd640 1 -- 172.21.15.1:0/395659526 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10037070 con 0x7efd280cbf60 2024-05-09T05:21:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.001+0000 7efd26ffd640 1 -- 172.21.15.1:0/395659526 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd10032440 con 0x7efd280cbf60 2024-05-09T05:21:04.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.001+0000 7efd26ffd640 1 -- 172.21.15.1:0/395659526 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10032810 con 0x7efd280cbf60 2024-05-09T05:21:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.002+0000 7efd2ded8640 1 -- 172.21.15.1:0/395659526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280cbf60 msgr2=0x7efd2806c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.002+0000 7efd2ded8640 1 --2- 172.21.15.1:0/395659526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280cbf60 0x7efd2806c6f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7efd28161930 tx=0x7efd1002a2a0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.002+0000 7efd2ded8640 1 -- 172.21.15.1:0/395659526 shutdown_connections 2024-05-09T05:21:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.002+0000 7efd2ded8640 1 --2- 172.21.15.1:0/395659526 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280cbf60 0x7efd2806c6f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.002+0000 7efd2ded8640 1 --2- 172.21.15.1:0/395659526 >> v2:172.21.15.1:3300/0 conn(0x7efd280c81a0 0x7efd280c5600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.002+0000 7efd2ded8640 1 -- 172.21.15.1:0/395659526 >> 172.21.15.1:0/395659526 conn(0x7efd280ce180 msgr2=0x7efd280ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.002+0000 7efd2ded8640 1 -- 172.21.15.1:0/395659526 shutdown_connections 2024-05-09T05:21:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.002+0000 7efd2ded8640 1 -- 172.21.15.1:0/395659526 wait complete. 2024-05-09T05:21:05.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.005+0000 7efd2ded8640 1 Processor -- start 2024-05-09T05:21:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.005+0000 7efd2ded8640 1 -- start start 2024-05-09T05:21:05.004 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.007+0000 7efd2ded8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd280742c0 0x7efd281e87d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:05.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd2ded8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280c81a0 0x7efd281ea6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:05.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd2ded8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd280cbf60 0x7efd2815bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:05.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd2ded8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efd28073da0 con 0x7efd280742c0 2024-05-09T05:21:05.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd277fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd280742c0 0x7efd281e87d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:05.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd2ded8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efd28073ac0 con 0x7efd280c81a0 2024-05-09T05:21:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd2ded8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efd280c77c0 con 0x7efd280cbf60 2024-05-09T05:21:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd27fff640 1 -- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd280cbf60 msgr2=0x7efd2815bfd0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3301/0 2024-05-09T05:21:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd1ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280c81a0 0x7efd281ea6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd27fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd280cbf60 0x7efd2815bfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd1ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280c81a0 0x7efd281ea6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:53290/0 (socket says 172.21.15.1:53290) 2024-05-09T05:21:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.008+0000 7efd1ffff640 1 -- 172.21.15.1:0/3062794691 learned_addr learned my addr 172.21.15.1:0/3062794691 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:05.007 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.009+0000 7efd1ffff640 1 -- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd280cbf60 msgr2=0x7efd2815bfd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.009+0000 7efd1ffff640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd280cbf60 0x7efd2815bfd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.009+0000 7efd1ffff640 1 -- 172.21.15.1:0/3062794691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd280742c0 msgr2=0x7efd281e87d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.010+0000 7efd1ffff640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd280742c0 0x7efd281e87d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.010+0000 7efd1ffff640 1 -- 172.21.15.1:0/3062794691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd281f6f50 con 0x7efd280c81a0 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.010+0000 7efd277fe640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd280742c0 0x7efd281e87d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.010+0000 7efd1ffff640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280c81a0 0x7efd281ea6b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efd100090b0 tx=0x7efd100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.011+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10041020 con 0x7efd280c81a0 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.011+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd1003d620 con 0x7efd280c81a0 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.011+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10037040 con 0x7efd280c81a0 2024-05-09T05:21:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.011+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd281f7230 con 0x7efd280c81a0 2024-05-09T05:21:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.011+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd281f77f0 con 0x7efd280c81a0 2024-05-09T05:21:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.013+0000 7efd2ded8640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd2815d530 0x7efd2815c510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.013+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efd2815dca0 con 0x7efd2815d530 2024-05-09T05:21:05.012 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.014+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efd10044020 con 0x7efd280c81a0 2024-05-09T05:21:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.014+0000 7efd277fe640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd2815d530 0x7efd2815c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.014+0000 7efd24ff9640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd08076300 0x7efd08078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.014+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7efd100b78b0 con 0x7efd280c81a0 2024-05-09T05:21:05.013 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.015+0000 7efd27fff640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd08076300 0x7efd08078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:05.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.016+0000 7efd277fe640 1 -- 172.21.15.1:0/3062794691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7efd28050870 con 0x7efd2815d530 2024-05-09T05:21:05.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.017+0000 7efd27fff640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd08076300 0x7efd08078810 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efd28157000 tx=0x7efd20007cd0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:05.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.017+0000 7efd277fe640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd2815d530 0x7efd2815c510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd18003980 tx=0x7efd18003e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:05.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.018+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18018070 con 0x7efd2815d530 2024-05-09T05:21:05.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.018+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7efd18010b70 con 0x7efd2815d530 2024-05-09T05:21:05.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.018+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd2815d530 msgr2=0x7efd2815c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:05.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.018+0000 7efd24ff9640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd2815d530 0x7efd2815c510 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd18003980 tx=0x7efd18003e40 comp rx=0 tx=0).stop 2024-05-09T05:21:05.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.018+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd2815d530 msgr2=0x7efd2815c510 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:05.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.018+0000 7efd24ff9640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd2815d530 0x7efd2815c510 secure :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7efd18003980 tx=0x7efd18003e40 comp rx=0 tx=0).stop 2024-05-09T05:21:05.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.032+0000 7efd2ded8640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd281e7770 0x7efd281fa3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:05.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.032+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efd280748f0 con 0x7efd281e7770 2024-05-09T05:21:05.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.033+0000 7efd277fe640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd281e7770 0x7efd281fa3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:05.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.034+0000 7efd277fe640 1 -- 172.21.15.1:0/3062794691 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7efd2806dba0 con 0x7efd281e7770 2024-05-09T05:21:05.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.034+0000 7efd277fe640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd281e7770 0x7efd281fa3b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efd18003980 tx=0x7efd18004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.034+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd18014630 con 0x7efd281e7770 2024-05-09T05:21:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.035+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5051 (secure 0 0 0) 0x7efd18014630 con 0x7efd281e7770 2024-05-09T05:21:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.035+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd281e7770 msgr2=0x7efd281fa3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.035+0000 7efd24ff9640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd281e7770 0x7efd281fa3b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efd18003980 tx=0x7efd18004730 comp rx=0 tx=0).stop 2024-05-09T05:21:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.035+0000 7efd24ff9640 1 -- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd281e7770 msgr2=0x7efd281fa3b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:05.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.035+0000 7efd24ff9640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd281e7770 0x7efd281fa3b0 secure :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7efd18003980 tx=0x7efd18004730 comp rx=0 tx=0).stop 2024-05-09T05:21:05.032 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:21:05.032 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:05.032 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:05.032 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:21:05.032 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 118, 2024-05-09T05:21:05.032 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 215438, 2024-05-09T05:21:05.032 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:21:05.032 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: 1 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:05.033 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:05.034 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:05.034 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:05.034 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:05.034 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:05.035 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:21:05.036 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:05.037 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:21:05.038 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:05.039 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:21:05.040 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:05.041 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:21:05.042 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.043 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.044 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 6 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:05.045 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:05.046 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:21:05.046 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:21:05.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd08076300 msgr2=0x7efd08078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:05.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd08076300 0x7efd08078810 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7efd28157000 tx=0x7efd20007cd0 comp rx=0 tx=0).stop 2024-05-09T05:21:05.048 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280c81a0 msgr2=0x7efd281ea6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:05.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280c81a0 0x7efd281ea6b0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7efd100090b0 tx=0x7efd100046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:05.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd27fff640 1 -- 172.21.15.1:0/3062794691 reap_dead start 2024-05-09T05:21:05.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 shutdown_connections 2024-05-09T05:21:05.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 --2- 172.21.15.1:0/3062794691 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd280c81a0 0x7efd281ea6b0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:05.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 >> 172.21.15.1:0/3062794691 conn(0x7efd280ce180 msgr2=0x7efd280c9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:05.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 shutdown_connections 2024-05-09T05:21:05.049 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:05.036+0000 7efd2ded8640 1 -- 172.21.15.1:0/3062794691 wait complete. 2024-05-09T05:21:05.051 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:21:05.051 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:21:05.581 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:21:05.583 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:21:25.052 INFO:tasks.mon_thrash.mon_thrasher:killing mon.c 2024-05-09T05:21:25.052 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.c 2024-05-09T05:21:25.052 INFO:tasks.ceph.mon.c:Restarting daemon 2024-05-09T05:21:25.053 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2024-05-09T05:21:25.056 INFO:tasks.ceph.mon.c:Started 2024-05-09T05:21:25.056 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:21:25.056 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:21:25.238 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.242+0000 7f1b57f5e640 1 Processor -- start 2024-05-09T05:21:25.239 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.242+0000 7f1b57f5e640 1 -- start start 2024-05-09T05:21:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.244+0000 7f1b57f5e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1b500cbef0 0x7f1b500744c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.244+0000 7f1b57f5e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b50152d50 0x7f1b500735c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.244+0000 7f1b57f5e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1b50074a00 con 0x7f1b50153160 2024-05-09T05:21:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.244+0000 7f1b57f5e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f1b500c5590 con 0x7f1b50152d50 2024-05-09T05:21:25.241 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.244+0000 7f1b57f5e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f1b500c7b20 con 0x7f1b500cbef0 2024-05-09T05:21:25.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b554f9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f1b50152d50 0x7f1b500735c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:25.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b564fb640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f1b50153160 0x7f1b500c5a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:52690/0 (socket says 172.21.15.1:52690) 2024-05-09T05:21:25.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b564fb640 1 -- 172.21.15.1:0/4274958871 learned_addr learned my addr 172.21.15.1:0/4274958871 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:21:25.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b55cfa640 1 --2- 172.21.15.1:0/4274958871 >> v2:172.21.15.196:3300/0 conn(0x7f1b500cbef0 0x7f1b500744c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:25.242 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b54cf8640 1 -- 172.21.15.1:0/4274958871 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2734111322 0 0) 0x7f1b50074a00 con 0x7f1b50153160 2024-05-09T05:21:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b554f9640 1 -- 172.21.15.1:0/4274958871 >> v2:172.21.15.196:3300/0 conn(0x7f1b500cbef0 msgr2=0x7f1b500744c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b554f9640 1 --2- 172.21.15.1:0/4274958871 >> v2:172.21.15.196:3300/0 conn(0x7f1b500cbef0 0x7f1b500744c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b554f9640 1 -- 172.21.15.1:0/4274958871 >> v1:172.21.15.196:6789/0 conn(0x7f1b50153160 legacy=0x7f1b500c5a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.245+0000 7f1b554f9640 1 -- 172.21.15.1:0/4274958871 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b500c7e00 con 0x7f1b50152d50 2024-05-09T05:21:25.243 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.246+0000 7f1b554f9640 1 --2- 172.21.15.1:0/4274958871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50152d50 0x7f1b500735c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b50072d90 tx=0x7f1b4002a250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57e83afd3f52684 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.247+0000 7f1b54cf8640 1 -- 172.21.15.1:0/4274958871 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40037070 con 0x7f1b50152d50 2024-05-09T05:21:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.247+0000 7f1b54cf8640 1 -- 172.21.15.1:0/4274958871 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b40032440 con 0x7f1b50152d50 2024-05-09T05:21:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.247+0000 7f1b54cf8640 1 -- 172.21.15.1:0/4274958871 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40032810 con 0x7f1b50152d50 2024-05-09T05:21:25.244 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.248+0000 7f1b57f5e640 1 -- 172.21.15.1:0/4274958871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50152d50 msgr2=0x7f1b500735c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.248+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/4274958871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50152d50 0x7f1b500735c0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f1b50072d90 tx=0x7f1b4002a250 comp rx=0 tx=0).stop 2024-05-09T05:21:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.248+0000 7f1b57f5e640 1 -- 172.21.15.1:0/4274958871 shutdown_connections 2024-05-09T05:21:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.248+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/4274958871 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50152d50 0x7f1b500735c0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.248+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/4274958871 >> v2:172.21.15.196:3300/0 conn(0x7f1b500cbef0 0x7f1b500744c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.248+0000 7f1b57f5e640 1 -- 172.21.15.1:0/4274958871 >> 172.21.15.1:0/4274958871 conn(0x7f1b500ce110 msgr2=0x7f1b500ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:25.245 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.249+0000 7f1b57f5e640 1 -- 172.21.15.1:0/4274958871 shutdown_connections 2024-05-09T05:21:25.246 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.249+0000 7f1b57f5e640 1 -- 172.21.15.1:0/4274958871 wait complete. 2024-05-09T05:21:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.252+0000 7f1b57f5e640 1 Processor -- start 2024-05-09T05:21:25.249 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.252+0000 7f1b57f5e640 1 -- start start 2024-05-09T05:21:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b57f5e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 0x7f1b501ee380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b57f5e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b50152d50 0x7f1b5014c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b55cfa640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 0x7f1b501ee380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b55cfa640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 0x7f1b501ee380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:54112/0 (socket says 172.21.15.1:54112) 2024-05-09T05:21:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b55cfa640 1 -- 172.21.15.1:0/2297913165 learned_addr learned my addr 172.21.15.1:0/2297913165 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50153160 0x7f1b5014e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b554f9640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b50152d50 0x7f1b5014c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:25.252 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1b500cddc0 con 0x7f1b50153160 2024-05-09T05:21:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1b500c83c0 con 0x7f1b50152d50 2024-05-09T05:21:25.253 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1b500cdf40 con 0x7f1b500cbef0 2024-05-09T05:21:25.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.255+0000 7f1b564fb640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50153160 0x7f1b5014e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:25.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b55cfa640 1 -- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 msgr2=0x7f1b501ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:21:25.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b55cfa640 1 -- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 msgr2=0x7f1b501ee380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:21:25.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b55cfa640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 0x7f1b501ee380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:25.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b55cfa640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 0x7f1b501ee380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:25.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b554f9640 1 -- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 msgr2=0x7f1b501ee380 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b554f9640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 0x7f1b501ee380 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b554f9640 1 -- 172.21.15.1:0/2297913165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50153160 msgr2=0x7f1b5014e7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b554f9640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50153160 0x7f1b5014e7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b554f9640 1 -- 172.21.15.1:0/2297913165 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b501ee8c0 con 0x7f1b50152d50 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.256+0000 7f1b554f9640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b50152d50 0x7f1b5014c370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1b40031000 tx=0x7f1b400046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.257+0000 7f1b467fc640 1 -- 172.21.15.1:0/2297913165 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b40037050 con 0x7f1b50152d50 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.257+0000 7f1b467fc640 1 -- 172.21.15.1:0/2297913165 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1b40038040 con 0x7f1b50152d50 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.257+0000 7f1b467fc640 1 -- 172.21.15.1:0/2297913165 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b4004b6c0 con 0x7f1b50152d50 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.257+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b5014f3a0 con 0x7f1b50152d50 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.257+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b501f1bb0 con 0x7f1b50152d50 2024-05-09T05:21:25.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.258+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b50050870 con 0x7f1b50152d50 2024-05-09T05:21:25.256 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.259+0000 7f1b467fc640 1 -- 172.21.15.1:0/2297913165 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1b4004b860 con 0x7f1b50152d50 2024-05-09T05:21:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.260+0000 7f1b467fc640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b34076320 0x7f1b34078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.260+0000 7f1b467fc640 1 -- 172.21.15.1:0/2297913165 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1b400c0090 con 0x7f1b50152d50 2024-05-09T05:21:25.257 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.261+0000 7f1b55cfa640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b34076320 0x7f1b34078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:25.258 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.262+0000 7f1b55cfa640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b34076320 0x7f1b34078830 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c006a90 tx=0x7f1b4c00f040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:25.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.268+0000 7f1b467fc640 1 -- 172.21.15.1:0/2297913165 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1b40089a60 con 0x7f1b50152d50 2024-05-09T05:21:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.773+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1b500ce6a0 con 0x7f1b50152d50 2024-05-09T05:21:25.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.774+0000 7f1b467fc640 1 -- 172.21.15.1:0/2297913165 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f1b40089360 con 0x7f1b50152d50 2024-05-09T05:21:25.771 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":118,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":21,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:21:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b34076320 msgr2=0x7f1b34078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b34076320 0x7f1b34078830 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f1b4c006a90 tx=0x7f1b4c00f040 comp rx=0 tx=0).stop 2024-05-09T05:21:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b50152d50 msgr2=0x7f1b5014c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b50152d50 0x7f1b5014c370 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1b40031000 tx=0x7f1b400046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.775 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 shutdown_connections 2024-05-09T05:21:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1b34076320 0x7f1b34078830 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1b50152d50 0x7f1b5014c370 secure :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1b40031000 tx=0x7f1b400046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1b50153160 0x7f1b5014e7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 --2- 172.21.15.1:0/2297913165 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1b500cbef0 0x7f1b501ee380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 >> 172.21.15.1:0/2297913165 conn(0x7f1b500ce110 msgr2=0x7f1b500744c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 shutdown_connections 2024-05-09T05:21:25.776 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:25.778+0000 7f1b57f5e640 1 -- 172.21.15.1:0/2297913165 wait complete. 2024-05-09T05:21:28.794 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:21:28.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.976+0000 7f9c8cbe0640 1 Processor -- start 2024-05-09T05:21:28.974 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.977+0000 7f9c8cbe0640 1 -- start start 2024-05-09T05:21:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.979+0000 7f9c8cbe0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.979+0000 7f9c8cbe0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c881642f0 con 0x7f9c880c5590 2024-05-09T05:21:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.979+0000 7f9c8cbe0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f9c881618e0 con 0x7f9c880c8130 2024-05-09T05:21:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.979+0000 7f9c8cbe0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9c8806dfc0 con 0x7f9c880cbef0 2024-05-09T05:21:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.979+0000 7f9c8659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f9c880c5590 0x7f9c880c7040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:52426/0 (socket says 172.21.15.1:52426) 2024-05-09T05:21:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.979+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 learned_addr learned my addr 172.21.15.1:0/1706228321 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:21:28.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.980+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.980+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:21:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.980+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.980+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:28.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:28.980+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.181+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:29.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.184+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:21:29.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.184+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:29.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.184+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:29.180 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.184+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:21:29.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.585+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:29.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.586+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:21:29.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.586+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:29.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.586+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:29.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:29.586+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:21:30.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:30.387+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:30.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:30.387+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:21:30.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:30.387+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:30.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:30.387+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:30.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:30.387+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:21:31.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c77fff640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 msgr2=0x7f9c88164d90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:31.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c77fff640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c880cbef0 0x7f9c88164d90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:31.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c77fff640 1 -- 172.21.15.1:0/1706228321 >> v1:172.21.15.196:6789/0 conn(0x7f9c880c8130 legacy=0x7f9c881695d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:31.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c77fff640 1 -- 172.21.15.1:0/1706228321 >> v1:172.21.15.1:6789/0 conn(0x7f9c880c5590 legacy=0x7f9c880c7040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:31.976 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c77fff640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c77fff640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c77fff640 1 -- 172.21.15.1:0/1706228321 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f9c8806dfc0 con 0x7f9c68006280 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c77fff640 1 -- 172.21.15.1:0/1706228321 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f9c68008e40 con 0x7f9c680036f0 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.979+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c85d9b640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 msgr2=0x7f9c68008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c85d9b640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 msgr2=0x7f9c68008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 msgr2=0x7f9c68005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 msgr2=0x7f9c68005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:31.977 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:31.980+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.181+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.181+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 msgr2=0x7f9c68005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:21:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.181+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 msgr2=0x7f9c68005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.181+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:32.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.181+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:21:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.182+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.182+0000 7f9c85d9b640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 msgr2=0x7f9c68008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:21:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.182+0000 7f9c85d9b640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 msgr2=0x7f9c68008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.182+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:32.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.182+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:21:32.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.582+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:32.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.582+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 msgr2=0x7f9c68005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:21:32.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.582+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 msgr2=0x7f9c68005ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:32.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.582+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:32.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.582+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:21:32.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.584+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:32.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.584+0000 7f9c85d9b640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 msgr2=0x7f9c68008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:21:32.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.584+0000 7f9c85d9b640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 msgr2=0x7f9c68008790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:21:32.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.584+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:32.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:32.584+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:21:33.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.383+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.196:3300/0 conn(0x7f9c680036f0 0x7f9c68005ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:33.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.383+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 msgr2=0x7f9c68008790 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:33.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.383+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> v2:172.21.15.1:3300/0 conn(0x7f9c68006280 0x7f9c68008790 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:33.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.383+0000 7f9c8659c640 1 -- 172.21.15.1:0/1706228321 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c88164590 con 0x7f9c680036f0 2024-05-09T05:21:33.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.384+0000 7f9c8659c640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c680036f0 0x7f9c68005ba0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c68003680 tx=0x7f9c8002af20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5372e7066b707793 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:33.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.385+0000 7f9c8559a640 1 -- 172.21.15.1:0/1706228321 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c80031070 con 0x7f9c680036f0 2024-05-09T05:21:33.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.385+0000 7f9c8559a640 1 -- 172.21.15.1:0/1706228321 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c8002ba20 con 0x7f9c680036f0 2024-05-09T05:21:33.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.385+0000 7f9c8559a640 1 -- 172.21.15.1:0/1706228321 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8002bdf0 con 0x7f9c680036f0 2024-05-09T05:21:33.383 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.386+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c680036f0 msgr2=0x7f9c68005ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:33.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.386+0000 7f9c8cbe0640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c680036f0 0x7f9c68005ba0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c68003680 tx=0x7f9c8002af20 comp rx=0 tx=0).stop 2024-05-09T05:21:33.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.386+0000 7f9c86d9d640 1 -- 172.21.15.1:0/1706228321 reap_dead start 2024-05-09T05:21:33.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.386+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/1706228321 shutdown_connections 2024-05-09T05:21:33.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.386+0000 7f9c8cbe0640 1 --2- 172.21.15.1:0/1706228321 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c680036f0 0x7f9c68005ba0 secure :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f9c68003680 tx=0x7f9c8002af20 comp rx=0 tx=0).stop 2024-05-09T05:21:33.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.386+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/1706228321 >> 172.21.15.1:0/1706228321 conn(0x7f9c880ce110 msgr2=0x7f9c880ce520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:33.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.386+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/1706228321 shutdown_connections 2024-05-09T05:21:33.384 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.386+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/1706228321 wait complete. 2024-05-09T05:21:33.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.389+0000 7f9c8cbe0640 1 Processor -- start 2024-05-09T05:21:33.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.390+0000 7f9c8cbe0640 1 -- start start 2024-05-09T05:21:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.391+0000 7f9c8cbe0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 0x7f9c8815d650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.391+0000 7f9c8cbe0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c88160fb0 0x7f9c881601f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.391+0000 7f9c8cbe0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c88160730 0x7f9c880c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.391+0000 7f9c8cbe0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f9c880c7750 con 0x7f9c88160730 2024-05-09T05:21:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.391+0000 7f9c8cbe0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f9c88161bc0 con 0x7f9c88160fb0 2024-05-09T05:21:33.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.391+0000 7f9c8cbe0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9c880c78d0 con 0x7f9c8815c080 2024-05-09T05:21:33.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c86d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c88160730 0x7f9c880c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:33.389 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c86d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c88160730 0x7f9c880c8d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34782/0 (socket says 172.21.15.1:34782) 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c86d9d640 1 -- 172.21.15.1:0/2014894418 learned_addr learned my addr 172.21.15.1:0/2014894418 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c85d9b640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c88160fb0 0x7f9c881601f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c8659c640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 0x7f9c8815d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c8659c640 1 -- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 msgr2=0x7f9c8815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c8659c640 1 -- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 msgr2=0x7f9c8815d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c8659c640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 0x7f9c8815d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c8659c640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 0x7f9c8815d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c86d9d640 1 -- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 msgr2=0x7f9c8815d650 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 0x7f9c8815d650 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c86d9d640 1 -- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c88160fb0 msgr2=0x7f9c881601f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c88160fb0 0x7f9c881601f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.392+0000 7f9c86d9d640 1 -- 172.21.15.1:0/2014894418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c8815db90 con 0x7f9c88160730 2024-05-09T05:21:33.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.393+0000 7f9c86d9d640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c88160730 0x7f9c880c8d10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9c880c5460 tx=0x7f9c7c013a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:33.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.393+0000 7f9c76ffd640 1 -- 172.21.15.1:0/2014894418 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c01c070 con 0x7f9c88160730 2024-05-09T05:21:33.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.393+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c881f7c20 con 0x7f9c88160730 2024-05-09T05:21:33.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.393+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c8815b2c0 con 0x7f9c88160730 2024-05-09T05:21:33.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.393+0000 7f9c76ffd640 1 -- 172.21.15.1:0/2014894418 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9c7c015690 con 0x7f9c88160730 2024-05-09T05:21:33.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.393+0000 7f9c76ffd640 1 -- 172.21.15.1:0/2014894418 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c7c015a60 con 0x7f9c88160730 2024-05-09T05:21:33.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.395+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c88050870 con 0x7f9c88160730 2024-05-09T05:21:33.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.396+0000 7f9c76ffd640 1 -- 172.21.15.1:0/2014894418 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9c7c01f070 con 0x7f9c88160730 2024-05-09T05:21:33.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.397+0000 7f9c76ffd640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c68077160 0x7f9c68079670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:33.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.397+0000 7f9c76ffd640 1 -- 172.21.15.1:0/2014894418 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f9c7c097a50 con 0x7f9c88160730 2024-05-09T05:21:33.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.397+0000 7f9c8659c640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c68077160 0x7f9c68079670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:33.395 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.399+0000 7f9c8659c640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c68077160 0x7f9c68079670 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9c80036000 tx=0x7f9c80010040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:33.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.406+0000 7f9c76ffd640 1 -- 172.21.15.1:0/2014894418 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f9c7c026020 con 0x7f9c88160730 2024-05-09T05:21:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.908+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f9c880c7f10 con 0x7f9c88160730 2024-05-09T05:21:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.910+0000 7f9c76ffd640 1 -- 172.21.15.1:0/2014894418 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f9c7c026020 con 0x7f9c88160730 2024-05-09T05:21:33.906 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":122,"quorum":[0,1],"quorum_names":["a","b"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.913+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c68077160 msgr2=0x7f9c68079670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.913+0000 7f9c8cbe0640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c68077160 0x7f9c68079670 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f9c80036000 tx=0x7f9c80010040 comp rx=0 tx=0).stop 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.913+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c88160730 msgr2=0x7f9c880c8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.913+0000 7f9c8cbe0640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c88160730 0x7f9c880c8d10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f9c880c5460 tx=0x7f9c7c013a20 comp rx=0 tx=0).stop 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.914+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 shutdown_connections 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.914+0000 7f9c8cbe0640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9c68077160 0x7f9c68079670 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.914+0000 7f9c8cbe0640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f9c88160fb0 0x7f9c881601f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.914+0000 7f9c8cbe0640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f9c88160730 0x7f9c880c8d10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.914+0000 7f9c8cbe0640 1 --2- 172.21.15.1:0/2014894418 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9c8815c080 0x7f9c8815d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:33.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.914+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 >> 172.21.15.1:0/2014894418 conn(0x7f9c880ce110 msgr2=0x7f9c880c9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:33.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.914+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 shutdown_connections 2024-05-09T05:21:33.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:33.914+0000 7f9c8cbe0640 1 -- 172.21.15.1:0/2014894418 wait complete. 2024-05-09T05:21:35.951 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:21:35.955 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:21:36.929 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:21:37.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.112+0000 7f54398ee640 1 Processor -- start 2024-05-09T05:21:37.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.112+0000 7f54398ee640 1 -- start start 2024-05-09T05:21:37.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f54398ee640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54340c87d0 0x7f54340c5ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f54398ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54341538b0 0x7f54340c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f54398ee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f54341594f0 con 0x7f54340c9810 2024-05-09T05:21:37.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f54398ee640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f543415a380 con 0x7f54340c87d0 2024-05-09T05:21:37.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f54398ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5434159770 con 0x7f54341538b0 2024-05-09T05:21:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f5432ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54340c87d0 0x7f54340c5ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f542a7fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54341538b0 0x7f54340c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f5432ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f54340c87d0 0x7f54340c5ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34790/0 (socket says 172.21.15.1:34790) 2024-05-09T05:21:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f5432ffd640 1 -- 172.21.15.1:0/4012425422 learned_addr learned my addr 172.21.15.1:0/4012425422 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f5432ffd640 1 -- 172.21.15.1:0/4012425422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54341538b0 msgr2=0x7f54340c92d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f5432ffd640 1 --2- 172.21.15.1:0/4012425422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54341538b0 0x7f54340c92d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f5432ffd640 1 -- 172.21.15.1:0/4012425422 >> v1:172.21.15.1:6789/0 conn(0x7f54340c9810 legacy=0x7f543415cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:37.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.115+0000 7f5432ffd640 1 -- 172.21.15.1:0/4012425422 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5434159a50 con 0x7f54340c87d0 2024-05-09T05:21:37.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.117+0000 7f5432ffd640 1 --2- 172.21.15.1:0/4012425422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54340c87d0 0x7f54340c5ca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5424009e50 tx=0x7f542402a680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=760c1c3c51f4e052 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.117+0000 7f54327fc640 1 -- 172.21.15.1:0/4012425422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424036070 con 0x7f54340c87d0 2024-05-09T05:21:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.117+0000 7f54327fc640 1 -- 172.21.15.1:0/4012425422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5424031600 con 0x7f54340c87d0 2024-05-09T05:21:37.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.117+0000 7f54327fc640 1 -- 172.21.15.1:0/4012425422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54240319d0 con 0x7f54340c87d0 2024-05-09T05:21:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.119+0000 7f54398ee640 1 -- 172.21.15.1:0/4012425422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54340c87d0 msgr2=0x7f54340c5ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.119+0000 7f54398ee640 1 --2- 172.21.15.1:0/4012425422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54340c87d0 0x7f54340c5ca0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5424009e50 tx=0x7f542402a680 comp rx=0 tx=0).stop 2024-05-09T05:21:37.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.119+0000 7f54398ee640 1 -- 172.21.15.1:0/4012425422 shutdown_connections 2024-05-09T05:21:37.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.119+0000 7f54398ee640 1 --2- 172.21.15.1:0/4012425422 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54341538b0 0x7f54340c92d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.119+0000 7f54398ee640 1 --2- 172.21.15.1:0/4012425422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54340c87d0 0x7f54340c5ca0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.119+0000 7f54398ee640 1 -- 172.21.15.1:0/4012425422 >> 172.21.15.1:0/4012425422 conn(0x7f54340745a0 msgr2=0x7f5434071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:37.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.119+0000 7f54398ee640 1 -- 172.21.15.1:0/4012425422 shutdown_connections 2024-05-09T05:21:37.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.119+0000 7f54398ee640 1 -- 172.21.15.1:0/4012425422 wait complete. 2024-05-09T05:21:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.122+0000 7f54398ee640 1 Processor -- start 2024-05-09T05:21:37.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.122+0000 7f54398ee640 1 -- start start 2024-05-09T05:21:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.125+0000 7f54398ee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54340c87d0 0x7f5434152b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.125+0000 7f54398ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54340c9810 0x7f5434140eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f54398ee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54341538b0 0x7f5434144050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f5432ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54340c87d0 0x7f5434152b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f5432ffd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54340c87d0 0x7f5434152b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:59906/0 (socket says 172.21.15.1:59906) 2024-05-09T05:21:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f542a7fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54340c9810 0x7f5434140eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f5432ffd640 1 -- 172.21.15.1:0/3091887546 learned_addr learned my addr 172.21.15.1:0/3091887546 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f54340c61e0 con 0x7f54341538b0 2024-05-09T05:21:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f543415a010 con 0x7f54340c87d0 2024-05-09T05:21:37.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f54340c70c0 con 0x7f54340c9810 2024-05-09T05:21:37.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f54337fe640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54341538b0 0x7f5434144050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f5432ffd640 1 -- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54340c9810 msgr2=0x7f5434140eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f5432ffd640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54340c9810 0x7f5434140eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.126+0000 7f5432ffd640 1 -- 172.21.15.1:0/3091887546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54341538b0 msgr2=0x7f5434144050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.127+0000 7f5432ffd640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54341538b0 0x7f5434144050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.127+0000 7f5432ffd640 1 -- 172.21.15.1:0/3091887546 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54341413f0 con 0x7f54340c87d0 2024-05-09T05:21:37.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.127+0000 7f54337fe640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54341538b0 0x7f5434144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:21:37.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.128+0000 7f5432ffd640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54340c87d0 0x7f5434152b50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5424009e50 tx=0x7f5424004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:37.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.128+0000 7f5430ff9640 1 -- 172.21.15.1:0/3091887546 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424041020 con 0x7f54340c87d0 2024-05-09T05:21:37.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.128+0000 7f5430ff9640 1 -- 172.21.15.1:0/3091887546 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5424031e40 con 0x7f54340c87d0 2024-05-09T05:21:37.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.128+0000 7f5430ff9640 1 -- 172.21.15.1:0/3091887546 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5424036040 con 0x7f54340c87d0 2024-05-09T05:21:37.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.131+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54341f41e0 con 0x7f54340c87d0 2024-05-09T05:21:37.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.131+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54341f47a0 con 0x7f54340c87d0 2024-05-09T05:21:37.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.132+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5434158720 con 0x7f54340c87d0 2024-05-09T05:21:37.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.133+0000 7f5430ff9640 1 -- 172.21.15.1:0/3091887546 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5424044020 con 0x7f54340c87d0 2024-05-09T05:21:37.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.134+0000 7f5430ff9640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5414076350 0x7f5414078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.134+0000 7f5430ff9640 1 -- 172.21.15.1:0/3091887546 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f54240b82a0 con 0x7f54340c87d0 2024-05-09T05:21:37.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.134+0000 7f542a7fc640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5414076350 0x7f5414078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.135+0000 7f542a7fc640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5414076350 0x7f5414078860 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f543414ca90 tx=0x7f541c014040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:37.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.143+0000 7f5430ff9640 1 -- 172.21.15.1:0/3091887546 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5434158720 con 0x7f54340c87d0 2024-05-09T05:21:37.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.654+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5434050870 con 0x7f54340c87d0 2024-05-09T05:21:37.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.655+0000 7f5430ff9640 1 -- 172.21.15.1:0/3091887546 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f542404b070 con 0x7f54340c87d0 2024-05-09T05:21:37.652 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:21:37.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.659+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5414076350 msgr2=0x7f5414078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.659+0000 7f54398ee640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5414076350 0x7f5414078860 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f543414ca90 tx=0x7f541c014040 comp rx=0 tx=0).stop 2024-05-09T05:21:37.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.659+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54340c87d0 msgr2=0x7f5434152b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.659+0000 7f54398ee640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54340c87d0 0x7f5434152b50 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f5424009e50 tx=0x7f5424004a40 comp rx=0 tx=0).stop 2024-05-09T05:21:37.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.660+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 shutdown_connections 2024-05-09T05:21:37.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.660+0000 7f54398ee640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5414076350 0x7f5414078860 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.660+0000 7f54398ee640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f54341538b0 0x7f5434144050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.660+0000 7f54398ee640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f54340c9810 0x7f5434140eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.660+0000 7f54398ee640 1 --2- 172.21.15.1:0/3091887546 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f54340c87d0 0x7f5434152b50 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.660+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 >> 172.21.15.1:0/3091887546 conn(0x7f54340745a0 msgr2=0x7f54340cb060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:37.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.660+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 shutdown_connections 2024-05-09T05:21:37.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.660+0000 7f54398ee640 1 -- 172.21.15.1:0/3091887546 wait complete. 2024-05-09T05:21:37.677 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:21:37.677 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:21:37.677 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:21:37.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.872+0000 7ffa5b33f640 1 Processor -- start 2024-05-09T05:21:37.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.872+0000 7ffa5b33f640 1 -- start start 2024-05-09T05:21:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.874+0000 7ffa5b33f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa540c4b60 0x7ffa540c86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.874+0000 7ffa5b33f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ffa5415b0e0 0x7ffa5406c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.874+0000 7ffa5b33f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa540743e0 con 0x7ffa5406cc30 2024-05-09T05:21:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.874+0000 7ffa5b33f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ffa54074ab0 con 0x7ffa5415b0e0 2024-05-09T05:21:37.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.874+0000 7ffa5b33f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffa54073620 con 0x7ffa540c4b60 2024-05-09T05:21:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.875+0000 7ffa590db640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa540c4b60 0x7ffa540c86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.875+0000 7ffa588da640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ffa5415b0e0 0x7ffa5406c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.875+0000 7ffa590db640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa540c4b60 0x7ffa540c86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:49266/0 (socket says 172.21.15.1:49266) 2024-05-09T05:21:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.875+0000 7ffa590db640 1 -- 172.21.15.1:0/261441320 learned_addr learned my addr 172.21.15.1:0/261441320 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:37.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.875+0000 7ffa590db640 1 -- 172.21.15.1:0/261441320 >> v2:172.21.15.196:3300/0 conn(0x7ffa5415b0e0 msgr2=0x7ffa5406c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.875+0000 7ffa590db640 1 --2- 172.21.15.1:0/261441320 >> v2:172.21.15.196:3300/0 conn(0x7ffa5415b0e0 0x7ffa5406c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.875+0000 7ffa590db640 1 -- 172.21.15.1:0/261441320 >> v1:172.21.15.196:6789/0 conn(0x7ffa5406cc30 legacy=0x7ffa540c91e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.875+0000 7ffa590db640 1 -- 172.21.15.1:0/261441320 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa54074100 con 0x7ffa540c4b60 2024-05-09T05:21:37.874 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.878+0000 7ffa590db640 1 --2- 172.21.15.1:0/261441320 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa540c4b60 0x7ffa540c86d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009870 tx=0x7ffa4802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f8aff9cdefb670c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:37.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.878+0000 7ffa47fff640 1 -- 172.21.15.1:0/261441320 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48037070 con 0x7ffa540c4b60 2024-05-09T05:21:37.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.878+0000 7ffa47fff640 1 -- 172.21.15.1:0/261441320 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa48032440 con 0x7ffa540c4b60 2024-05-09T05:21:37.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.878+0000 7ffa47fff640 1 -- 172.21.15.1:0/261441320 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48037070 con 0x7ffa540c4b60 2024-05-09T05:21:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.879+0000 7ffa5b33f640 1 -- 172.21.15.1:0/261441320 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa540c4b60 msgr2=0x7ffa540c86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.879+0000 7ffa5b33f640 1 --2- 172.21.15.1:0/261441320 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa540c4b60 0x7ffa540c86d0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009870 tx=0x7ffa4802a250 comp rx=0 tx=0).stop 2024-05-09T05:21:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.880+0000 7ffa5b33f640 1 -- 172.21.15.1:0/261441320 shutdown_connections 2024-05-09T05:21:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.880+0000 7ffa5b33f640 1 --2- 172.21.15.1:0/261441320 >> v2:172.21.15.196:3300/0 conn(0x7ffa5415b0e0 0x7ffa5406c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.880+0000 7ffa5b33f640 1 --2- 172.21.15.1:0/261441320 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa540c4b60 0x7ffa540c86d0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.880+0000 7ffa5b33f640 1 -- 172.21.15.1:0/261441320 >> 172.21.15.1:0/261441320 conn(0x7ffa540cdcd0 msgr2=0x7ffa540ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.880+0000 7ffa5b33f640 1 -- 172.21.15.1:0/261441320 shutdown_connections 2024-05-09T05:21:37.877 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.880+0000 7ffa5b33f640 1 -- 172.21.15.1:0/261441320 wait complete. 2024-05-09T05:21:37.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.883+0000 7ffa5b33f640 1 Processor -- start 2024-05-09T05:21:37.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.883+0000 7ffa5b33f640 1 -- start start 2024-05-09T05:21:37.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.885+0000 7ffa5b33f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa5406cc30 0x7ffa5414d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.885+0000 7ffa5b33f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa540c4b60 0x7ffa541fa930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.885+0000 7ffa5b33f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa5415b0e0 0x7ffa541f72d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.885+0000 7ffa5b33f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa540739c0 con 0x7ffa540c4b60 2024-05-09T05:21:37.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.886+0000 7ffa5b33f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffa540732c0 con 0x7ffa5415b0e0 2024-05-09T05:21:37.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.886+0000 7ffa5b33f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffa540c98f0 con 0x7ffa5406cc30 2024-05-09T05:21:37.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.886+0000 7ffa590db640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa5406cc30 0x7ffa5414d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.886+0000 7ffa590db640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa5406cc30 0x7ffa5414d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:49272/0 (socket says 172.21.15.1:49272) 2024-05-09T05:21:37.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.886+0000 7ffa590db640 1 -- 172.21.15.1:0/2218880799 learned_addr learned my addr 172.21.15.1:0/2218880799 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:37.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.886+0000 7ffa598dc640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa5415b0e0 0x7ffa541f72d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.884 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.887+0000 7ffa588da640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa540c4b60 0x7ffa541fa930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.887+0000 7ffa590db640 1 -- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa5415b0e0 msgr2=0x7ffa541f72d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.887+0000 7ffa590db640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa5415b0e0 0x7ffa541f72d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.887+0000 7ffa590db640 1 -- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa540c4b60 msgr2=0x7ffa541fa930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.887+0000 7ffa590db640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa540c4b60 0x7ffa541fa930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.887+0000 7ffa590db640 1 -- 172.21.15.1:0/2218880799 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa541fae70 con 0x7ffa5406cc30 2024-05-09T05:21:37.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.887+0000 7ffa590db640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa5406cc30 0x7ffa5414d710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffa48000c00 tx=0x7ffa480046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:37.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.888+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48037040 con 0x7ffa5406cc30 2024-05-09T05:21:37.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.888+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffa48003c60 con 0x7ffa5406cc30 2024-05-09T05:21:37.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.888+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa480327f0 con 0x7ffa5406cc30 2024-05-09T05:21:37.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.888+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa541f7810 con 0x7ffa5406cc30 2024-05-09T05:21:37.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.888+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa5414f080 con 0x7ffa5406cc30 2024-05-09T05:21:37.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.889+0000 7ffa5b33f640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414d180 0x7ffa5414f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.889+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa541f79e0 con 0x7ffa5414d180 2024-05-09T05:21:37.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.890+0000 7ffa588da640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414d180 0x7ffa5414f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.890+0000 7ffa588da640 1 -- 172.21.15.1:0/2218880799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa54050870 con 0x7ffa5414d180 2024-05-09T05:21:37.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.890+0000 7ffa588da640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414d180 0x7ffa5414f360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa50011500 tx=0x7ffa500119c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:37.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.891+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa48032990 con 0x7ffa5406cc30 2024-05-09T05:21:37.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.892+0000 7ffa45ffb640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa380762d0 0x7ffa380787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.892+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ffa480b7d50 con 0x7ffa5406cc30 2024-05-09T05:21:37.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.893+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50015870 con 0x7ffa5414d180 2024-05-09T05:21:37.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.893+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ffa50015a10 con 0x7ffa5414d180 2024-05-09T05:21:37.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.893+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414d180 msgr2=0x7ffa5414f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.893+0000 7ffa45ffb640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414d180 0x7ffa5414f360 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa50011500 tx=0x7ffa500119c0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.893+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414d180 msgr2=0x7ffa5414f360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:37.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.893+0000 7ffa45ffb640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414d180 0x7ffa5414f360 secure :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7ffa50011500 tx=0x7ffa500119c0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.893+0000 7ffa598dc640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa380762d0 0x7ffa380787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.894+0000 7ffa598dc640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa380762d0 0x7ffa380787e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ffa5415e850 tx=0x7ffa4c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:37.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.906+0000 7ffa5b33f640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414e490 0x7ffa5414c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:37.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.906+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa54158650 con 0x7ffa5414e490 2024-05-09T05:21:37.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.907+0000 7ffa588da640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414e490 0x7ffa5414c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:37.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.907+0000 7ffa588da640 1 -- 172.21.15.1:0/2218880799 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ffa54151110 con 0x7ffa5414e490 2024-05-09T05:21:37.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.908+0000 7ffa588da640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414e490 0x7ffa5414c1b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffa50011500 tx=0x7ffa500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:37.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.908+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa50004220 con 0x7ffa5414e490 2024-05-09T05:21:37.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.908+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7ffa50004220 con 0x7ffa5414e490 2024-05-09T05:21:37.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.908+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414e490 msgr2=0x7ffa5414c1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.908+0000 7ffa45ffb640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414e490 0x7ffa5414c1b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7ffa50011500 tx=0x7ffa500046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.908+0000 7ffa45ffb640 1 -- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414e490 msgr2=0x7ffa5414c1b0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:37.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.908+0000 7ffa45ffb640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa5414e490 0x7ffa5414c1b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.906 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:21:37.906 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:37.906 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:37.906 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:21:37.906 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 126, 2024-05-09T05:21:37.906 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 86468, 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:37.907 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:37.908 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.909 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:21:37.910 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:37.911 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:37.912 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:21:37.913 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.914 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:37.915 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.916 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:37.917 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:37.918 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:37.919 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.920 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.921 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:21:37.922 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:21:37.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa380762d0 msgr2=0x7ffa380787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa5b33f640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa380762d0 0x7ffa380787e0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7ffa5415e850 tx=0x7ffa4c009040 comp rx=0 tx=0).stop 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa5406cc30 msgr2=0x7ffa5414d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa5b33f640 1 --2- 172.21.15.1:0/2218880799 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa5406cc30 0x7ffa5414d710 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7ffa48000c00 tx=0x7ffa480046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa598dc640 1 -- 172.21.15.1:0/2218880799 reap_dead start 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa598dc640 1 -- 172.21.15.1:0/2218880799 reap_dead start 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 shutdown_connections 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 >> 172.21.15.1:0/2218880799 conn(0x7ffa540cdcd0 msgr2=0x7ffa54075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 shutdown_connections 2024-05-09T05:21:37.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:37.909+0000 7ffa5b33f640 1 -- 172.21.15.1:0/2218880799 wait complete. 2024-05-09T05:21:37.927 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:21:37.928 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:21:37.928 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:21:38.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.113+0000 7ff6c6470640 1 Processor -- start 2024-05-09T05:21:38.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.114+0000 7ff6c6470640 1 -- start start 2024-05-09T05:21:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.115+0000 7ff6c6470640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6c0159700 0x7ff6c0159b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c6470640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c00cb450 con 0x7ff6c00c8b70 2024-05-09T05:21:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c6470640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6c00c9e50 con 0x7ff6c00c9520 2024-05-09T05:21:38.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c6470640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff6c00caed0 con 0x7ff6c0159700 2024-05-09T05:21:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c4a0d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6c0159700 0x7ff6c0159b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c4a0d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6c0159700 0x7ff6c0159b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:34836/0 (socket says 172.21.15.1:34836) 2024-05-09T05:21:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c4a0d640 1 -- 172.21.15.1:0/1439460050 learned_addr learned my addr 172.21.15.1:0/1439460050 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c4a0d640 1 -- 172.21.15.1:0/1439460050 >> v1:172.21.15.196:6789/0 conn(0x7ff6c00c9520 legacy=0x7ff6c00717f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:38.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c4a0d640 1 -- 172.21.15.1:0/1439460050 >> v1:172.21.15.1:6789/0 conn(0x7ff6c00c8b70 legacy=0x7ff6c0156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.116+0000 7ff6c4a0d640 1 -- 172.21.15.1:0/1439460050 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c00cb170 con 0x7ff6c0159700 2024-05-09T05:21:38.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.118+0000 7ff6c4a0d640 1 --2- 172.21.15.1:0/1439460050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6c0159700 0x7ff6c0159b10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009e50 tx=0x7ff6b802a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=120a3504f015369a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.118+0000 7ff6beffd640 1 -- 172.21.15.1:0/1439460050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b8036070 con 0x7ff6c0159700 2024-05-09T05:21:38.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.119+0000 7ff6beffd640 1 -- 172.21.15.1:0/1439460050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b8031600 con 0x7ff6c0159700 2024-05-09T05:21:38.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.119+0000 7ff6beffd640 1 -- 172.21.15.1:0/1439460050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b80319d0 con 0x7ff6c0159700 2024-05-09T05:21:38.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.120+0000 7ff6c6470640 1 -- 172.21.15.1:0/1439460050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6c0159700 msgr2=0x7ff6c0159b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.120+0000 7ff6c6470640 1 --2- 172.21.15.1:0/1439460050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6c0159700 0x7ff6c0159b10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7ff6b8009e50 tx=0x7ff6b802a710 comp rx=0 tx=0).stop 2024-05-09T05:21:38.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.120+0000 7ff6c6470640 1 -- 172.21.15.1:0/1439460050 shutdown_connections 2024-05-09T05:21:38.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.120+0000 7ff6c6470640 1 --2- 172.21.15.1:0/1439460050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6c0159700 0x7ff6c0159b10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.120+0000 7ff6c6470640 1 -- 172.21.15.1:0/1439460050 >> 172.21.15.1:0/1439460050 conn(0x7ff6c006d800 msgr2=0x7ff6c006dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:38.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.120+0000 7ff6c6470640 1 -- 172.21.15.1:0/1439460050 shutdown_connections 2024-05-09T05:21:38.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.120+0000 7ff6c6470640 1 -- 172.21.15.1:0/1439460050 wait complete. 2024-05-09T05:21:38.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.123+0000 7ff6c6470640 1 Processor -- start 2024-05-09T05:21:38.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.124+0000 7ff6c6470640 1 -- start start 2024-05-09T05:21:38.123 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c6470640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6c00c8b70 0x7ff6c01f7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c6470640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6c00c9520 0x7ff6c01ec5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c6470640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0159700 0x7ff6c014ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c6470640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c015a050 con 0x7ff6c00c9520 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c6470640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c00c3f30 con 0x7ff6c0159700 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c6470640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff6c0071210 con 0x7ff6c00c8b70 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c4a0d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0159700 0x7ff6c014ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6bffff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6c00c8b70 0x7ff6c01f7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c4a0d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0159700 0x7ff6c014ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:59920/0 (socket says 172.21.15.1:59920) 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6c4a0d640 1 -- 172.21.15.1:0/3388495953 learned_addr learned my addr 172.21.15.1:0/3388495953 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.126+0000 7ff6bffff640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6c00c8b70 0x7ff6c01f7300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:49286/0 (socket says 172.21.15.1:49286) 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.127+0000 7ff6bf7fe640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6c00c9520 0x7ff6c01ec5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.124 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.127+0000 7ff6bffff640 1 -- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0159700 msgr2=0x7ff6c014ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.127+0000 7ff6bffff640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0159700 0x7ff6c014ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.127+0000 7ff6bffff640 1 -- 172.21.15.1:0/3388495953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6c00c9520 msgr2=0x7ff6c01ec5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.127+0000 7ff6bffff640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6c00c9520 0x7ff6c01ec5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.127+0000 7ff6bffff640 1 -- 172.21.15.1:0/3388495953 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6c00cc920 con 0x7ff6c00c8b70 2024-05-09T05:21:38.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.127+0000 7ff6c4a0d640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0159700 0x7ff6c014ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:21:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.128+0000 7ff6bffff640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6c00c8b70 0x7ff6c01f7300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000c370 tx=0x7ff6b000c830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.128+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0017070 con 0x7ff6c00c8b70 2024-05-09T05:21:38.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.128+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff6b00052d0 con 0x7ff6c00c8b70 2024-05-09T05:21:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.128+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b0003810 con 0x7ff6c00c8b70 2024-05-09T05:21:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.128+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6c01f7840 con 0x7ff6c00c8b70 2024-05-09T05:21:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.128+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6c0153840 con 0x7ff6c00c8b70 2024-05-09T05:21:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.130+0000 7ff6c6470640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c01f69f0 0x7ff6c014f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.130+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c00ccaf0 con 0x7ff6c01f69f0 2024-05-09T05:21:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.130+0000 7ff6bf7fe640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c01f69f0 0x7ff6c014f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.130+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff6b001a070 con 0x7ff6c00c8b70 2024-05-09T05:21:38.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.130+0000 7ff6bf7fe640 1 -- 172.21.15.1:0/3388495953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff6c00696d0 con 0x7ff6c01f69f0 2024-05-09T05:21:38.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.130+0000 7ff6bf7fe640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c01f69f0 0x7ff6c014f150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ce710 tx=0x7ff6b400ec00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.131+0000 7ff6bcff9640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6ac0762d0 0x7ff6ac0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.131+0000 7ff6c4a0d640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6ac0762d0 0x7ff6ac0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.131+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff6b0096900 con 0x7ff6c00c8b70 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.132+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4017070 con 0x7ff6c01f69f0 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.132+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff6b400c740 con 0x7ff6c01f69f0 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.132+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c01f69f0 msgr2=0x7ff6c014f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.132+0000 7ff6c4a0d640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6ac0762d0 0x7ff6ac0787e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff6b802a750 tx=0x7ff6b8009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.132+0000 7ff6bcff9640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c01f69f0 0x7ff6c014f150 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ce710 tx=0x7ff6b400ec00 comp rx=0 tx=0).stop 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.132+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c01f69f0 msgr2=0x7ff6c014f150 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:38.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.132+0000 7ff6bcff9640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c01f69f0 0x7ff6c014f150 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.143+0000 7ff6c6470640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0152270 0x7ff6c0152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.143+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff6c0050870 con 0x7ff6c0152270 2024-05-09T05:21:38.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.143+0000 7ff6bf7fe640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0152270 0x7ff6c0152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.143+0000 7ff6bf7fe640 1 -- 172.21.15.1:0/3388495953 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff6c014f690 con 0x7ff6c0152270 2024-05-09T05:21:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.144+0000 7ff6bf7fe640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0152270 0x7ff6c0152680 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ce710 tx=0x7ff6b40046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.144+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b4017040 con 0x7ff6c0152270 2024-05-09T05:21:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7ff6b4013d20 con 0x7ff6c0152270 2024-05-09T05:21:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0152270 msgr2=0x7ff6c0152680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6bcff9640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0152270 0x7ff6c0152680 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ce710 tx=0x7ff6b40046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6bcff9640 1 -- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0152270 msgr2=0x7ff6c0152680 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:38.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6bcff9640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6c0152270 0x7ff6c0152680 secure :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ff6c00ce710 tx=0x7ff6b40046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.142 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:21:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:21:38.142 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 126, 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 248548, 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:21:38.143 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:38.144 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:21:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:21:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:38.145 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.146 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:21:38.147 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:38.148 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:38.149 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.150 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:38.151 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.152 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:38.153 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:38.154 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:21:38.155 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:21:38.156 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.157 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:21:38.158 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:21:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6ac0762d0 msgr2=0x7ff6ac0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6c6470640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6ac0762d0 0x7ff6ac0787e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7ff6b802a750 tx=0x7ff6b8009000 comp rx=0 tx=0).stop 2024-05-09T05:21:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6c00c8b70 msgr2=0x7ff6c01f7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6c6470640 1 --2- 172.21.15.1:0/3388495953 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6c00c8b70 0x7ff6c01f7300 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7ff6b000c370 tx=0x7ff6b000c830 comp rx=0 tx=0).stop 2024-05-09T05:21:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.145+0000 7ff6c4a0d640 1 -- 172.21.15.1:0/3388495953 reap_dead start 2024-05-09T05:21:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.146+0000 7ff6c4a0d640 1 -- 172.21.15.1:0/3388495953 reap_dead start 2024-05-09T05:21:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.146+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 shutdown_connections 2024-05-09T05:21:38.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.146+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 >> 172.21.15.1:0/3388495953 conn(0x7ff6c006d800 msgr2=0x7ff6c006fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:38.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.146+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 shutdown_connections 2024-05-09T05:21:38.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.146+0000 7ff6c6470640 1 -- 172.21.15.1:0/3388495953 wait complete. 2024-05-09T05:21:38.160 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:21:38.160 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:21:38.160 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:21:38.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.352+0000 7f4770fd9640 1 Processor -- start 2024-05-09T05:21:38.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.352+0000 7f4770fd9640 1 -- start start 2024-05-09T05:21:38.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.354+0000 7f4770fd9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f476c15cfa0 0x7f476c0c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.354+0000 7f4770fd9640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f476c15aea0 con 0x7f476c0c3870 2024-05-09T05:21:38.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.354+0000 7f4770fd9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f476c15d790 con 0x7f476c1533f0 2024-05-09T05:21:38.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.354+0000 7f4770fd9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f476c159ce0 con 0x7f476c15cfa0 2024-05-09T05:21:38.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.355+0000 7f476b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f476c1533f0 0x7f476c15a790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:38318/0 (socket says 172.21.15.1:38318) 2024-05-09T05:21:38.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.355+0000 7f476b7fe640 1 -- 172.21.15.1:0/180982099 learned_addr learned my addr 172.21.15.1:0/180982099 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:21:38.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.355+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2723591760 0 0) 0x7f476c15d790 con 0x7f476c1533f0 2024-05-09T05:21:38.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.355+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f4758003520 con 0x7f476c1533f0 2024-05-09T05:21:38.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 <== mon.1 v1:172.21.15.196:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2256673586 0 0) 0x7f4758003520 con 0x7f476c1533f0 2024-05-09T05:21:38.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 >> v2:172.21.15.1:3300/0 conn(0x7f476c15cfa0 msgr2=0x7f476c0c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:38.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 --2- 172.21.15.1:0/180982099 >> v2:172.21.15.1:3300/0 conn(0x7f476c15cfa0 0x7f476c0c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 >> v1:172.21.15.1:6789/0 conn(0x7f476c0c3870 legacy=0x7f476c15c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 --> v1:172.21.15.196:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c15d4b0 con 0x7f476c1533f0 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 <== mon.1 v1:172.21.15.196:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f475c001a70 con 0x7f476c1533f0 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 >> v1:172.21.15.196:6789/0 conn(0x7f476c1533f0 legacy=0x7f476c15a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4758003a00 0x7f4758003e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4758004800 0x7f4758004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758005150 0x7f475800ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f476c159ce0 con 0x7f4758005150 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f47580106b0 con 0x7f4758003a00 2024-05-09T05:21:38.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f47580108b0 con 0x7f4758004800 2024-05-09T05:21:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.356+0000 7f476b7fe640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4758003a00 0x7f4758003e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.357+0000 7f476bfff640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758005150 0x7f475800ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.357+0000 7f4763fff640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4758004800 0x7f4758004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.358+0000 7f476b7fe640 1 -- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4758004800 msgr2=0x7f4758004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.358+0000 7f476b7fe640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f4758004800 0x7f4758004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.358+0000 7f476b7fe640 1 -- 172.21.15.1:0/180982099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758005150 msgr2=0x7f475800ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.358+0000 7f476b7fe640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758005150 0x7f475800ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.358+0000 7f476b7fe640 1 -- 172.21.15.1:0/180982099 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c15d4b0 con 0x7f4758003a00 2024-05-09T05:21:38.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.358+0000 7f476bfff640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758005150 0x7f475800ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:21:38.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.358+0000 7f476bfff640 1 -- 172.21.15.1:0/180982099 reap_dead start 2024-05-09T05:21:38.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.359+0000 7f476b7fe640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4758003a00 0x7f4758003e50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f475c00a900 tx=0x7f4758005830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c516eb65331814d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.360+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c00a520 con 0x7f4758003a00 2024-05-09T05:21:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.360+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f475c02f070 con 0x7f4758003a00 2024-05-09T05:21:38.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.360+0000 7f476affd640 1 -- 172.21.15.1:0/180982099 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c02bd90 con 0x7f4758003a00 2024-05-09T05:21:38.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.361+0000 7f4770fd9640 1 -- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4758003a00 msgr2=0x7f4758003e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.361+0000 7f4770fd9640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4758003a00 0x7f4758003e50 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f475c00a900 tx=0x7f4758005830 comp rx=0 tx=0).stop 2024-05-09T05:21:38.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.361+0000 7f4770fd9640 1 -- 172.21.15.1:0/180982099 shutdown_connections 2024-05-09T05:21:38.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.361+0000 7f4770fd9640 1 --2- 172.21.15.1:0/180982099 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f4758003a00 0x7f4758003e50 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.361+0000 7f4770fd9640 1 -- 172.21.15.1:0/180982099 >> 172.21.15.1:0/180982099 conn(0x7f476c071930 msgr2=0x7f476c071d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:38.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.361+0000 7f4770fd9640 1 -- 172.21.15.1:0/180982099 shutdown_connections 2024-05-09T05:21:38.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.361+0000 7f4770fd9640 1 -- 172.21.15.1:0/180982099 wait complete. 2024-05-09T05:21:38.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.364+0000 7f4770fd9640 1 Processor -- start 2024-05-09T05:21:38.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.364+0000 7f4770fd9640 1 -- start start 2024-05-09T05:21:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f4770fd9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758003a00 0x7f476c1eec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f4770fd9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f476c0c60f0 0x7f476c0ca230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f4770fd9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c14cf10 0x7f476c14d320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f4770fd9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f476c159970 con 0x7f4758003a00 2024-05-09T05:21:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f4770fd9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f476c159690 con 0x7f476c0c60f0 2024-05-09T05:21:38.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f4770fd9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f476c15a170 con 0x7f476c14cf10 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f476bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c14cf10 0x7f476c14d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f476bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c14cf10 0x7f476c14d320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:49304/0 (socket says 172.21.15.1:49304) 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f476bfff640 1 -- 172.21.15.1:0/3110419166 learned_addr learned my addr 172.21.15.1:0/3110419166 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.368+0000 7f476b7fe640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758003a00 0x7f476c1eec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.369+0000 7f476bfff640 1 -- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f476c0c60f0 msgr2=0x7f476c0ca230 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.369+0000 7f476bfff640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f476c0c60f0 0x7f476c0ca230 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.369+0000 7f476bfff640 1 -- 172.21.15.1:0/3110419166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758003a00 msgr2=0x7f476c1eec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.369+0000 7f476bfff640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f4758003a00 0x7f476c1eec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.369+0000 7f476bfff640 1 -- 172.21.15.1:0/3110419166 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f476c0c7f30 con 0x7f476c14cf10 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.369+0000 7f476bfff640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c14cf10 0x7f476c14d320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f476c156eb0 tx=0x7f476400ec00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.369+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4764009480 con 0x7f476c14cf10 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.369+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f476401a070 con 0x7f476c14cf10 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.370+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f476400c880 con 0x7f476c14cf10 2024-05-09T05:21:38.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.370+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f476c0c7530 con 0x7f476c14cf10 2024-05-09T05:21:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.370+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f476c0cb050 con 0x7f476c14cf10 2024-05-09T05:21:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.372+0000 7f4770fd9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c0c6500 0x7f476c1f16f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.372 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.372+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f476c0ceef0 con 0x7f476c0c6500 2024-05-09T05:21:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.374+0000 7f476b7fe640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c0c6500 0x7f476c1f16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.374+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f476400ca20 con 0x7f476c14cf10 2024-05-09T05:21:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.375+0000 7f4768ff9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f474c076320 0x7f474c078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.375+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f47640970f0 con 0x7f476c14cf10 2024-05-09T05:21:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.376+0000 7f4763fff640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f474c076320 0x7f474c078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.376+0000 7f476b7fe640 1 -- 172.21.15.1:0/3110419166 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f476c0c8210 con 0x7f476c0c6500 2024-05-09T05:21:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.379+0000 7f476b7fe640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c0c6500 0x7f476c1f16f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f475c030000 tx=0x7f475c001c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.380+0000 7f4763fff640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f474c076320 0x7f474c078830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f476c0c7390 tx=0x7f4754009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.380+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c02bce0 con 0x7f476c0c6500 2024-05-09T05:21:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.381+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f475c044070 con 0x7f476c0c6500 2024-05-09T05:21:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.381+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c0c6500 msgr2=0x7f476c1f16f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.381+0000 7f4768ff9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c0c6500 0x7f476c1f16f0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f475c030000 tx=0x7f475c001c10 comp rx=0 tx=0).stop 2024-05-09T05:21:38.378 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.381+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c0c6500 msgr2=0x7f476c1f16f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:38.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.381+0000 7f4768ff9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c0c6500 0x7f476c1f16f0 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f475c030000 tx=0x7f475c001c10 comp rx=0 tx=0).stop 2024-05-09T05:21:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.394+0000 7f4770fd9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c1f1c30 0x7f476c1f2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.394+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f476c1f2580 con 0x7f476c1f1c30 2024-05-09T05:21:38.391 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.395+0000 7f476b7fe640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c1f1c30 0x7f476c1f2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.395+0000 7f476b7fe640 1 -- 172.21.15.1:0/3110419166 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f476c14d860 con 0x7f476c1f1c30 2024-05-09T05:21:38.392 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.396+0000 7f476b7fe640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c1f1c30 0x7f476c1f2040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f475c001350 tx=0x7f475c009030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.396+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f475c03fda0 con 0x7f476c1f1c30 2024-05-09T05:21:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.396+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4551 (secure 0 0 0) 0x7f475c02f070 con 0x7f476c1f1c30 2024-05-09T05:21:38.393 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.397+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c1f1c30 msgr2=0x7f476c1f2040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.397+0000 7f4768ff9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c1f1c30 0x7f476c1f2040 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f475c001350 tx=0x7f475c009030 comp rx=0 tx=0).stop 2024-05-09T05:21:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.397+0000 7f4768ff9640 1 -- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c1f1c30 msgr2=0x7f476c1f2040 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:38.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.397+0000 7f4768ff9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c1f1c30 0x7f476c1f2040 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f475c001350 tx=0x7f475c009030 comp rx=0 tx=0).stop 2024-05-09T05:21:38.394 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:21:38.394 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 126, 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 13221, 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.395 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 4, 2024-05-09T05:21:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:21:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:21:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:38.396 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:21:38.397 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.398 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:21:38.399 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:38.400 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:38.401 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.402 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:21:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:21:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:38.403 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:21:38.404 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.405 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:21:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:21:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:38.406 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:21:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.407 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:21:38.408 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:21:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:21:38.409 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:21:38.410 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:21:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:38.411 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:21:38.411 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:21:38.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.397+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f474c076320 msgr2=0x7f474c078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.397+0000 7f4770fd9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f474c076320 0x7f474c078830 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f476c0c7390 tx=0x7f4754009040 comp rx=0 tx=0).stop 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.397+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c14cf10 msgr2=0x7f476c14d320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.397+0000 7f4770fd9640 1 --2- 172.21.15.1:0/3110419166 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f476c14cf10 0x7f476c14d320 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f476c156eb0 tx=0x7f476400ec00 comp rx=0 tx=0).stop 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.398+0000 7f476bfff640 1 -- 172.21.15.1:0/3110419166 reap_dead start 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.398+0000 7f476bfff640 1 -- 172.21.15.1:0/3110419166 reap_dead start 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.398+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 shutdown_connections 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.398+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 >> 172.21.15.1:0/3110419166 conn(0x7f476c071930 msgr2=0x7f476c157be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.398+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 shutdown_connections 2024-05-09T05:21:38.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.398+0000 7f4770fd9640 1 -- 172.21.15.1:0/3110419166 wait complete. 2024-05-09T05:21:38.414 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:21:38.414 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:21:38.414 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:21:38.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.597+0000 7f11348ee640 1 Processor -- start 2024-05-09T05:21:38.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.597+0000 7f11348ee640 1 -- start start 2024-05-09T05:21:38.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f11348ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f11300c53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f11348ee640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11300c6e00 con 0x7f11300c89b0 2024-05-09T05:21:38.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f11348ee640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11300744e0 con 0x7f11300c9e30 2024-05-09T05:21:38.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f11348ee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f11300c76c0 con 0x7f11300c8090 2024-05-09T05:21:38.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f112dd9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f11300c53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f112e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f11300c89b0 0x7f113015ad90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:45842/0 (socket says 172.21.15.1:45842) 2024-05-09T05:21:38.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f112e59c640 1 -- 172.21.15.1:0/732726027 learned_addr learned my addr 172.21.15.1:0/732726027 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:21:38.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f112dd9b640 1 -- 172.21.15.1:0/732726027 >> v1:172.21.15.196:6789/0 conn(0x7f11300c9e30 legacy=0x7f11300736f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f112dd9b640 1 -- 172.21.15.1:0/732726027 >> v1:172.21.15.1:6789/0 conn(0x7f11300c89b0 legacy=0x7f113015ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.600+0000 7f112dd9b640 1 -- 172.21.15.1:0/732726027 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11300747c0 con 0x7f11300c8090 2024-05-09T05:21:38.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.602+0000 7f112dd9b640 1 --2- 172.21.15.1:0/732726027 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f11300c53c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1120009dc0 tx=0x7f112002a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1a6290341048a7bf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.602+0000 7f112d59a640 1 -- 172.21.15.1:0/732726027 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120007740 con 0x7f11300c8090 2024-05-09T05:21:38.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.602+0000 7f112d59a640 1 -- 172.21.15.1:0/732726027 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f11200078e0 con 0x7f11300c8090 2024-05-09T05:21:38.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.603+0000 7f112d59a640 1 -- 172.21.15.1:0/732726027 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1120007cb0 con 0x7f11300c8090 2024-05-09T05:21:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.604+0000 7f11348ee640 1 -- 172.21.15.1:0/732726027 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 msgr2=0x7f11300c53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:38.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.604+0000 7f11348ee640 1 --2- 172.21.15.1:0/732726027 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f11300c53c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1120009dc0 tx=0x7f112002a6d0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.604+0000 7f11348ee640 1 -- 172.21.15.1:0/732726027 shutdown_connections 2024-05-09T05:21:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.604+0000 7f11348ee640 1 --2- 172.21.15.1:0/732726027 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f11300c53c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.604+0000 7f11348ee640 1 -- 172.21.15.1:0/732726027 >> 172.21.15.1:0/732726027 conn(0x7f11300caa40 msgr2=0x7f11300cae50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.604+0000 7f11348ee640 1 -- 172.21.15.1:0/732726027 shutdown_connections 2024-05-09T05:21:38.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.604+0000 7f11348ee640 1 -- 172.21.15.1:0/732726027 wait complete. 2024-05-09T05:21:38.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.607+0000 7f11348ee640 1 Processor -- start 2024-05-09T05:21:38.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.607+0000 7f11348ee640 1 -- start start 2024-05-09T05:21:38.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.609+0000 7f11348ee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f113015d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.609+0000 7f11348ee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11300c89b0 0x7f113014d470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.610+0000 7f112e59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f113015d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.610+0000 7f112e59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f113015d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:49328/0 (socket says 172.21.15.1:49328) 2024-05-09T05:21:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.610+0000 7f112e59c640 1 -- 172.21.15.1:0/4105073311 learned_addr learned my addr 172.21.15.1:0/4105073311 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.610+0000 7f11348ee640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11300c9e30 0x7f11301626c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.610+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11300c7350 con 0x7f11300c9e30 2024-05-09T05:21:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.610+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f11300c5900 con 0x7f11300c89b0 2024-05-09T05:21:38.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.610+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1130073e00 con 0x7f11300c8090 2024-05-09T05:21:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.611+0000 7f112e59c640 1 -- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11300c89b0 msgr2=0x7f113014d470 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:21:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.611+0000 7f112e59c640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11300c89b0 0x7f113014d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.611+0000 7f112e59c640 1 -- 172.21.15.1:0/4105073311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11300c9e30 msgr2=0x7f11301626c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.611+0000 7f112e59c640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11300c9e30 0x7f11301626c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.611+0000 7f112e59c640 1 -- 172.21.15.1:0/4105073311 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113015de40 con 0x7f11300c8090 2024-05-09T05:21:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.611+0000 7f112e59c640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f113015d900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f112800c400 tx=0x7f112800c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.612+0000 7f1126ffd640 1 -- 172.21.15.1:0/4105073311 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128017070 con 0x7f11300c8090 2024-05-09T05:21:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.612+0000 7f1126ffd640 1 -- 172.21.15.1:0/4105073311 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1128005d00 con 0x7f11300c8090 2024-05-09T05:21:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.612+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f113015b470 con 0x7f11300c8090 2024-05-09T05:21:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.612+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f113015ba30 con 0x7f11300c8090 2024-05-09T05:21:38.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.613+0000 7f1126ffd640 1 -- 172.21.15.1:0/4105073311 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1128007310 con 0x7f11300c8090 2024-05-09T05:21:38.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.615+0000 7f1126ffd640 1 -- 172.21.15.1:0/4105073311 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f112801a070 con 0x7f11300c8090 2024-05-09T05:21:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.615+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1114005290 con 0x7f11300c8090 2024-05-09T05:21:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.616+0000 7f1126ffd640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11100763a0 0x7f11100788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:38.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.616+0000 7f1126ffd640 1 -- 172.21.15.1:0/4105073311 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1128096af0 con 0x7f11300c8090 2024-05-09T05:21:38.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.616+0000 7f112dd9b640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11100763a0 0x7f11100788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:38.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.617+0000 7f112dd9b640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11100763a0 0x7f11100788b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1120034000 tx=0x7f112002f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:38.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:38.623+0000 7f1126ffd640 1 -- 172.21.15.1:0/4105073311 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1128060320 con 0x7f11300c8090 2024-05-09T05:21:39.126 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.129+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7f1114005d90 con 0x7f11300c8090 2024-05-09T05:21:39.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.133+0000 7f1126ffd640 1 -- 172.21.15.1:0/4105073311 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7f112805fc20 con 0x7f11300c8090 2024-05-09T05:21:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.136+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11100763a0 msgr2=0x7f11100788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.136+0000 7f11348ee640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11100763a0 0x7f11100788b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f1120034000 tx=0x7f112002f000 comp rx=0 tx=0).stop 2024-05-09T05:21:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.136+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 msgr2=0x7f113015d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:39.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.136+0000 7f11348ee640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f113015d900 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f112800c400 tx=0x7f112800c8c0 comp rx=0 tx=0).stop 2024-05-09T05:21:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.137+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 shutdown_connections 2024-05-09T05:21:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.137+0000 7f11348ee640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f11100763a0 0x7f11100788b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.137+0000 7f11348ee640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11300c89b0 0x7f113014d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.137+0000 7f11348ee640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11300c9e30 0x7f11301626c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.137+0000 7f11348ee640 1 --2- 172.21.15.1:0/4105073311 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11300c8090 0x7f113015d900 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.137+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 >> 172.21.15.1:0/4105073311 conn(0x7f11300caa40 msgr2=0x7f113015a150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:39.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.138+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 shutdown_connections 2024-05-09T05:21:39.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:39.138+0000 7f11348ee640 1 -- 172.21.15.1:0/4105073311 wait complete. 2024-05-09T05:21:39.153 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:21:49.154 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:21:49.154 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:21:49.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.341+0000 7f272e8a8640 1 Processor -- start 2024-05-09T05:21:49.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.341+0000 7f272e8a8640 1 -- start start 2024-05-09T05:21:49.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.344+0000 7f272e8a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27280c8a60 0x7f272814f690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:49.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.344+0000 7f272e8a8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f272806da90 0x7f2728073660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:49.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.344+0000 7f27277fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f272806da90 0x7f2728073660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:49.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.345+0000 7f27277fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f272806da90 0x7f2728073660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37252/0 (socket says 172.21.15.1:37252) 2024-05-09T05:21:49.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.345+0000 7f27277fe640 1 -- 172.21.15.1:0/2439894004 learned_addr learned my addr 172.21.15.1:0/2439894004 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:49.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.348+0000 7f272e8a8640 1 -- 172.21.15.1:0/2439894004 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f272814fbd0 con 0x7f2728073ba0 2024-05-09T05:21:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.348+0000 7f272e8a8640 1 -- 172.21.15.1:0/2439894004 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f27280c8f30 con 0x7f272806da90 2024-05-09T05:21:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.348+0000 7f272e8a8640 1 -- 172.21.15.1:0/2439894004 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f27280c5710 con 0x7f27280c8a60 2024-05-09T05:21:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.348+0000 7f2726ffd640 1 -- 172.21.15.1:0/2439894004 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2547274130 0 0) 0x7f272814fbd0 con 0x7f2728073ba0 2024-05-09T05:21:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.345+0000 7f2727fff640 1 --2- 172.21.15.1:0/2439894004 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27280c8a60 0x7f272814f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:49.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.349+0000 7f2726ffd640 1 -- 172.21.15.1:0/2439894004 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2714003520 con 0x7f2728073ba0 2024-05-09T05:21:49.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.349+0000 7f2727fff640 1 -- 172.21.15.1:0/2439894004 >> v2:172.21.15.1:3300/0 conn(0x7f272806da90 msgr2=0x7f2728073660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:49.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.349+0000 7f2727fff640 1 --2- 172.21.15.1:0/2439894004 >> v2:172.21.15.1:3300/0 conn(0x7f272806da90 0x7f2728073660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.346 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.349+0000 7f2727fff640 1 -- 172.21.15.1:0/2439894004 >> v1:172.21.15.1:6789/0 conn(0x7f2728073ba0 legacy=0x7f27280c78b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.349+0000 7f2727fff640 1 -- 172.21.15.1:0/2439894004 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27280c59f0 con 0x7f27280c8a60 2024-05-09T05:21:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.351+0000 7f2727fff640 1 --2- 172.21.15.1:0/2439894004 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27280c8a60 0x7f272814f690 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2718009d90 tx=0x7f271802a660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c651ff32494db65e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.351+0000 7f2726ffd640 1 -- 172.21.15.1:0/2439894004 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718007740 con 0x7f27280c8a60 2024-05-09T05:21:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.351+0000 7f2726ffd640 1 -- 172.21.15.1:0/2439894004 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f27180078e0 con 0x7f27280c8a60 2024-05-09T05:21:49.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.351+0000 7f2726ffd640 1 -- 172.21.15.1:0/2439894004 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718007cb0 con 0x7f27280c8a60 2024-05-09T05:21:49.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.352+0000 7f272e8a8640 1 -- 172.21.15.1:0/2439894004 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27280c8a60 msgr2=0x7f272814f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:49.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.352+0000 7f272e8a8640 1 --2- 172.21.15.1:0/2439894004 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27280c8a60 0x7f272814f690 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f2718009d90 tx=0x7f271802a660 comp rx=0 tx=0).stop 2024-05-09T05:21:49.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.353+0000 7f272e8a8640 1 -- 172.21.15.1:0/2439894004 shutdown_connections 2024-05-09T05:21:49.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.353+0000 7f272e8a8640 1 --2- 172.21.15.1:0/2439894004 >> v2:172.21.15.1:3300/0 conn(0x7f272806da90 0x7f2728073660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.353+0000 7f272e8a8640 1 --2- 172.21.15.1:0/2439894004 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f27280c8a60 0x7f272814f690 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.353+0000 7f272e8a8640 1 -- 172.21.15.1:0/2439894004 >> 172.21.15.1:0/2439894004 conn(0x7f27280ce0e0 msgr2=0x7f27280ce4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:49.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.353+0000 7f272e8a8640 1 -- 172.21.15.1:0/2439894004 shutdown_connections 2024-05-09T05:21:49.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.353+0000 7f272e8a8640 1 -- 172.21.15.1:0/2439894004 wait complete. 2024-05-09T05:21:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.356+0000 7f272e8a8640 1 Processor -- start 2024-05-09T05:21:49.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.356+0000 7f272e8a8640 1 -- start start 2024-05-09T05:21:49.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.358+0000 7f272e8a8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f272806da90 0x7f27280cc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.358+0000 7f272e8a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2728073ba0 0x7f27281f3940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f272e8a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27280c8a60 0x7f27281f3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f2727fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f272806da90 0x7f27280cc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f272e8a8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f27280ca7b0 con 0x7f27280c8a60 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f272e8a8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f27280c5fb0 con 0x7f272806da90 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f272e8a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f27280ca930 con 0x7f2728073ba0 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f2727fff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f272806da90 0x7f27280cc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:54280/0 (socket says 172.21.15.1:54280) 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f2727fff640 1 -- 172.21.15.1:0/1604886963 learned_addr learned my addr 172.21.15.1:0/1604886963 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f272ce45640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27280c8a60 0x7f27281f3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f27277fe640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2728073ba0 0x7f27281f3940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:49.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f2727fff640 1 -- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2728073ba0 msgr2=0x7f27281f3940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:49.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f2727fff640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2728073ba0 0x7f27281f3940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f2727fff640 1 -- 172.21.15.1:0/1604886963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27280c8a60 msgr2=0x7f27281f3e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:49.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f2727fff640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27280c8a60 0x7f27281f3e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.359+0000 7f2727fff640 1 -- 172.21.15.1:0/1604886963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27281f45a0 con 0x7f272806da90 2024-05-09T05:21:49.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.361+0000 7f2727fff640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f272806da90 0x7f27280cc1f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2718009d90 tx=0x7f2718004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:49.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.362+0000 7f2724ff9640 1 -- 172.21.15.1:0/1604886963 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f271803f070 con 0x7f272806da90 2024-05-09T05:21:49.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.362+0000 7f2724ff9640 1 -- 172.21.15.1:0/1604886963 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2718031e40 con 0x7f272806da90 2024-05-09T05:21:49.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.362+0000 7f2724ff9640 1 -- 172.21.15.1:0/1604886963 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2718044680 con 0x7f272806da90 2024-05-09T05:21:49.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.362+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2728155430 con 0x7f272806da90 2024-05-09T05:21:49.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.362+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2728157400 con 0x7f272806da90 2024-05-09T05:21:49.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.363+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2728050870 con 0x7f272806da90 2024-05-09T05:21:49.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.365+0000 7f2724ff9640 1 -- 172.21.15.1:0/1604886963 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2718007790 con 0x7f272806da90 2024-05-09T05:21:49.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.366+0000 7f2724ff9640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2708076300 0x7f2708078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:49.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.366+0000 7f2724ff9640 1 -- 172.21.15.1:0/1604886963 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f27180b8900 con 0x7f272806da90 2024-05-09T05:21:49.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.366+0000 7f27277fe640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2708076300 0x7f2708078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:49.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.367+0000 7f27277fe640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2708076300 0x7f2708078810 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2728156380 tx=0x7f2710008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:49.368 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.371+0000 7f2724ff9640 1 -- 172.21.15.1:0/1604886963 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f2718081af0 con 0x7f272806da90 2024-05-09T05:21:49.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.889+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f27280cb640 con 0x7f272806da90 2024-05-09T05:21:49.886 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.889+0000 7f2724ff9640 1 -- 172.21.15.1:0/1604886963 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f2718085df0 con 0x7f272806da90 2024-05-09T05:21:49.886 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2708076300 msgr2=0x7f2708078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2708076300 0x7f2708078810 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f2728156380 tx=0x7f2710008040 comp rx=0 tx=0).stop 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f272806da90 msgr2=0x7f27280cc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f272806da90 0x7f27280cc1f0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2718009d90 tx=0x7f2718004a40 comp rx=0 tx=0).stop 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 shutdown_connections 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2708076300 0x7f2708078810 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f27280c8a60 0x7f27281f3e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2728073ba0 0x7f27281f3940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 --2- 172.21.15.1:0/1604886963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f272806da90 0x7f27280cc1f0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.894+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 >> 172.21.15.1:0/1604886963 conn(0x7f27280ce0e0 msgr2=0x7f272814d120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:49.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.895+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 shutdown_connections 2024-05-09T05:21:49.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:49.895+0000 7f272e8a8640 1 -- 172.21.15.1:0/1604886963 wait complete. 2024-05-09T05:21:49.909 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:21:49.909 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:21:49.909 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:21:49.909 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:21:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.097+0000 7f2cccb3f640 1 Processor -- start 2024-05-09T05:21:50.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.098+0000 7f2cccb3f640 1 -- start start 2024-05-09T05:21:50.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.100+0000 7f2cccb3f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f2cc80c46e0 0x7f2cc8159650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.100+0000 7f2cccb3f640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cc80c3be0 con 0x7f2cc80c4af0 2024-05-09T05:21:50.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.100+0000 7f2cccb3f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2cc806e890 con 0x7f2cc80c89a0 2024-05-09T05:21:50.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.100+0000 7f2cccb3f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f2cc80cb040 con 0x7f2cc80c46e0 2024-05-09T05:21:50.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.100+0000 7f2cc6d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2cc80c4af0 0x7f2cc8153490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42800/0 (socket says 172.21.15.1:42800) 2024-05-09T05:21:50.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.100+0000 7f2cc6d9d640 1 -- 172.21.15.1:0/3082493349 learned_addr learned my addr 172.21.15.1:0/3082493349 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:21:50.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.100+0000 7f2cbdd9b640 1 --2- 172.21.15.1:0/3082493349 >> v2:172.21.15.196:3300/0 conn(0x7f2cc80c46e0 0x7f2cc8159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.101+0000 7f2cbdd9b640 1 -- 172.21.15.1:0/3082493349 >> v1:172.21.15.196:6789/0 conn(0x7f2cc80c89a0 legacy=0x7f2cc80ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.101+0000 7f2cc5d9b640 1 -- 172.21.15.1:0/3082493349 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2443467842 0 0) 0x7f2cc0001490 con 0x7f2cc80c4af0 2024-05-09T05:21:50.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.101+0000 7f2cbdd9b640 1 -- 172.21.15.1:0/3082493349 >> v1:172.21.15.1:6789/0 conn(0x7f2cc80c4af0 legacy=0x7f2cc8153490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.101+0000 7f2cbdd9b640 1 -- 172.21.15.1:0/3082493349 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc80c3e60 con 0x7f2cc80c46e0 2024-05-09T05:21:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.102+0000 7f2cbdd9b640 1 --2- 172.21.15.1:0/3082493349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c46e0 0x7f2cc8159650 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2cb00097a0 tx=0x7f2cb002a1a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=17705da5466822b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.102+0000 7f2cc5d9b640 1 -- 172.21.15.1:0/3082493349 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0037070 con 0x7f2cc80c46e0 2024-05-09T05:21:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.102+0000 7f2cc5d9b640 1 -- 172.21.15.1:0/3082493349 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cb0032440 con 0x7f2cc80c46e0 2024-05-09T05:21:50.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.102+0000 7f2cc5d9b640 1 -- 172.21.15.1:0/3082493349 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb0032810 con 0x7f2cc80c46e0 2024-05-09T05:21:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.104+0000 7f2cccb3f640 1 -- 172.21.15.1:0/3082493349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c46e0 msgr2=0x7f2cc8159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.104+0000 7f2cccb3f640 1 --2- 172.21.15.1:0/3082493349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c46e0 0x7f2cc8159650 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2cb00097a0 tx=0x7f2cb002a1a0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.104+0000 7f2cccb3f640 1 -- 172.21.15.1:0/3082493349 shutdown_connections 2024-05-09T05:21:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.104+0000 7f2cccb3f640 1 --2- 172.21.15.1:0/3082493349 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c46e0 0x7f2cc8159650 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.104+0000 7f2cccb3f640 1 -- 172.21.15.1:0/3082493349 >> 172.21.15.1:0/3082493349 conn(0x7f2cc80747b0 msgr2=0x7f2cc80ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.104+0000 7f2cccb3f640 1 -- 172.21.15.1:0/3082493349 shutdown_connections 2024-05-09T05:21:50.101 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.104+0000 7f2cccb3f640 1 -- 172.21.15.1:0/3082493349 wait complete. 2024-05-09T05:21:50.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.107+0000 7f2cccb3f640 1 Processor -- start 2024-05-09T05:21:50.104 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.108+0000 7f2cccb3f640 1 -- start start 2024-05-09T05:21:50.106 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.110+0000 7f2cccb3f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2cc80c46e0 0x7f2cc80c70a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.110+0000 7f2cccb3f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c4af0 0x7f2cc8154730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.107 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.110+0000 7f2cccb3f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c89a0 0x7f2cc81f5ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cccb3f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc80c94e0 con 0x7f2cc80c4af0 2024-05-09T05:21:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cbdd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c4af0 0x7f2cc8154730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cbdd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c4af0 0x7f2cc8154730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45088/0 (socket says 172.21.15.1:45088) 2024-05-09T05:21:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cbdd9b640 1 -- 172.21.15.1:0/194023916 learned_addr learned my addr 172.21.15.1:0/194023916 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc80c9290 con 0x7f2cc80c89a0 2024-05-09T05:21:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2cc80c97c0 con 0x7f2cc80c46e0 2024-05-09T05:21:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cc6d9d640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c89a0 0x7f2cc81f5ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cc659c640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2cc80c46e0 0x7f2cc80c70a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cc6d9d640 1 -- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2cc80c46e0 msgr2=0x7f2cc80c70a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cc6d9d640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2cc80c46e0 0x7f2cc80c70a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cc6d9d640 1 -- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c4af0 msgr2=0x7f2cc8154730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cc6d9d640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c4af0 0x7f2cc8154730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.111+0000 7f2cc6d9d640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cc8154c70 con 0x7f2cc80c89a0 2024-05-09T05:21:50.109 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.112+0000 7f2cc6d9d640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c89a0 0x7f2cc81f5ea0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0012f70 tx=0x7f2cc0014420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.112+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0014da0 con 0x7f2cc80c89a0 2024-05-09T05:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.112+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2cc0004b60 con 0x7f2cc80c89a0 2024-05-09T05:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.112+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cc8152380 con 0x7f2cc80c89a0 2024-05-09T05:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.112+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc0012110 con 0x7f2cc80c89a0 2024-05-09T05:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.112+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cc80c8620 con 0x7f2cc80c89a0 2024-05-09T05:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.114+0000 7f2cccb3f640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc8154e40 0x7f2cc81f63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.110 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.114+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc8050870 con 0x7f2cc8154e40 2024-05-09T05:21:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.114+0000 7f2cc659c640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc8154e40 0x7f2cc81f63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.115+0000 7f2cc659c640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2cc80ca360 con 0x7f2cc8154e40 2024-05-09T05:21:50.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.115+0000 7f2cc659c640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc8154e40 0x7f2cc81f63e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800e390 tx=0x7f2cb800e850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.115+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2cc00122b0 con 0x7f2cc80c89a0 2024-05-09T05:21:50.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.116+0000 7f2cbf7fe640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2ca8076320 0x7f2ca8078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.116+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2cc009f7f0 con 0x7f2cc80c89a0 2024-05-09T05:21:50.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.116+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8019070 con 0x7f2cc8154e40 2024-05-09T05:21:50.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.116+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f2cb80052e0 con 0x7f2cc8154e40 2024-05-09T05:21:50.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.116+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc8154e40 msgr2=0x7f2cc81f63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.116+0000 7f2cbf7fe640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc8154e40 0x7f2cc81f63e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800e390 tx=0x7f2cb800e850 comp rx=0 tx=0).stop 2024-05-09T05:21:50.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.116+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc8154e40 msgr2=0x7f2cc81f63e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:50.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.116+0000 7f2cbf7fe640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc8154e40 0x7f2cc81f63e0 secure :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800e390 tx=0x7f2cb800e850 comp rx=0 tx=0).stop 2024-05-09T05:21:50.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.117+0000 7f2cbdd9b640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2ca8076320 0x7f2ca8078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.120 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.124+0000 7f2cbdd9b640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2ca8076320 0x7f2ca8078830 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2cb00097a0 tx=0x7f2cb002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.131+0000 7f2cccb3f640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c7850 0x7f2cc8159a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.131+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2cc80cad40 con 0x7f2cc80c7850 2024-05-09T05:21:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.131+0000 7f2cc659c640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c7850 0x7f2cc8159a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.132+0000 7f2cc659c640 1 -- 172.21.15.1:0/194023916 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2cc80caa60 con 0x7f2cc80c7850 2024-05-09T05:21:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.132+0000 7f2cc659c640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c7850 0x7f2cc8159a30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800e390 tx=0x7f2cb8001080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.132+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cb8019040 con 0x7f2cc80c7850 2024-05-09T05:21:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.133+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7f2cb8007020 con 0x7f2cc80c7850 2024-05-09T05:21:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.133+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c7850 msgr2=0x7f2cc8159a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.133+0000 7f2cbf7fe640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c7850 0x7f2cc8159a30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800e390 tx=0x7f2cb8001080 comp rx=0 tx=0).stop 2024-05-09T05:21:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.133+0000 7f2cbf7fe640 1 -- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c7850 msgr2=0x7f2cc8159a30 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:50.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.133+0000 7f2cbf7fe640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2cc80c7850 0x7f2cc8159a30 secure :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2cb800e390 tx=0x7f2cb8001080 comp rx=0 tx=0).stop 2024-05-09T05:21:50.130 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:21:50.130 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 126, 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 98692, 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.131 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 15, 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.132 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.133 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:50.134 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.135 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:50.136 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.137 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:21:50.138 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.139 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.140 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:21:50.141 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.142 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.143 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:21:50.144 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.134+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2ca8076320 msgr2=0x7f2ca8078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.134+0000 7f2cccb3f640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2ca8076320 0x7f2ca8078830 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f2cb00097a0 tx=0x7f2cb002e000 comp rx=0 tx=0).stop 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.134+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c89a0 msgr2=0x7f2cc81f5ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.134+0000 7f2cc6d9d640 1 -- 172.21.15.1:0/194023916 reap_dead start 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.134+0000 7f2cccb3f640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c89a0 0x7f2cc81f5ea0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2cc0012f70 tx=0x7f2cc0014420 comp rx=0 tx=0).stop 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.134+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 shutdown_connections 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.135+0000 7f2cccb3f640 1 --2- 172.21.15.1:0/194023916 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2cc80c89a0 0x7f2cc81f5ea0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.135+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 >> 172.21.15.1:0/194023916 conn(0x7f2cc80747b0 msgr2=0x7f2cc814f650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.135+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 shutdown_connections 2024-05-09T05:21:50.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.135+0000 7f2cccb3f640 1 -- 172.21.15.1:0/194023916 wait complete. 2024-05-09T05:21:50.149 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:21:50.149 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:21:50.149 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:21:50.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.334+0000 7ff11f59e640 1 Processor -- start 2024-05-09T05:21:50.331 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.334+0000 7ff11f59e640 1 -- start start 2024-05-09T05:21:50.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.337+0000 7ff11f59e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120070370 0x7ff120070780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.337+0000 7ff11f59e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1201596f0 0x7ff120159b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11e59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120070370 0x7ff120070780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11e59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120070370 0x7ff120070780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60714/0 (socket says 172.21.15.1:60714) 2024-05-09T05:21:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11e59c640 1 -- 172.21.15.1:0/2099872013 learned_addr learned my addr 172.21.15.1:0/2099872013 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11dd9b640 1 --2- 172.21.15.1:0/2099872013 >> v2:172.21.15.1:3300/0 conn(0x7ff1201596f0 0x7ff120159b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11f59e640 1 -- 172.21.15.1:0/2099872013 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff12015ca10 con 0x7ff120073d80 2024-05-09T05:21:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11f59e640 1 -- 172.21.15.1:0/2099872013 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff120070cc0 con 0x7ff1201596f0 2024-05-09T05:21:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11f59e640 1 -- 172.21.15.1:0/2099872013 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff12015ccf0 con 0x7ff120070370 2024-05-09T05:21:50.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11e59c640 1 -- 172.21.15.1:0/2099872013 >> v2:172.21.15.1:3300/0 conn(0x7ff1201596f0 msgr2=0x7ff120159b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:50.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11e59c640 1 --2- 172.21.15.1:0/2099872013 >> v2:172.21.15.1:3300/0 conn(0x7ff1201596f0 0x7ff120159b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11e59c640 1 -- 172.21.15.1:0/2099872013 >> v1:172.21.15.1:6789/0 conn(0x7ff120073d80 legacy=0x7ff120074190 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:50.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.338+0000 7ff11e59c640 1 -- 172.21.15.1:0/2099872013 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff12015cfd0 con 0x7ff120070370 2024-05-09T05:21:50.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.339+0000 7ff11e59c640 1 --2- 172.21.15.1:0/2099872013 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120070370 0x7ff120070780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff118009ea0 tx=0x7ff11802a790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3180d4133bc5c9d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.336 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.340+0000 7ff11d59a640 1 -- 172.21.15.1:0/2099872013 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118036070 con 0x7ff120070370 2024-05-09T05:21:50.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.340+0000 7ff11d59a640 1 -- 172.21.15.1:0/2099872013 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff118031600 con 0x7ff120070370 2024-05-09T05:21:50.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.340+0000 7ff11d59a640 1 -- 172.21.15.1:0/2099872013 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1180319d0 con 0x7ff120070370 2024-05-09T05:21:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.341+0000 7ff11f59e640 1 -- 172.21.15.1:0/2099872013 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120070370 msgr2=0x7ff120070780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.341+0000 7ff11f59e640 1 --2- 172.21.15.1:0/2099872013 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120070370 0x7ff120070780 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7ff118009ea0 tx=0x7ff11802a790 comp rx=0 tx=0).stop 2024-05-09T05:21:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.341+0000 7ff11f59e640 1 -- 172.21.15.1:0/2099872013 shutdown_connections 2024-05-09T05:21:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.341+0000 7ff11f59e640 1 --2- 172.21.15.1:0/2099872013 >> v2:172.21.15.1:3300/0 conn(0x7ff1201596f0 0x7ff120159b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.341+0000 7ff11f59e640 1 --2- 172.21.15.1:0/2099872013 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120070370 0x7ff120070780 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.341+0000 7ff11f59e640 1 -- 172.21.15.1:0/2099872013 >> 172.21.15.1:0/2099872013 conn(0x7ff120071920 msgr2=0x7ff120071d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.342+0000 7ff11f59e640 1 -- 172.21.15.1:0/2099872013 shutdown_connections 2024-05-09T05:21:50.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.342+0000 7ff11f59e640 1 -- 172.21.15.1:0/2099872013 wait complete. 2024-05-09T05:21:50.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.345+0000 7ff11f59e640 1 Processor -- start 2024-05-09T05:21:50.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.348+0000 7ff11f59e640 1 -- start start 2024-05-09T05:21:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.351+0000 7ff11f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff120070370 0x7ff1200c60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.351+0000 7ff11f59e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120073d80 0x7ff12014d310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.351+0000 7ff11f59e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1201596f0 0x7ff120155350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.351+0000 7ff11f59e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff12015a040 con 0x7ff120070370 2024-05-09T05:21:50.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.352+0000 7ff11f59e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff12015d590 con 0x7ff1201596f0 2024-05-09T05:21:50.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.352+0000 7ff11f59e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff1200cbb60 con 0x7ff120073d80 2024-05-09T05:21:50.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.353+0000 7ff11e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff120070370 0x7ff1200c60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.350 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.354+0000 7ff11e59c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff120070370 0x7ff1200c60f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45134/0 (socket says 172.21.15.1:45134) 2024-05-09T05:21:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.354+0000 7ff11e59c640 1 -- 172.21.15.1:0/896716312 learned_addr learned my addr 172.21.15.1:0/896716312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.354+0000 7ff11e59c640 1 -- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120073d80 msgr2=0x7ff12014d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.354+0000 7ff11e59c640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff120073d80 0x7ff12014d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.354+0000 7ff11e59c640 1 -- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1201596f0 msgr2=0x7ff120155350 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.354+0000 7ff11e59c640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1201596f0 0x7ff120155350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.354+0000 7ff11e59c640 1 -- 172.21.15.1:0/896716312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1200c6630 con 0x7ff120070370 2024-05-09T05:21:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.355+0000 7ff11e59c640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff120070370 0x7ff1200c60f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff118034040 tx=0x7ff118004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.356+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118036050 con 0x7ff120070370 2024-05-09T05:21:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.356+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff118042070 con 0x7ff120070370 2024-05-09T05:21:50.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.356+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff118009510 con 0x7ff120070370 2024-05-09T05:21:50.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.356+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1201f6060 con 0x7ff120070370 2024-05-09T05:21:50.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.357+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1201f65c0 con 0x7ff120070370 2024-05-09T05:21:50.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.358+0000 7ff11f59e640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff120152d60 0x7ff1201f6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.358+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff1200c6b00 con 0x7ff120152d60 2024-05-09T05:21:50.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.358+0000 7ff11dd9b640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff120152d60 0x7ff1201f6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.359+0000 7ff11dd9b640 1 -- 172.21.15.1:0/896716312 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff12006f8f0 con 0x7ff120152d60 2024-05-09T05:21:50.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.359+0000 7ff11dd9b640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff120152d60 0x7ff1201f6800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff11000e770 tx=0x7ff11000ec30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.359+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff118007c70 con 0x7ff120070370 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff10effd640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff100076320 0x7ff100078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff1180b86d0 con 0x7ff120070370 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff11ed9d640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff100076320 0x7ff100078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110017070 con 0x7ff120152d60 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff110004020 con 0x7ff120152d60 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff120152d60 msgr2=0x7ff1201f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff10effd640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff120152d60 0x7ff1201f6800 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff11000e770 tx=0x7ff11000ec30 comp rx=0 tx=0).stop 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff120152d60 msgr2=0x7ff1201f6800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:50.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.360+0000 7ff10effd640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff120152d60 0x7ff1201f6800 secure :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ff11000e770 tx=0x7ff11000ec30 comp rx=0 tx=0).stop 2024-05-09T05:21:50.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.361+0000 7ff11ed9d640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff100076320 0x7ff100078830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff114006fd0 tx=0x7ff114006b40 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.372+0000 7ff11f59e640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1200c78c0 0x7ff120155890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.372+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff120150880 con 0x7ff1200c78c0 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.372+0000 7ff11dd9b640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1200c78c0 0x7ff120155890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.373+0000 7ff11dd9b640 1 -- 172.21.15.1:0/896716312 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff1200c4fb0 con 0x7ff1200c78c0 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.373+0000 7ff11dd9b640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1200c78c0 0x7ff120155890 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff110009fd0 tx=0x7ff1100046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.373+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff110017070 con 0x7ff1200c78c0 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.373+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5063 (secure 0 0 0) 0x7ff11000cbb0 con 0x7ff1200c78c0 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.373+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1200c78c0 msgr2=0x7ff120155890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.370 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.373+0000 7ff10effd640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1200c78c0 0x7ff120155890 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff110009fd0 tx=0x7ff1100046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.373+0000 7ff10effd640 1 -- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1200c78c0 msgr2=0x7ff120155890 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:50.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.373+0000 7ff10effd640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1200c78c0 0x7ff120155890 secure :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff110009fd0 tx=0x7ff1100046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.371 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:21:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:50.371 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 126, 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 260777, 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.372 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.373 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:21:50.374 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:50.375 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:21:50.376 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.377 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:50.378 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.379 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.380 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.381 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.382 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.383 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:21:50.384 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:21:50.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.374+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff100076320 msgr2=0x7ff100078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.374+0000 7ff11f59e640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff100076320 0x7ff100078830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7ff114006fd0 tx=0x7ff114006b40 comp rx=0 tx=0).stop 2024-05-09T05:21:50.385 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.374+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff120070370 msgr2=0x7ff1200c60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.374+0000 7ff11f59e640 1 --2- 172.21.15.1:0/896716312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff120070370 0x7ff1200c60f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7ff118034040 tx=0x7ff118004a40 comp rx=0 tx=0).stop 2024-05-09T05:21:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.374+0000 7ff11ed9d640 1 -- 172.21.15.1:0/896716312 reap_dead start 2024-05-09T05:21:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.374+0000 7ff11ed9d640 1 -- 172.21.15.1:0/896716312 reap_dead start 2024-05-09T05:21:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.375+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 shutdown_connections 2024-05-09T05:21:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.375+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 >> 172.21.15.1:0/896716312 conn(0x7ff120071920 msgr2=0x7ff12015a460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.375+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 shutdown_connections 2024-05-09T05:21:50.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.375+0000 7ff11f59e640 1 -- 172.21.15.1:0/896716312 wait complete. 2024-05-09T05:21:50.389 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:21:50.390 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:21:50.390 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:21:50.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.583+0000 7f0ce3a47640 1 Processor -- start 2024-05-09T05:21:50.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.584+0000 7f0ce3a47640 1 -- start start 2024-05-09T05:21:50.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.585+0000 7f0ce3a47640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c7ac0 0x7f0cdc153800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.585+0000 7f0ce3a47640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0cdc0c5210 0x7f0cdc0c9410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.585+0000 7f0ce3a47640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0cdc073e10 con 0x7f0cdc0c4e00 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.585+0000 7f0ce3a47640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0cdc074ce0 con 0x7f0cdc0c5210 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.585+0000 7f0ce3a47640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc0c5b00 con 0x7f0cdc0c7ac0 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce17e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c7ac0 0x7f0cdc153800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce17e3640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c7ac0 0x7f0cdc153800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60732/0 (socket says 172.21.15.1:60732) 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce17e3640 1 -- 172.21.15.1:0/172853510 learned_addr learned my addr 172.21.15.1:0/172853510 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce0fe2640 1 --1- 172.21.15.1:0/172853510 >> v1:172.21.15.1:6789/0 conn(0x7f0cdc0c4e00 0x7f0cdc0745d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42806/0 (socket says 172.21.15.1:42806) 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce1fe4640 1 --2- 172.21.15.1:0/172853510 >> v2:172.21.15.196:3300/0 conn(0x7f0cdc0c5210 0x7f0cdc0c9410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce17e3640 1 -- 172.21.15.1:0/172853510 >> v2:172.21.15.196:3300/0 conn(0x7f0cdc0c5210 msgr2=0x7f0cdc0c9410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce17e3640 1 --2- 172.21.15.1:0/172853510 >> v2:172.21.15.196:3300/0 conn(0x7f0cdc0c5210 0x7f0cdc0c9410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce17e3640 1 -- 172.21.15.1:0/172853510 >> v1:172.21.15.1:6789/0 conn(0x7f0cdc0c4e00 legacy=0x7f0cdc0745d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce17e3640 1 -- 172.21.15.1:0/172853510 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc0c5de0 con 0x7f0cdc0c7ac0 2024-05-09T05:21:50.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.586+0000 7f0ce1fe4640 1 --2- 172.21.15.1:0/172853510 >> v2:172.21.15.196:3300/0 conn(0x7f0cdc0c5210 0x7f0cdc0c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:21:50.584 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.587+0000 7f0ce17e3640 1 --2- 172.21.15.1:0/172853510 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c7ac0 0x7f0cdc153800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009870 tx=0x7f0cd802a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c631748698efde2a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.591+0000 7f0cd3fff640 1 -- 172.21.15.1:0/172853510 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8037070 con 0x7f0cdc0c7ac0 2024-05-09T05:21:50.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.591+0000 7f0cd3fff640 1 -- 172.21.15.1:0/172853510 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0cd8032440 con 0x7f0cdc0c7ac0 2024-05-09T05:21:50.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.591+0000 7f0cd3fff640 1 -- 172.21.15.1:0/172853510 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8032810 con 0x7f0cdc0c7ac0 2024-05-09T05:21:50.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.593+0000 7f0ce3a47640 1 -- 172.21.15.1:0/172853510 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c7ac0 msgr2=0x7f0cdc153800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.593+0000 7f0ce3a47640 1 --2- 172.21.15.1:0/172853510 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c7ac0 0x7f0cdc153800 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f0cd8009870 tx=0x7f0cd802a250 comp rx=0 tx=0).stop 2024-05-09T05:21:50.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.593+0000 7f0ce3a47640 1 -- 172.21.15.1:0/172853510 shutdown_connections 2024-05-09T05:21:50.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.593+0000 7f0ce3a47640 1 --2- 172.21.15.1:0/172853510 >> v2:172.21.15.196:3300/0 conn(0x7f0cdc0c5210 0x7f0cdc0c9410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.593+0000 7f0ce3a47640 1 --2- 172.21.15.1:0/172853510 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c7ac0 0x7f0cdc153800 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.593+0000 7f0ce3a47640 1 -- 172.21.15.1:0/172853510 >> 172.21.15.1:0/172853510 conn(0x7f0cdc070f40 msgr2=0x7f0cdc06c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:50.590 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.594+0000 7f0ce3a47640 1 -- 172.21.15.1:0/172853510 shutdown_connections 2024-05-09T05:21:50.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.594+0000 7f0ce3a47640 1 -- 172.21.15.1:0/172853510 wait complete. 2024-05-09T05:21:50.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.597+0000 7f0ce3a47640 1 Processor -- start 2024-05-09T05:21:50.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.597+0000 7f0ce3a47640 1 -- start start 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.599+0000 7f0ce3a47640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cdc0c4e00 0x7f0cdc15a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.600+0000 7f0ce3a47640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c5210 0x7f0cdc1efbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.600+0000 7f0ce3a47640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cdc0c7ac0 0x7f0cdc15d300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.600+0000 7f0ce3a47640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc0740b0 con 0x7f0cdc0c4e00 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.600+0000 7f0ce3a47640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0cdc0c63a0 con 0x7f0cdc0c7ac0 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.600+0000 7f0ce3a47640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc1594d0 con 0x7f0cdc0c5210 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.600+0000 7f0ce0fe2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c5210 0x7f0cdc1efbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.600+0000 7f0ce0fe2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c5210 0x7f0cdc1efbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60748/0 (socket says 172.21.15.1:60748) 2024-05-09T05:21:50.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.600+0000 7f0ce0fe2640 1 -- 172.21.15.1:0/3350158626 learned_addr learned my addr 172.21.15.1:0/3350158626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce17e3640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cdc0c4e00 0x7f0cdc15a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce1fe4640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cdc0c7ac0 0x7f0cdc15d300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce0fe2640 1 -- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cdc0c7ac0 msgr2=0x7f0cdc15d300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce0fe2640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0cdc0c7ac0 0x7f0cdc15d300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce0fe2640 1 -- 172.21.15.1:0/3350158626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cdc0c4e00 msgr2=0x7f0cdc15a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce0fe2640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cdc0c4e00 0x7f0cdc15a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce0fe2640 1 -- 172.21.15.1:0/3350158626 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cdc14c7b0 con 0x7f0cdc0c5210 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce17e3640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0cdc0c4e00 0x7f0cdc15a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.601+0000 7f0ce0fe2640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c5210 0x7f0cdc1efbc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00c400 tx=0x7f0ccc00c8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.602+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc0074e0 con 0x7f0cdc0c5210 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.602+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0ccc007680 con 0x7f0cdc0c5210 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.602+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc0049e0 con 0x7f0cdc0c5210 2024-05-09T05:21:50.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.602+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cdc1f0100 con 0x7f0cdc0c5210 2024-05-09T05:21:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.602+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cdc155bb0 con 0x7f0cdc0c5210 2024-05-09T05:21:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.603+0000 7f0ce3a47640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc155db0 0x7f0cdc1565e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.603+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc1ef590 con 0x7f0cdc155db0 2024-05-09T05:21:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.603+0000 7f0ce17e3640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc155db0 0x7f0cdc1565e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.604+0000 7f0ce17e3640 1 -- 172.21.15.1:0/3350158626 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0cdc0c6f20 con 0x7f0cdc155db0 2024-05-09T05:21:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.604+0000 7f0ce17e3640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc155db0 0x7f0cdc1565e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0cb4c0 tx=0x7f0cd80046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.604+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0ccc01a020 con 0x7f0cdc0c5210 2024-05-09T05:21:50.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.605+0000 7f0cd1ffb640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cc0076320 0x7f0cc0078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.605+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0ccc096ca0 con 0x7f0cdc0c5210 2024-05-09T05:21:50.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.605+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8041070 con 0x7f0cdc155db0 2024-05-09T05:21:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.605+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f0cd8038040 con 0x7f0cdc155db0 2024-05-09T05:21:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.605+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc155db0 msgr2=0x7f0cdc1565e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.605+0000 7f0cd1ffb640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc155db0 0x7f0cdc1565e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0cb4c0 tx=0x7f0cd80046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.605+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc155db0 msgr2=0x7f0cdc1565e0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.605+0000 7f0cd1ffb640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc155db0 0x7f0cdc1565e0 secure :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc0cb4c0 tx=0x7f0cd80046c0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.606+0000 7f0ce1fe4640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cc0076320 0x7f0cc0078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.610+0000 7f0ce1fe4640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cc0076320 0x7f0cc0078830 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc06ff80 tx=0x7f0cd400b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.622+0000 7f0ce3a47640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc14b3f0 0x7f0cdc14b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.622+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0cdc1561c0 con 0x7f0cdc14b3f0 2024-05-09T05:21:50.618 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.622+0000 7f0ce17e3640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc14b3f0 0x7f0cdc14b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.623+0000 7f0ce17e3640 1 -- 172.21.15.1:0/3350158626 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f0cdc06d4d0 con 0x7f0cdc14b3f0 2024-05-09T05:21:50.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.623+0000 7f0ce17e3640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc14b3f0 0x7f0cdc14b800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cd802a290 tx=0x7f0cd8009240 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.625+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd8037070 con 0x7f0cdc14b3f0 2024-05-09T05:21:50.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.625+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7f0cd803d3f0 con 0x7f0cdc14b3f0 2024-05-09T05:21:50.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.625+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc14b3f0 msgr2=0x7f0cdc14b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.625+0000 7f0cd1ffb640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc14b3f0 0x7f0cdc14b800 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cd802a290 tx=0x7f0cd8009240 comp rx=0 tx=0).stop 2024-05-09T05:21:50.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.625+0000 7f0cd1ffb640 1 -- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc14b3f0 msgr2=0x7f0cdc14b800 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:21:50.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.625+0000 7f0cd1ffb640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc14b3f0 0x7f0cdc14b800 secure :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f0cd802a290 tx=0x7f0cd8009240 comp rx=0 tx=0).stop 2024-05-09T05:21:50.624 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:21:50.624 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 126, 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 25449, 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.625 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.626 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.627 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:21:50.628 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:21:50.629 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:21:50.630 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:21:50.631 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:21:50.631 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:21:50.631 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:21:50.631 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:21:50.631 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:21:50.631 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:21:50.631 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.632 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.633 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.634 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:21:50.635 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.636 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.637 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:21:50.638 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:21:50.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.628+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cc0076320 msgr2=0x7f0cc0078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.628+0000 7f0ce3a47640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0cc0076320 0x7f0cc0078830 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc06ff80 tx=0x7f0cd400b040 comp rx=0 tx=0).stop 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.628+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c5210 msgr2=0x7f0cdc1efbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.628+0000 7f0ce3a47640 1 --2- 172.21.15.1:0/3350158626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0cdc0c5210 0x7f0cdc1efbc0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc00c400 tx=0x7f0ccc00c8c0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.628+0000 7f0ce1fe4640 1 -- 172.21.15.1:0/3350158626 reap_dead start 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.628+0000 7f0ce1fe4640 1 -- 172.21.15.1:0/3350158626 reap_dead start 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.628+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 shutdown_connections 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.628+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 >> 172.21.15.1:0/3350158626 conn(0x7f0cdc070f40 msgr2=0x7f0cdc14f070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.629+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 shutdown_connections 2024-05-09T05:21:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.629+0000 7f0ce3a47640 1 -- 172.21.15.1:0/3350158626 wait complete. 2024-05-09T05:21:50.643 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:21:50.643 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-05-09T05:21:50.644 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:21:50.644 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-05-09T05:21:50.644 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:21:50.644 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T05:21:50.644 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:21:50.664 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T05:21:50.664 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:21:50.665 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:21:50.848 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.852+0000 7fc86e772640 1 Processor -- start 2024-05-09T05:21:50.849 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.852+0000 7fc86e772640 1 -- start start 2024-05-09T05:21:50.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc86e772640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc868160d40 0x7fc8680c7680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.851 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc86e772640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc86815ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc86e772640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc8680b3cd0 con 0x7fc8680746e0 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc86e772640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc8680c33e0 con 0x7fc868160d40 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc86e772640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc8680c3100 con 0x7fc868072e20 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc867fff640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7fc868160d40 msgr2=0x7fc8680c7680 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc867fff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc868160d40 0x7fc8680c7680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc8677fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc86815ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc86cd0f640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc8680746e0 0x7fc8680ca6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42816/0 (socket says 172.21.15.1:42816) 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc86cd0f640 1 -- 172.21.15.1:0/2944962948 learned_addr learned my addr 172.21.15.1:0/2944962948 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.854+0000 7fc8677fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc86815ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60822/0 (socket says 172.21.15.1:60822) 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.855+0000 7fc866ffd640 1 -- 172.21.15.1:0/2944962948 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 783072601 0 0) 0x7fc860001490 con 0x7fc8680746e0 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.855+0000 7fc866ffd640 1 -- 172.21.15.1:0/2944962948 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc850003520 con 0x7fc8680746e0 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.855+0000 7fc8677fe640 1 -- 172.21.15.1:0/2944962948 >> v2:172.21.15.196:3300/0 conn(0x7fc868160d40 msgr2=0x7fc8680c7680 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.855+0000 7fc8677fe640 1 --2- 172.21.15.1:0/2944962948 >> v2:172.21.15.196:3300/0 conn(0x7fc868160d40 0x7fc8680c7680 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.855+0000 7fc8677fe640 1 -- 172.21.15.1:0/2944962948 >> v1:172.21.15.1:6789/0 conn(0x7fc8680746e0 legacy=0x7fc8680ca6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.852 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.855+0000 7fc8677fe640 1 -- 172.21.15.1:0/2944962948 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc8680c4160 con 0x7fc868072e20 2024-05-09T05:21:50.854 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.858+0000 7fc8677fe640 1 --2- 172.21.15.1:0/2944962948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc86815ae10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc85c0097a0 tx=0x7fc85c02a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=24975f2c0975af8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.859+0000 7fc866ffd640 1 -- 172.21.15.1:0/2944962948 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c037070 con 0x7fc868072e20 2024-05-09T05:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.859+0000 7fc866ffd640 1 -- 172.21.15.1:0/2944962948 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc85c032440 con 0x7fc868072e20 2024-05-09T05:21:50.855 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.859+0000 7fc866ffd640 1 -- 172.21.15.1:0/2944962948 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc85c032830 con 0x7fc868072e20 2024-05-09T05:21:50.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.860+0000 7fc86e772640 1 -- 172.21.15.1:0/2944962948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 msgr2=0x7fc86815ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.856 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.860+0000 7fc86e772640 1 --2- 172.21.15.1:0/2944962948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc86815ae10 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc85c0097a0 tx=0x7fc85c02a270 comp rx=0 tx=0).stop 2024-05-09T05:21:50.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.863+0000 7fc86e772640 1 -- 172.21.15.1:0/2944962948 shutdown_connections 2024-05-09T05:21:50.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.863+0000 7fc86e772640 1 --2- 172.21.15.1:0/2944962948 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc86815ae10 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.863+0000 7fc86e772640 1 --2- 172.21.15.1:0/2944962948 >> v2:172.21.15.196:3300/0 conn(0x7fc868160d40 0x7fc8680c7680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.859 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.863+0000 7fc86e772640 1 -- 172.21.15.1:0/2944962948 >> 172.21.15.1:0/2944962948 conn(0x7fc868070dd0 msgr2=0x7fc8680711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:50.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.863+0000 7fc86e772640 1 -- 172.21.15.1:0/2944962948 shutdown_connections 2024-05-09T05:21:50.860 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.863+0000 7fc86e772640 1 -- 172.21.15.1:0/2944962948 wait complete. 2024-05-09T05:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.867+0000 7fc86e772640 1 Processor -- start 2024-05-09T05:21:50.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.867+0000 7fc86e772640 1 -- start start 2024-05-09T05:21:50.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86e772640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc868152160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86e772640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8680746e0 0x7fc86814dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86e772640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc868160d40 0x7fc86806c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.865 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86e772640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc8680c7bc0 con 0x7fc868160d40 2024-05-09T05:21:50.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86e772640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc8680c4720 con 0x7fc8680746e0 2024-05-09T05:21:50.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86e772640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc86815b350 con 0x7fc868072e20 2024-05-09T05:21:50.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86cd0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc868160d40 0x7fc86806c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.866 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc867fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc868152160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86cd0f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc868160d40 0x7fc86806c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45174/0 (socket says 172.21.15.1:45174) 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc86cd0f640 1 -- 172.21.15.1:0/3526904635 learned_addr learned my addr 172.21.15.1:0/3526904635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc8677fe640 1 -- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8680746e0 msgr2=0x7fc86814dc90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.869+0000 7fc8677fe640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8680746e0 0x7fc86814dc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.870+0000 7fc867fff640 1 -- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8680746e0 msgr2=0x7fc86814dc90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.870+0000 7fc867fff640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8680746e0 0x7fc86814dc90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.870+0000 7fc867fff640 1 -- 172.21.15.1:0/3526904635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc868160d40 msgr2=0x7fc86806c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.870+0000 7fc867fff640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc868160d40 0x7fc86806c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:50.867 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.870+0000 7fc867fff640 1 -- 172.21.15.1:0/3526904635 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc868153660 con 0x7fc868072e20 2024-05-09T05:21:50.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.870+0000 7fc867fff640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc868152160 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc85800b480 tx=0x7fc85800b940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.871+0000 7fc864ff9640 1 -- 172.21.15.1:0/3526904635 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc858003db0 con 0x7fc868072e20 2024-05-09T05:21:50.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.871+0000 7fc864ff9640 1 -- 172.21.15.1:0/3526904635 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc858015070 con 0x7fc868072e20 2024-05-09T05:21:50.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.871+0000 7fc864ff9640 1 -- 172.21.15.1:0/3526904635 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc8580052b0 con 0x7fc868072e20 2024-05-09T05:21:50.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.871+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc868159d20 con 0x7fc868072e20 2024-05-09T05:21:50.868 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.871+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fc8681557b0 con 0x7fc868072e20 2024-05-09T05:21:50.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.874+0000 7fc864ff9640 1 -- 172.21.15.1:0/3526904635 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc858005450 con 0x7fc868072e20 2024-05-09T05:21:50.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.875+0000 7fc864ff9640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc854076300 0x7fc854078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:50.871 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.875+0000 7fc864ff9640 1 -- 172.21.15.1:0/3526904635 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc858097260 con 0x7fc868072e20 2024-05-09T05:21:50.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.875+0000 7fc8677fe640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc854076300 0x7fc854078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:50.872 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.876+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc848005290 con 0x7fc868072e20 2024-05-09T05:21:50.873 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.877+0000 7fc8677fe640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc854076300 0x7fc854078810 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc85c034000 tx=0x7fc85c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:50.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:50.883+0000 7fc864ff9640 1 -- 172.21.15.1:0/3526904635 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc858060bb0 con 0x7fc868072e20 2024-05-09T05:21:51.387 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":17,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:21:51.387 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.389+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc848005580 con 0x7fc868072e20 2024-05-09T05:21:51.388 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.389+0000 7fc864ff9640 1 -- 172.21.15.1:0/3526904635 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fc8580604b0 con 0x7fc868072e20 2024-05-09T05:21:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.393+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc854076300 msgr2=0x7fc854078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.393+0000 7fc86e772640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc854076300 0x7fc854078810 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fc85c034000 tx=0x7fc85c02e000 comp rx=0 tx=0).stop 2024-05-09T05:21:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.393+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 msgr2=0x7fc868152160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:51.390 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.393+0000 7fc86e772640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc868152160 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc85800b480 tx=0x7fc85800b940 comp rx=0 tx=0).stop 2024-05-09T05:21:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.395+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 shutdown_connections 2024-05-09T05:21:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.395+0000 7fc86e772640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc854076300 0x7fc854078810 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.395+0000 7fc86e772640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc8680746e0 0x7fc86814dc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.395+0000 7fc86e772640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc868160d40 0x7fc86806c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.395+0000 7fc86e772640 1 --2- 172.21.15.1:0/3526904635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc868072e20 0x7fc868152160 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.395+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 >> 172.21.15.1:0/3526904635 conn(0x7fc868070dd0 msgr2=0x7fc8680711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.395+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 shutdown_connections 2024-05-09T05:21:51.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:51.395+0000 7fc86e772640 1 -- 172.21.15.1:0/3526904635 wait complete. 2024-05-09T05:21:54.412 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:21:54.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.595+0000 7f4098f0d640 1 Processor -- start 2024-05-09T05:21:54.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.595+0000 7f4098f0d640 1 -- start start 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f4098f0d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f40940d16b0 0x7f40940d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f4098f0d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f409415ca90 0x7f409415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f4098f0d640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f409406f6d0 con 0x7f409406f1b0 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f4098f0d640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f40940cee50 con 0x7f409415ca90 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f4098f0d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f40940cd0e0 con 0x7f40940d16b0 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f409259c640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f40940d16b0 msgr2=0x7f40940d03d0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f409259c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f40940d16b0 0x7f40940d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f4091d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f409406f1b0 0x7f40940ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:42820/0 (socket says 172.21.15.1:42820) 2024-05-09T05:21:54.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.597+0000 7f4091d9b640 1 -- 172.21.15.1:0/4203924477 learned_addr learned my addr 172.21.15.1:0/4203924477 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:21:54.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.598+0000 7f4092d9d640 1 --2- 172.21.15.1:0/4203924477 >> v2:172.21.15.1:3300/0 conn(0x7f409415ca90 0x7f409415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:54.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.598+0000 7f4092d9d640 1 -- 172.21.15.1:0/4203924477 >> v2:172.21.15.196:3300/0 conn(0x7f40940d16b0 msgr2=0x7f40940d03d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:54.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.598+0000 7f4092d9d640 1 --2- 172.21.15.1:0/4203924477 >> v2:172.21.15.196:3300/0 conn(0x7f40940d16b0 0x7f40940d03d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:54.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.598+0000 7f4092d9d640 1 -- 172.21.15.1:0/4203924477 >> v1:172.21.15.1:6789/0 conn(0x7f409406f1b0 legacy=0x7f40940ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:54.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.598+0000 7f409159a640 1 -- 172.21.15.1:0/4203924477 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 206973462 0 0) 0x7f409406f6d0 con 0x7f409406f1b0 2024-05-09T05:21:54.595 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.598+0000 7f4092d9d640 1 -- 172.21.15.1:0/4203924477 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40940cc4d0 con 0x7f409415ca90 2024-05-09T05:21:54.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.600+0000 7f4092d9d640 1 --2- 172.21.15.1:0/4203924477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409415ca90 0x7f409415cea0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f409406d120 tx=0x7f408802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26e7733b2edf2b42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:54.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.600+0000 7f409159a640 1 -- 172.21.15.1:0/4203924477 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088037070 con 0x7f409415ca90 2024-05-09T05:21:54.596 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.600+0000 7f409159a640 1 -- 172.21.15.1:0/4203924477 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f4088032440 con 0x7f409415ca90 2024-05-09T05:21:54.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.600+0000 7f409159a640 1 -- 172.21.15.1:0/4203924477 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4088037070 con 0x7f409415ca90 2024-05-09T05:21:54.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.601+0000 7f4098f0d640 1 -- 172.21.15.1:0/4203924477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409415ca90 msgr2=0x7f409415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:54.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.601+0000 7f4098f0d640 1 --2- 172.21.15.1:0/4203924477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409415ca90 0x7f409415cea0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f409406d120 tx=0x7f408802a270 comp rx=0 tx=0).stop 2024-05-09T05:21:54.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.602+0000 7f4098f0d640 1 -- 172.21.15.1:0/4203924477 shutdown_connections 2024-05-09T05:21:54.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.602+0000 7f4098f0d640 1 --2- 172.21.15.1:0/4203924477 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f409415ca90 0x7f409415cea0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:54.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.602+0000 7f4098f0d640 1 --2- 172.21.15.1:0/4203924477 >> v2:172.21.15.196:3300/0 conn(0x7f40940d16b0 0x7f40940d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:54.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.602+0000 7f4098f0d640 1 -- 172.21.15.1:0/4203924477 >> 172.21.15.1:0/4203924477 conn(0x7f40940b3660 msgr2=0x7f40940b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:54.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.602+0000 7f4098f0d640 1 -- 172.21.15.1:0/4203924477 shutdown_connections 2024-05-09T05:21:54.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.602+0000 7f4098f0d640 1 -- 172.21.15.1:0/4203924477 wait complete. 2024-05-09T05:21:54.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.604+0000 7f4098f0d640 1 Processor -- start 2024-05-09T05:21:54.601 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.604+0000 7f4098f0d640 1 -- start start 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4098f0d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f409406f1b0 0x7f409414f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4098f0d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40940d16b0 0x7f40941f5d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4098f0d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f409415ca90 0x7f4094151530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4098f0d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f40940ccd70 con 0x7f40940d16b0 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4098f0d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f40940cca90 con 0x7f409415ca90 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4098f0d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f40940d0910 con 0x7f409406f1b0 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4092d9d640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f409415ca90 msgr2=0x7f4094151530 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4092d9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f409415ca90 0x7f4094151530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4091d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40940d16b0 0x7f40941f5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4091d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40940d16b0 0x7f40941f5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45202/0 (socket says 172.21.15.1:45202) 2024-05-09T05:21:54.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f4091d9b640 1 -- 172.21.15.1:0/2357203754 learned_addr learned my addr 172.21.15.1:0/2357203754 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.606+0000 7f409259c640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f409406f1b0 0x7f409414f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.607+0000 7f409259c640 1 -- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f409415ca90 msgr2=0x7f4094151530 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.607+0000 7f409259c640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f409415ca90 0x7f4094151530 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.607+0000 7f409259c640 1 -- 172.21.15.1:0/2357203754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40940d16b0 msgr2=0x7f40941f5d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:54.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.607+0000 7f409259c640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40940d16b0 0x7f40941f5d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.607+0000 7f409259c640 1 -- 172.21.15.1:0/2357203754 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f409414f780 con 0x7f409406f1b0 2024-05-09T05:21:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.607+0000 7f4091d9b640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40940d16b0 0x7f40941f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:21:54.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.608+0000 7f409259c640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f409406f1b0 0x7f409414f240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f408c00b520 tx=0x7f408c00b9e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.609+0000 7f4082ffd640 1 -- 172.21.15.1:0/2357203754 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c003bc0 con 0x7f409406f1b0 2024-05-09T05:21:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.609+0000 7f4082ffd640 1 -- 172.21.15.1:0/2357203754 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f408c015070 con 0x7f409406f1b0 2024-05-09T05:21:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.609+0000 7f4082ffd640 1 -- 172.21.15.1:0/2357203754 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f408c005070 con 0x7f409406f1b0 2024-05-09T05:21:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.610+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40941f6280 con 0x7f409406f1b0 2024-05-09T05:21:54.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.610+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f40940b6830 con 0x7f409406f1b0 2024-05-09T05:21:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.611+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4078005290 con 0x7f409406f1b0 2024-05-09T05:21:54.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.612+0000 7f4082ffd640 1 -- 172.21.15.1:0/2357203754 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f408c005210 con 0x7f409406f1b0 2024-05-09T05:21:54.609 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.613+0000 7f4082ffd640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f40740762d0 0x7f40740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.613+0000 7f4082ffd640 1 -- 172.21.15.1:0/2357203754 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f408c096d60 con 0x7f409406f1b0 2024-05-09T05:21:54.610 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.613+0000 7f4091d9b640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f40740762d0 0x7f40740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:54.611 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.615+0000 7f4091d9b640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f40740762d0 0x7f40740787e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4084007fa0 tx=0x7f4084007f10 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:54.619+0000 7f4082ffd640 1 -- 172.21.15.1:0/2357203754 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f409414f780 con 0x7f409406f1b0 2024-05-09T05:21:55.130 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.134+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f4078005850 con 0x7f409406f1b0 2024-05-09T05:21:55.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.135+0000 7f4082ffd640 1 -- 172.21.15.1:0/2357203754 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f408c0607b0 con 0x7f409406f1b0 2024-05-09T05:21:55.131 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:21:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f40740762d0 msgr2=0x7f40740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f40740762d0 0x7f40740787e0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f4084007fa0 tx=0x7f4084007f10 comp rx=0 tx=0).stop 2024-05-09T05:21:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f409406f1b0 msgr2=0x7f409414f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f409406f1b0 0x7f409414f240 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f408c00b520 tx=0x7f408c00b9e0 comp rx=0 tx=0).stop 2024-05-09T05:21:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 shutdown_connections 2024-05-09T05:21:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f40740762d0 0x7f40740787e0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f409415ca90 0x7f4094151530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:55.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f40940d16b0 0x7f40941f5d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 --2- 172.21.15.1:0/2357203754 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f409406f1b0 0x7f409414f240 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.138+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 >> 172.21.15.1:0/2357203754 conn(0x7f40940b3660 msgr2=0x7f4094075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.139+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 shutdown_connections 2024-05-09T05:21:55.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:55.139+0000 7f4098f0d640 1 -- 172.21.15.1:0/2357203754 wait complete. 2024-05-09T05:21:58.153 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:21:58.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.337+0000 7f982ffff640 1 Processor -- start 2024-05-09T05:21:58.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.337+0000 7f982ffff640 1 -- start start 2024-05-09T05:21:58.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982ffff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f98280e3fb0 0x7f9828007cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:58.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982ffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98280e43c0 0x7f9828008210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:58.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982ffff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828007130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:58.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982ffff640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f98280e3b20 con 0x7f98280e43c0 2024-05-09T05:21:58.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982ffff640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f98280e37f0 con 0x7f98280e3fb0 2024-05-09T05:21:58.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982ffff640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9828007670 con 0x7f9828006d20 2024-05-09T05:21:58.341 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982effd640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f98280e3fb0 msgr2=0x7f9828007cd0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:21:58.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982effd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f98280e3fb0 0x7f9828007cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:58.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98280e43c0 0x7f9828008210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:58.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982e7fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f98280e43c0 0x7f9828008210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:45212/0 (socket says 172.21.15.1:45212) 2024-05-09T05:21:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982e7fc640 1 -- 172.21.15.1:0/3289476910 learned_addr learned my addr 172.21.15.1:0/3289476910 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.339+0000 7f982f7fe640 1 --2- 172.21.15.1:0/3289476910 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828007130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.340+0000 7f982f7fe640 1 -- 172.21.15.1:0/3289476910 >> v2:172.21.15.196:3300/0 conn(0x7f98280e3fb0 msgr2=0x7f9828007cd0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:21:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.340+0000 7f982f7fe640 1 --2- 172.21.15.1:0/3289476910 >> v2:172.21.15.196:3300/0 conn(0x7f98280e3fb0 0x7f9828007cd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.340+0000 7f982f7fe640 1 -- 172.21.15.1:0/3289476910 >> v2:172.21.15.1:3300/0 conn(0x7f98280e43c0 msgr2=0x7f9828008210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:21:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.340+0000 7f982f7fe640 1 --2- 172.21.15.1:0/3289476910 >> v2:172.21.15.1:3300/0 conn(0x7f98280e43c0 0x7f9828008210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.340+0000 7f982f7fe640 1 -- 172.21.15.1:0/3289476910 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9828046d20 con 0x7f9828006d20 2024-05-09T05:21:58.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.341+0000 7f982f7fe640 1 --2- 172.21.15.1:0/3289476910 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828007130 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f983002e1c0 tx=0x7f98300a7940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=80d8b7c824b2c785 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.342+0000 7f982dffb640 1 -- 172.21.15.1:0/3289476910 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98300b1020 con 0x7f9828006d20 2024-05-09T05:21:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.342+0000 7f982dffb640 1 -- 172.21.15.1:0/3289476910 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f98300ad4f0 con 0x7f9828006d20 2024-05-09T05:21:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.342+0000 7f982dffb640 1 -- 172.21.15.1:0/3289476910 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98300ad8a0 con 0x7f9828006d20 2024-05-09T05:21:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.343+0000 7f982ffff640 1 -- 172.21.15.1:0/3289476910 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 msgr2=0x7f9828007130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.343+0000 7f982ffff640 1 --2- 172.21.15.1:0/3289476910 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828007130 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f983002e1c0 tx=0x7f98300a7940 comp rx=0 tx=0).stop 2024-05-09T05:21:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.344+0000 7f982ffff640 1 -- 172.21.15.1:0/3289476910 shutdown_connections 2024-05-09T05:21:58.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.344+0000 7f982ffff640 1 --2- 172.21.15.1:0/3289476910 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828007130 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.344+0000 7f982ffff640 1 --2- 172.21.15.1:0/3289476910 >> v2:172.21.15.1:3300/0 conn(0x7f98280e43c0 0x7f9828008210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.344+0000 7f982ffff640 1 --2- 172.21.15.1:0/3289476910 >> v2:172.21.15.196:3300/0 conn(0x7f98280e3fb0 0x7f9828007cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.344+0000 7f982ffff640 1 -- 172.21.15.1:0/3289476910 >> 172.21.15.1:0/3289476910 conn(0x7f9828050c40 msgr2=0x7f9828051050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:58.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.344+0000 7f982ffff640 1 -- 172.21.15.1:0/3289476910 shutdown_connections 2024-05-09T05:21:58.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.344+0000 7f982ffff640 1 -- 172.21.15.1:0/3289476910 wait complete. 2024-05-09T05:21:58.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.347+0000 7f982ffff640 1 Processor -- start 2024-05-09T05:21:58.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.347+0000 7f982ffff640 1 -- start start 2024-05-09T05:21:58.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982ffff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828177ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:58.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98280e3fb0 0x7f98280e73c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:58.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982ffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98280e43c0 0x7f98280d3c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:58.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982effd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828177ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:58.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982effd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828177ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:60860/0 (socket says 172.21.15.1:60860) 2024-05-09T05:21:58.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982effd640 1 -- 172.21.15.1:0/3210694268 learned_addr learned my addr 172.21.15.1:0/3210694268 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:21:58.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982e7fc640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98280e3fb0 0x7f98280e73c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:58.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982f7fe640 1 -- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98280e43c0 msgr2=0x7f98280d3c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:21:58.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982f7fe640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98280e43c0 0x7f98280d3c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:21:58.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.349+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f98280475a0 con 0x7f98280e3fb0 2024-05-09T05:21:58.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.350+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f98280472c0 con 0x7f98280e43c0 2024-05-09T05:21:58.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.350+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f9828047720 con 0x7f9828006d20 2024-05-09T05:21:58.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.350+0000 7f982effd640 1 -- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98280e43c0 msgr2=0x7f98280d3c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:21:58.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.350+0000 7f982effd640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98280e43c0 0x7f98280d3c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.350+0000 7f982effd640 1 -- 172.21.15.1:0/3210694268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98280e3fb0 msgr2=0x7f98280e73c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:58.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.350+0000 7f982effd640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98280e3fb0 0x7f98280e73c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.351+0000 7f982effd640 1 -- 172.21.15.1:0/3210694268 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98280d7440 con 0x7f9828006d20 2024-05-09T05:21:58.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.352+0000 7f982effd640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828177ce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f982000b580 tx=0x7f982000ba40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:58.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.352+0000 7f98177fe640 1 -- 172.21.15.1:0/3210694268 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9820013020 con 0x7f9828006d20 2024-05-09T05:21:58.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.353+0000 7f98177fe640 1 -- 172.21.15.1:0/3210694268 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9820003de0 con 0x7f9828006d20 2024-05-09T05:21:58.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.353+0000 7f98177fe640 1 -- 172.21.15.1:0/3210694268 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98200052b0 con 0x7f9828006d20 2024-05-09T05:21:58.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.353+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98280d76e0 con 0x7f9828006d20 2024-05-09T05:21:58.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.353+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f98280d9930 con 0x7f9828006d20 2024-05-09T05:21:58.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.354+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98280058f0 con 0x7f9828006d20 2024-05-09T05:21:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.357+0000 7f98177fe640 1 -- 172.21.15.1:0/3210694268 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f9820016020 con 0x7f9828006d20 2024-05-09T05:21:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.357+0000 7f98177fe640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f981c076300 0x7f981c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:21:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.358+0000 7f98177fe640 1 -- 172.21.15.1:0/3210694268 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f982005d1b0 con 0x7f9828006d20 2024-05-09T05:21:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.358+0000 7f982e7fc640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f981c076300 0x7f981c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:21:58.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.359+0000 7f982e7fc640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f981c076300 0x7f981c078810 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f982804c790 tx=0x7f9824009240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:21:58.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.363+0000 7f98177fe640 1 -- 172.21.15.1:0/3210694268 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f982001d020 con 0x7f9828006d20 2024-05-09T05:21:58.862 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.866+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f98280d78b0 con 0x7f9828006d20 2024-05-09T05:21:58.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.866+0000 7f98177fe640 1 -- 172.21.15.1:0/3210694268 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f982001d020 con 0x7f9828006d20 2024-05-09T05:21:58.863 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":126,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:21:58.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.870+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f981c076300 msgr2=0x7f981c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:58.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.870+0000 7f982ffff640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f981c076300 0x7f981c078810 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f982804c790 tx=0x7f9824009240 comp rx=0 tx=0).stop 2024-05-09T05:21:58.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.870+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 msgr2=0x7f9828177ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:21:58.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.870+0000 7f982ffff640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828177ce0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f982000b580 tx=0x7f982000ba40 comp rx=0 tx=0).stop 2024-05-09T05:21:58.869 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.871+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 shutdown_connections 2024-05-09T05:21:58.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.871+0000 7f982ffff640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f981c076300 0x7f981c078810 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.871+0000 7f982ffff640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f98280e43c0 0x7f98280d3c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.871+0000 7f982ffff640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f98280e3fb0 0x7f98280e73c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.871+0000 7f982ffff640 1 --2- 172.21.15.1:0/3210694268 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f9828006d20 0x7f9828177ce0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:21:58.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.871+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 >> 172.21.15.1:0/3210694268 conn(0x7f9828050c40 msgr2=0x7f9828035250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:21:58.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.871+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 shutdown_connections 2024-05-09T05:21:58.870 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:21:58.872+0000 7f982ffff640 1 -- 172.21.15.1:0/3210694268 wait complete. 2024-05-09T05:22:01.885 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:02.067 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.071+0000 7fbaaaac6640 1 Processor -- start 2024-05-09T05:22:02.068 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.071+0000 7fbaaaac6640 1 -- start start 2024-05-09T05:22:02.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.073+0000 7fbaaaac6640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:02.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.073+0000 7fbaa8862640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fbaa40c4ba0 legacy=0x7fbaa40c9740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:22:02.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.073+0000 7fbaaaac6640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 0x7fbaa406e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:02.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.073+0000 7fbaaaac6640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa40c3a10 con 0x7fbaa40c4ba0 2024-05-09T05:22:02.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.073+0000 7fbaaaac6640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa40c8fd0 con 0x7fbaa415afd0 2024-05-09T05:22:02.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.073+0000 7fbaaaac6640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa40c3cd0 con 0x7fbaa40c4650 2024-05-09T05:22:02.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.073+0000 7fbaa3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:02.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.073+0000 7fbaa9063640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 msgr2=0x7fbaa406e9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:02.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.074+0000 7fbaa9063640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 0x7fbaa406e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:02.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.074+0000 7fbaa3fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:44074/0 (socket says 172.21.15.1:44074) 2024-05-09T05:22:02.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.074+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 learned_addr learned my addr 172.21.15.1:0/314805557 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:02.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.074+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:22:02.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.074+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:02.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.074+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:02.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.074+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:02.270 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.274+0000 7fbaa8862640 1 -- 172.21.15.1:0/314805557 >> v1:172.21.15.196:6789/0 conn(0x7fbaa40c4ba0 legacy=0x7fbaa40c9740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:22:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.275+0000 7fbaa9063640 1 -- 172.21.15.1:0/314805557 >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 msgr2=0x7fbaa406e9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.275+0000 7fbaa9063640 1 --2- 172.21.15.1:0/314805557 >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 0x7fbaa406e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:22:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.275+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.275+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:22:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.275+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.275+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:02.271 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.275+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:22:02.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.675+0000 7fbaa8862640 1 -- 172.21.15.1:0/314805557 >> v1:172.21.15.196:6789/0 conn(0x7fbaa40c4ba0 legacy=0x7fbaa40c9740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:22:02.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.679+0000 7fbaa9063640 1 -- 172.21.15.1:0/314805557 >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 msgr2=0x7fbaa406e9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:02.675 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.679+0000 7fbaa9063640 1 --2- 172.21.15.1:0/314805557 >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 0x7fbaa406e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:22:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.679+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.679+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:22:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.679+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.679+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:02.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:02.679+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:22:03.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:03.476+0000 7fbaa8862640 1 -- 172.21.15.1:0/314805557 >> v1:172.21.15.196:6789/0 conn(0x7fbaa40c4ba0 legacy=0x7fbaa40c9740 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:22:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:03.480+0000 7fbaa9063640 1 -- 172.21.15.1:0/314805557 >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 msgr2=0x7fbaa406e9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:03.476 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:03.480+0000 7fbaa9063640 1 --2- 172.21.15.1:0/314805557 >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 0x7fbaa406e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:22:03.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:03.480+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:03.481+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:22:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:03.481+0000 7fbaa3fff640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:03.481+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:03.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:03.481+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:22:05.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.073+0000 7fbaa27fc640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 msgr2=0x7fbaa40c8660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:05.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.073+0000 7fbaa27fc640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa40c4650 0x7fbaa40c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.073+0000 7fbaa27fc640 1 -- 172.21.15.1:0/314805557 >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 msgr2=0x7fbaa406e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:05.070 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.073+0000 7fbaa27fc640 1 --2- 172.21.15.1:0/314805557 >> v2:172.21.15.196:3300/0 conn(0x7fbaa415afd0 0x7fbaa406e9e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.073+0000 7fbaa27fc640 1 -- 172.21.15.1:0/314805557 >> v1:172.21.15.196:6789/0 conn(0x7fbaa40c4ba0 legacy=0x7fbaa40c9740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:05.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.073+0000 7fbaa27fc640 1 --2- 172.21.15.1:0/314805557 >> v2:172.21.15.1:3300/0 conn(0x7fba880036a0 0x7fba88005b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.073+0000 7fbaa27fc640 1 -- 172.21.15.1:0/314805557 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbaa40c3fb0 con 0x7fba880061c0 2024-05-09T05:22:05.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.074+0000 7fbaa27fc640 1 -- 172.21.15.1:0/314805557 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbaa40c8fd0 con 0x7fba880036a0 2024-05-09T05:22:05.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.074+0000 7fbaa8862640 1 --2- 172.21.15.1:0/314805557 >> v2:172.21.15.1:3300/0 conn(0x7fba880036a0 0x7fba88005b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.074+0000 7fbaa37fe640 1 -- 172.21.15.1:0/314805557 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3306975681 0 0) 0x7fbaa40c3fb0 con 0x7fba880061c0 2024-05-09T05:22:05.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.074+0000 7fbaa37fe640 1 -- 172.21.15.1:0/314805557 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fba94003520 con 0x7fba880061c0 2024-05-09T05:22:05.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.075+0000 7fbaa8862640 1 -- 172.21.15.1:0/314805557 >> v1:172.21.15.1:6789/0 conn(0x7fba880061c0 legacy=0x7fba880086d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.075+0000 7fbaa8862640 1 -- 172.21.15.1:0/314805557 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa40c3a10 con 0x7fba880036a0 2024-05-09T05:22:05.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.076+0000 7fbaa8862640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fba88005b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba98003930 tx=0x7fba9802a8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b67c040435f5461 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:05.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.077+0000 7fbaa37fe640 1 -- 172.21.15.1:0/314805557 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba98036070 con 0x7fba880036a0 2024-05-09T05:22:05.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.077+0000 7fbaa37fe640 1 -- 172.21.15.1:0/314805557 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9802d070 con 0x7fba880036a0 2024-05-09T05:22:05.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.078+0000 7fbaaaac6640 1 -- 172.21.15.1:0/314805557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 msgr2=0x7fba88005b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.078+0000 7fbaa37fe640 1 -- 172.21.15.1:0/314805557 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba98036070 con 0x7fba880036a0 2024-05-09T05:22:05.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.078+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fba88005b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fba98003930 tx=0x7fba9802a8f0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.078+0000 7fbaa9063640 1 -- 172.21.15.1:0/314805557 reap_dead start 2024-05-09T05:22:05.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.079+0000 7fbaaaac6640 1 -- 172.21.15.1:0/314805557 shutdown_connections 2024-05-09T05:22:05.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.079+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/314805557 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fba88005b70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.079+0000 7fbaaaac6640 1 -- 172.21.15.1:0/314805557 >> 172.21.15.1:0/314805557 conn(0x7fbaa40ce290 msgr2=0x7fbaa40ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:05.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.079+0000 7fbaaaac6640 1 -- 172.21.15.1:0/314805557 shutdown_connections 2024-05-09T05:22:05.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.079+0000 7fbaaaac6640 1 -- 172.21.15.1:0/314805557 wait complete. 2024-05-09T05:22:05.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.082+0000 7fbaaaac6640 1 Processor -- start 2024-05-09T05:22:05.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.082+0000 7fbaaaac6640 1 -- start start 2024-05-09T05:22:05.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.084+0000 7fbaaaac6640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fbaa41624c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.085+0000 7fbaa8862640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fbaa41624c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.086+0000 7fbaa8862640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fbaa41624c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39550/0 (socket says 172.21.15.1:39550) 2024-05-09T05:22:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.086+0000 7fbaa8862640 1 -- 172.21.15.1:0/1176274052 learned_addr learned my addr 172.21.15.1:0/1176274052 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:05.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.086+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbaa40c74d0 0x7fbaa40c68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.086+0000 7fbaa3fff640 1 -- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbaa40c74d0 msgr2=0x7fbaa40c68e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.086+0000 7fbaa3fff640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbaa40c74d0 0x7fbaa40c68e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.086+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa4153e20 0x7fbaa4154230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.087+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa4164100 con 0x7fba880036a0 2024-05-09T05:22:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.087+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbaa40c9220 con 0x7fbaa40c74d0 2024-05-09T05:22:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.087+0000 7fbaa9063640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa4153e20 0x7fbaa4154230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.084 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.087+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbaa4164280 con 0x7fbaa4153e20 2024-05-09T05:22:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.088+0000 7fbaa9063640 1 -- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbaa40c74d0 msgr2=0x7fbaa40c68e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.088+0000 7fbaa9063640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbaa40c74d0 0x7fbaa40c68e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.088+0000 7fbaa9063640 1 -- 172.21.15.1:0/1176274052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 msgr2=0x7fbaa41624c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.088+0000 7fbaa9063640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fbaa41624c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.088+0000 7fbaa9063640 1 -- 172.21.15.1:0/1176274052 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbaa40c7d60 con 0x7fbaa4153e20 2024-05-09T05:22:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.088+0000 7fbaa8862640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fbaa41624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:22:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.088+0000 7fbaa9063640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa4153e20 0x7fbaa4154230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fba88009e70 tx=0x7fba9c00bda0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:05.085 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.089+0000 7fbaa17fa640 1 -- 172.21.15.1:0/1176274052 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c00cc70 con 0x7fbaa4153e20 2024-05-09T05:22:05.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.089+0000 7fbaa17fa640 1 -- 172.21.15.1:0/1176274052 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fba9c010030 con 0x7fbaa4153e20 2024-05-09T05:22:05.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.089+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbaa4162280 con 0x7fbaa4153e20 2024-05-09T05:22:05.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.089+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7fbaa41f7260 con 0x7fbaa4153e20 2024-05-09T05:22:05.087 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.091+0000 7fbaa17fa640 1 -- 172.21.15.1:0/1176274052 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba9c016670 con 0x7fbaa4153e20 2024-05-09T05:22:05.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.092+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbaa4050870 con 0x7fbaa4153e20 2024-05-09T05:22:05.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.092+0000 7fbaa17fa640 1 -- 172.21.15.1:0/1176274052 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fba9c0153e0 con 0x7fbaa4153e20 2024-05-09T05:22:05.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.093+0000 7fbaa17fa640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fba88076df0 0x7fba88079300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.093+0000 7fbaa8862640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fba88076df0 0x7fba88079300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.093+0000 7fbaa17fa640 1 -- 172.21.15.1:0/1176274052 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fba9c097310 con 0x7fbaa4153e20 2024-05-09T05:22:05.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.094+0000 7fbaa8862640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fba88076df0 0x7fba88079300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba980095d0 tx=0x7fba9802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:05.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.102+0000 7fbaa17fa640 1 -- 172.21.15.1:0/1176274052 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fba9c00ce10 con 0x7fbaa4153e20 2024-05-09T05:22:05.606 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.610+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fbaa415e4d0 con 0x7fbaa4153e20 2024-05-09T05:22:05.607 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.611+0000 7fbaa17fa640 1 -- 172.21.15.1:0/1176274052 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7fba9c0605a0 con 0x7fbaa4153e20 2024-05-09T05:22:05.608 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:22:05.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fba88076df0 msgr2=0x7fba88079300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fba88076df0 0x7fba88079300 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fba980095d0 tx=0x7fba9802e000 comp rx=0 tx=0).stop 2024-05-09T05:22:05.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa4153e20 msgr2=0x7fbaa4154230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa4153e20 0x7fbaa4154230 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fba88009e70 tx=0x7fba9c00bda0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.612 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 shutdown_connections 2024-05-09T05:22:05.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fba88076df0 0x7fba88079300 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbaa40c74d0 0x7fbaa40c68e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbaa4153e20 0x7fbaa4154230 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 --2- 172.21.15.1:0/1176274052 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fba880036a0 0x7fbaa41624c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.613 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.615+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 >> 172.21.15.1:0/1176274052 conn(0x7fbaa40ce290 msgr2=0x7fbaa4075ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:05.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.616+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 shutdown_connections 2024-05-09T05:22:05.614 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.616+0000 7fbaaaac6640 1 -- 172.21.15.1:0/1176274052 wait complete. 2024-05-09T05:22:05.631 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:22:05.632 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:22:05.632 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:22:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.824+0000 7f7cdea53640 1 Processor -- start 2024-05-09T05:22:05.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.825+0000 7f7cdea53640 1 -- start start 2024-05-09T05:22:05.823 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.827+0000 7f7cdea53640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80cbde0 0x7f7cd8154500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.827+0000 7f7cdea53640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7cd80c49b0 0x7f7cd80cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.827+0000 7f7cdcff0640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f7cd80d04f0 legacy=0x7f7cd80cc7c0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:22:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.827+0000 7f7cdea53640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f7cd8154a40 con 0x7f7cd80d04f0 2024-05-09T05:22:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.828+0000 7f7cdea53640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f7cd80cfb20 con 0x7f7cd80c49b0 2024-05-09T05:22:05.824 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.828+0000 7f7cdea53640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd815b170 con 0x7f7cd80cbde0 2024-05-09T05:22:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.829+0000 7f7cd7fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80cbde0 0x7f7cd8154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.829+0000 7f7ccffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f7cd80c49b0 0x7f7cd80cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.829+0000 7f7cd7fff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80cbde0 0x7f7cd8154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:44128/0 (socket says 172.21.15.1:44128) 2024-05-09T05:22:05.826 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.829+0000 7f7cd7fff640 1 -- 172.21.15.1:0/574594314 learned_addr learned my addr 172.21.15.1:0/574594314 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.830+0000 7f7cd7fff640 1 -- 172.21.15.1:0/574594314 >> v2:172.21.15.1:3300/0 conn(0x7f7cd80c49b0 msgr2=0x7f7cd80cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.830+0000 7f7cd7fff640 1 --2- 172.21.15.1:0/574594314 >> v2:172.21.15.1:3300/0 conn(0x7f7cd80c49b0 0x7f7cd80cffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.830+0000 7f7cd7fff640 1 -- 172.21.15.1:0/574594314 >> v1:172.21.15.196:6789/0 conn(0x7f7cd80d04f0 legacy=0x7f7cd80cc7c0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.830+0000 7f7cd7fff640 1 -- 172.21.15.1:0/574594314 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd80cb120 con 0x7f7cd80cbde0 2024-05-09T05:22:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.831+0000 7f7ccffff640 1 --2- 172.21.15.1:0/574594314 >> v2:172.21.15.1:3300/0 conn(0x7f7cd80c49b0 0x7f7cd80cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:22:05.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.831+0000 7f7cd7fff640 1 --2- 172.21.15.1:0/574594314 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80cbde0 0x7f7cd8154500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009870 tx=0x7f7cc802a1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=37fa44d0bf8f9422 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.832+0000 7f7cd77fe640 1 -- 172.21.15.1:0/574594314 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8037070 con 0x7f7cd80cbde0 2024-05-09T05:22:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.832+0000 7f7cd77fe640 1 -- 172.21.15.1:0/574594314 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cc8032440 con 0x7f7cd80cbde0 2024-05-09T05:22:05.828 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.832+0000 7f7cd77fe640 1 -- 172.21.15.1:0/574594314 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8032810 con 0x7f7cd80cbde0 2024-05-09T05:22:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.833+0000 7f7cdea53640 1 -- 172.21.15.1:0/574594314 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80cbde0 msgr2=0x7f7cd8154500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.833+0000 7f7cdea53640 1 --2- 172.21.15.1:0/574594314 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80cbde0 0x7f7cd8154500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f7cc8009870 tx=0x7f7cc802a1c0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.833+0000 7f7cdea53640 1 -- 172.21.15.1:0/574594314 shutdown_connections 2024-05-09T05:22:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.833+0000 7f7cdea53640 1 --2- 172.21.15.1:0/574594314 >> v2:172.21.15.1:3300/0 conn(0x7f7cd80c49b0 0x7f7cd80cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.834+0000 7f7cdea53640 1 --2- 172.21.15.1:0/574594314 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80cbde0 0x7f7cd8154500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.834+0000 7f7cdea53640 1 -- 172.21.15.1:0/574594314 >> 172.21.15.1:0/574594314 conn(0x7f7cd8068830 msgr2=0x7f7cd8068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.834+0000 7f7cdea53640 1 -- 172.21.15.1:0/574594314 shutdown_connections 2024-05-09T05:22:05.830 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.834+0000 7f7cdea53640 1 -- 172.21.15.1:0/574594314 wait complete. 2024-05-09T05:22:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.837+0000 7f7cdea53640 1 Processor -- start 2024-05-09T05:22:05.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.837+0000 7f7cdea53640 1 -- start start 2024-05-09T05:22:05.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.839+0000 7f7cdea53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd80c49b0 0x7f7cd815d550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.839+0000 7f7cdea53640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cd80cbde0 0x7f7cd8203430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.839+0000 7f7cdea53640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80d04f0 0x7f7cd8203970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.839+0000 7f7cdea53640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd80cb8b0 con 0x7f7cd80c49b0 2024-05-09T05:22:05.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.839+0000 7f7cdea53640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd8154c80 con 0x7f7cd80cbde0 2024-05-09T05:22:05.836 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.839+0000 7f7cdea53640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f7cd815b430 con 0x7f7cd80d04f0 2024-05-09T05:22:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.839+0000 7f7ccffff640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cd80cbde0 msgr2=0x7f7cd8203430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80d04f0 0x7f7cd8203970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7ccffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cd80cbde0 0x7f7cd8203430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80d04f0 0x7f7cd8203970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:44136/0 (socket says 172.21.15.1:44136) 2024-05-09T05:22:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 -- 172.21.15.1:0/4259270721 learned_addr learned my addr 172.21.15.1:0/4259270721 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:05.837 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cd7fff640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd80c49b0 0x7f7cd815d550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 -- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cd80cbde0 msgr2=0x7f7cd8203430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:05.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f7cd80cbde0 0x7f7cd8203430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 -- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd80c49b0 msgr2=0x7f7cd815d550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd80c49b0 0x7f7cd815d550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 -- 172.21.15.1:0/4259270721 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd815a9a0 con 0x7f7cd80d04f0 2024-05-09T05:22:05.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.840+0000 7f7cdcff0640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80d04f0 0x7f7cd8203970 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7cd000b990 tx=0x7f7cd000be50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:05.838 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.841+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd000cb20 con 0x7f7cd80d04f0 2024-05-09T05:22:05.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.841+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f7cd0010070 con 0x7f7cd80d04f0 2024-05-09T05:22:05.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.841+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0015470 con 0x7f7cd80d04f0 2024-05-09T05:22:05.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.841+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd80c64a0 con 0x7f7cd80d04f0 2024-05-09T05:22:05.839 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.841+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd815e270 con 0x7f7cd80d04f0 2024-05-09T05:22:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.842+0000 7f7cdea53640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8203eb0 0x7f7cd8201150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.842+0000 7f7cd7fff640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8203eb0 0x7f7cd8201150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.843+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd815cc40 con 0x7f7cd8203eb0 2024-05-09T05:22:05.840 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.843+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f7cd0015610 con 0x7f7cd80d04f0 2024-05-09T05:22:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.844+0000 7f7cd57fa640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cb8076280 0x7f7cb8078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.844+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f7cd0097870 con 0x7f7cd80d04f0 2024-05-09T05:22:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.844+0000 7f7cd7fff640 1 -- 172.21.15.1:0/4259270721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7cd8151a80 con 0x7f7cd8203eb0 2024-05-09T05:22:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.845+0000 7f7cd7fff640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8203eb0 0x7f7cd8201150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80745d0 tx=0x7f7cc80046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.845+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8042070 con 0x7f7cd8203eb0 2024-05-09T05:22:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.845+0000 7f7ccffff640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cb8076280 0x7f7cb8078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.842 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.846+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f7cc803e660 con 0x7f7cd8203eb0 2024-05-09T05:22:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.846+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8203eb0 msgr2=0x7f7cd8201150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.846+0000 7f7cd57fa640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8203eb0 0x7f7cd8201150 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80745d0 tx=0x7f7cc80046c0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.846+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8203eb0 msgr2=0x7f7cd8201150 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.846+0000 7f7cd57fa640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8203eb0 0x7f7cd8201150 secure :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f7cd80745d0 tx=0x7f7cc80046c0 comp rx=0 tx=0).stop 2024-05-09T05:22:05.843 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.847+0000 7f7ccffff640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cb8076280 0x7f7cb8078790 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0006fd0 tx=0x7f7cc0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:05.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.859+0000 7f7cdea53640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8201690 0x7f7cd8201aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:05.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.860+0000 7f7cd7fff640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8201690 0x7f7cd8201aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:05.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.860+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd8201fe0 con 0x7f7cd8201690 2024-05-09T05:22:05.857 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.860+0000 7f7cd7fff640 1 -- 172.21.15.1:0/4259270721 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f7cd80d0900 con 0x7f7cd8201690 2024-05-09T05:22:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.861+0000 7f7cd7fff640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8201690 0x7f7cd8201aa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7cc802a200 tx=0x7f7cc8008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.861+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc8037040 con 0x7f7cd8201690 2024-05-09T05:22:05.858 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5053 (secure 0 0 0) 0x7f7cc803ccb0 con 0x7f7cd8201690 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 114420, 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:22:05.859 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:05.860 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:05.861 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:05.861 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:05.861 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:05.861 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:05.861 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:05.861 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:05.862 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:22:05.863 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:05.864 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:05.865 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:05.866 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:05.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:05.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.866 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:05.866 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:22:05.866 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:05.866 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.866 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.867 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:05.868 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:05.869 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:05.870 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.870 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.870 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:05.870 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:05.871 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:05.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:05.871 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.871 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:05.871 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:22:05.871 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:05.872 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.872 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.872 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:05.872 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:22:05.873 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:05.873 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.873 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:05.873 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.873 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:22:05.873 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:22:05.874 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:05.874 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:05.874 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:05.874 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.874 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:05.874 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.874 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:22:05.874 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:22:05.875 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.875 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:05.875 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:05.875 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:05.876 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.876 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:05.876 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:22:05.876 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.876 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:05.876 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:05.876 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:22:05.876 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.877 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:22:05.878 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:22:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8201690 msgr2=0x7f7cd8201aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cd57fa640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8201690 0x7f7cd8201aa0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7cc802a200 tx=0x7f7cc8008030 comp rx=0 tx=0).stop 2024-05-09T05:22:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cd57fa640 1 -- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8201690 msgr2=0x7f7cd8201aa0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:05.881 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cd57fa640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f7cd8201690 0x7f7cd8201aa0 secure :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f7cc802a200 tx=0x7f7cc8008030 comp rx=0 tx=0).stop 2024-05-09T05:22:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cb8076280 msgr2=0x7f7cb8078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cdea53640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f7cb8076280 0x7f7cb8078790 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7cc0006fd0 tx=0x7f7cc0008040 comp rx=0 tx=0).stop 2024-05-09T05:22:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80d04f0 msgr2=0x7f7cd8203970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cdea53640 1 --2- 172.21.15.1:0/4259270721 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f7cd80d04f0 0x7f7cd8203970 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f7cd000b990 tx=0x7f7cd000be50 comp rx=0 tx=0).stop 2024-05-09T05:22:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cdcff0640 1 -- 172.21.15.1:0/4259270721 reap_dead start 2024-05-09T05:22:05.882 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cdcff0640 1 -- 172.21.15.1:0/4259270721 reap_dead start 2024-05-09T05:22:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 shutdown_connections 2024-05-09T05:22:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.862+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 >> 172.21.15.1:0/4259270721 conn(0x7f7cd8068830 msgr2=0x7f7cd80c4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.863+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 shutdown_connections 2024-05-09T05:22:05.883 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:05.863+0000 7f7cdea53640 1 -- 172.21.15.1:0/4259270721 wait complete. 2024-05-09T05:22:05.883 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:22:05.884 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:22:05.884 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:22:06.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.076+0000 7f78e4984640 1 Processor -- start 2024-05-09T05:22:06.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.076+0000 7f78e4984640 1 -- start start 2024-05-09T05:22:06.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.079+0000 7f78e4984640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f78e0073410 0x7f78e0154a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:06.075 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.079+0000 7f78e4984640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e00748a0 0x7f78e00c9500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.079+0000 7f78e4984640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f78e015b560 con 0x7f78e00c9a40 2024-05-09T05:22:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.079+0000 7f78de59c640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f78e0073410 msgr2=0x7f78e0154a60 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.079+0000 7f78ded9d640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f78e00c9a40 legacy=0x7f78e006c6f0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:22:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.079+0000 7f78de59c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f78e0073410 0x7f78e0154a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.079+0000 7f78e4984640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f78e00c9070 con 0x7f78e0073410 2024-05-09T05:22:06.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.079+0000 7f78e4984640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f78e0073da0 con 0x7f78e00748a0 2024-05-09T05:22:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.081+0000 7f78ddd9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e00748a0 0x7f78e00c9500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.081+0000 7f78ddd9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e00748a0 0x7f78e00c9500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:44144/0 (socket says 172.21.15.1:44144) 2024-05-09T05:22:06.077 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.081+0000 7f78ddd9b640 1 -- 172.21.15.1:0/3694116168 learned_addr learned my addr 172.21.15.1:0/3694116168 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:06.078 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.081+0000 7f78ddd9b640 1 -- 172.21.15.1:0/3694116168 >> v2:172.21.15.196:3300/0 conn(0x7f78e0073410 msgr2=0x7f78e0154a60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:06.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.081+0000 7f78ddd9b640 1 --2- 172.21.15.1:0/3694116168 >> v2:172.21.15.196:3300/0 conn(0x7f78e0073410 0x7f78e0154a60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.082+0000 7f78ddd9b640 1 -- 172.21.15.1:0/3694116168 >> v1:172.21.15.196:6789/0 conn(0x7f78e00c9a40 legacy=0x7f78e006c6f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:06.079 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.082+0000 7f78ddd9b640 1 -- 172.21.15.1:0/3694116168 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e0074080 con 0x7f78e00748a0 2024-05-09T05:22:06.080 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.083+0000 7f78ddd9b640 1 --2- 172.21.15.1:0/3694116168 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e00748a0 0x7f78e00c9500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78d8009e50 tx=0x7f78d802a6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55b1caefc03ed7e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:06.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.084+0000 7f78dd59a640 1 -- 172.21.15.1:0/3694116168 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8036070 con 0x7f78e00748a0 2024-05-09T05:22:06.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.084+0000 7f78dd59a640 1 -- 172.21.15.1:0/3694116168 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78d8031600 con 0x7f78e00748a0 2024-05-09T05:22:06.081 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.084+0000 7f78dd59a640 1 -- 172.21.15.1:0/3694116168 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d80319d0 con 0x7f78e00748a0 2024-05-09T05:22:06.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.086+0000 7f78e4984640 1 -- 172.21.15.1:0/3694116168 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e00748a0 msgr2=0x7f78e00c9500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:06.082 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.086+0000 7f78e4984640 1 --2- 172.21.15.1:0/3694116168 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e00748a0 0x7f78e00c9500 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f78d8009e50 tx=0x7f78d802a6a0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.086+0000 7f78e4984640 1 -- 172.21.15.1:0/3694116168 shutdown_connections 2024-05-09T05:22:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.086+0000 7f78e4984640 1 --2- 172.21.15.1:0/3694116168 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e00748a0 0x7f78e00c9500 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.086+0000 7f78e4984640 1 --2- 172.21.15.1:0/3694116168 >> v2:172.21.15.196:3300/0 conn(0x7f78e0073410 0x7f78e0154a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.086+0000 7f78e4984640 1 -- 172.21.15.1:0/3694116168 >> 172.21.15.1:0/3694116168 conn(0x7f78e00cdcd0 msgr2=0x7f78e00ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.087+0000 7f78e4984640 1 -- 172.21.15.1:0/3694116168 shutdown_connections 2024-05-09T05:22:06.083 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.087+0000 7f78e4984640 1 -- 172.21.15.1:0/3694116168 wait complete. 2024-05-09T05:22:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.090+0000 7f78e4984640 1 Processor -- start 2024-05-09T05:22:06.086 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.090+0000 7f78e4984640 1 -- start start 2024-05-09T05:22:06.088 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.092+0000 7f78e4984640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e0073410 0x7f78e00c6840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:06.089 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.092+0000 7f78e4984640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78e00748a0 0x7f78e01e50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78e4984640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78e00c9a40 0x7f78e01e5600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78ddd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78e00748a0 0x7f78e01e50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78ddd9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78e00748a0 0x7f78e01e50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:39602/0 (socket says 172.21.15.1:39602) 2024-05-09T05:22:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78de59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e0073410 0x7f78e00c6840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78de59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e0073410 0x7f78e00c6840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:44160/0 (socket says 172.21.15.1:44160) 2024-05-09T05:22:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78ddd9b640 1 -- 172.21.15.1:0/3847148822 learned_addr learned my addr 172.21.15.1:0/3847148822 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:06.090 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78ded9d640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78e00c9a40 msgr2=0x7f78e01e5600 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78ded9d640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78e00c9a40 0x7f78e01e5600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78e4984640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f78e015b1b0 con 0x7f78e00748a0 2024-05-09T05:22:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.093+0000 7f78e4984640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f78e006ce00 con 0x7f78e00c9a40 2024-05-09T05:22:06.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.094+0000 7f78e4984640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f78e00c7210 con 0x7f78e0073410 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.094+0000 7f78ddd9b640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e0073410 msgr2=0x7f78e00c6840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.094+0000 7f78ddd9b640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78e0073410 0x7f78e00c6840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.094+0000 7f78ddd9b640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78e00c9a40 msgr2=0x7f78e01e5600 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.094+0000 7f78ddd9b640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f78e00c9a40 0x7f78e01e5600 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.094+0000 7f78ddd9b640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78e01ee5f0 con 0x7f78e00748a0 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.095+0000 7f78ddd9b640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78e00748a0 0x7f78e01e50c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f78d8000f80 tx=0x7f78d8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.095+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8041020 con 0x7f78e00748a0 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.095+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f78d8037040 con 0x7f78e00748a0 2024-05-09T05:22:06.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.096+0000 7f78e4984640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78e01ee8d0 con 0x7f78e00748a0 2024-05-09T05:22:06.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.096+0000 7f78e4984640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78e01eee90 con 0x7f78e00748a0 2024-05-09T05:22:06.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.096+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d8036070 con 0x7f78e00748a0 2024-05-09T05:22:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.097+0000 7f78e4984640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4002a70 0x7f78c4004f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.098+0000 7f78e4984640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f78c4007f10 con 0x7f78c4002a70 2024-05-09T05:22:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.098+0000 7f78de59c640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4002a70 0x7f78c4004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:06.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.098+0000 7f78de59c640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f78c4008110 con 0x7f78c4002a70 2024-05-09T05:22:06.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.099+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f78d8044020 con 0x7f78e00748a0 2024-05-09T05:22:06.096 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.099+0000 7f78de59c640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4002a70 0x7f78c4004f80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f78d4003930 tx=0x7f78d4009b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.099+0000 7f78ceffd640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78c0076300 0x7f78c0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.099+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f78d80b8270 con 0x7f78e00748a0 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.100+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d4016070 con 0x7f78c4002a70 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.100+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f78d4012600 con 0x7f78c4002a70 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.100+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4002a70 msgr2=0x7f78c4004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.100+0000 7f78ceffd640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4002a70 0x7f78c4004f80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f78d4003930 tx=0x7f78d4009b00 comp rx=0 tx=0).stop 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.100+0000 7f78ded9d640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78c0076300 0x7f78c0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.100+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4002a70 msgr2=0x7f78c4004f80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.100+0000 7f78ceffd640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4002a70 0x7f78c4004f80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.101+0000 7f78ded9d640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78c0076300 0x7f78c0078810 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f78e00c6610 tx=0x7f78d0006c60 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.115+0000 7f78e4984640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4006f30 0x7f78c4007410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.115+0000 7f78e4984640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f78c400a4f0 con 0x7f78c4006f30 2024-05-09T05:22:06.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.116+0000 7f78de59c640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4006f30 0x7f78c4007410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:06.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.116+0000 7f78de59c640 1 -- 172.21.15.1:0/3847148822 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f78c4007950 con 0x7f78c4006f30 2024-05-09T05:22:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.117+0000 7f78de59c640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4006f30 0x7f78c4007410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f78d4018000 tx=0x7f78d40059d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.117+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78d4016040 con 0x7f78c4006f30 2024-05-09T05:22:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.117+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5050 (secure 0 0 0) 0x7f78d401d0d0 con 0x7f78c4006f30 2024-05-09T05:22:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.118+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4006f30 msgr2=0x7f78c4007410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.118+0000 7f78ceffd640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4006f30 0x7f78c4007410 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f78d4018000 tx=0x7f78d40059d0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.118+0000 7f78ceffd640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4006f30 msgr2=0x7f78c4007410 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:06.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.118+0000 7f78ceffd640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f78c4006f30 0x7f78c4007410 secure :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f78d4018000 tx=0x7f78d40059d0 comp rx=0 tx=0).stop 2024-05-09T05:22:06.115 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:22:06.115 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:06.115 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:06.115 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:22:06.115 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 130, 2024-05-09T05:22:06.115 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 40942, 2024-05-09T05:22:06.115 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:22:06.116 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:22:06.116 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:22:06.116 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:06.116 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:22:06.116 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:06.116 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:22:06.116 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:22:06.117 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:06.117 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:06.117 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:06.117 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:06.117 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:06.117 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:06.118 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:06.118 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:06.118 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:06.118 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:06.118 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:06.118 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:06.119 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:22:06.119 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:22:06.119 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:06.119 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:06.119 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:06.119 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:06.119 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:06.120 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:06.120 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:06.120 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:06.120 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:06.120 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:06.120 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:06.120 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:06.120 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:06.121 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:22:06.122 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:06.123 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:06.123 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:06.123 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:06.123 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:06.123 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:06.123 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:06.123 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.124 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.125 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.126 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:06.126 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:22:06.126 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:06.126 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.126 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:06.126 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.126 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:22:06.127 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:06.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.128 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.128 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:06.128 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:22:06.128 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:06.128 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.128 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:06.128 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.131 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:22:06.131 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:22:06.131 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:06.131 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:06.131 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:06.131 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.132 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.132 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:06.132 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:06.133 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:06.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:06.133 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.133 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:06.133 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:22:06.133 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:06.134 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.134 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.134 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:06.134 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:22:06.134 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:06.134 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.134 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.135 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:06.136 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:06.137 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:06.137 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.137 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:06.137 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:22:06.137 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.137 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:06.137 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:22:06.138 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:22:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ccff9640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78c0076300 msgr2=0x7f78c0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ccff9640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f78c0076300 0x7f78c0078810 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f78e00c6610 tx=0x7f78d0006c60 comp rx=0 tx=0).stop 2024-05-09T05:22:06.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ccff9640 1 -- 172.21.15.1:0/3847148822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78e00748a0 msgr2=0x7f78e01e50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ccff9640 1 --2- 172.21.15.1:0/3847148822 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f78e00748a0 0x7f78e01e50c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f78d8000f80 tx=0x7f78d8004a40 comp rx=0 tx=0).stop 2024-05-09T05:22:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ded9d640 1 -- 172.21.15.1:0/3847148822 reap_dead start 2024-05-09T05:22:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ded9d640 1 -- 172.21.15.1:0/3847148822 reap_dead start 2024-05-09T05:22:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ccff9640 1 -- 172.21.15.1:0/3847148822 shutdown_connections 2024-05-09T05:22:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ccff9640 1 -- 172.21.15.1:0/3847148822 >> 172.21.15.1:0/3847148822 conn(0x7f78e00cdcd0 msgr2=0x7f78e00ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ccff9640 1 -- 172.21.15.1:0/3847148822 shutdown_connections 2024-05-09T05:22:06.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:06.119+0000 7f78ccff9640 1 -- 172.21.15.1:0/3847148822 wait complete. 2024-05-09T05:22:06.142 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:22:06.142 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:22:06.224 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:22:06.227 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:22:26.142 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:22:26.143 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-05-09T05:22:26.143 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T05:22:26.144 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T05:22:26.148 INFO:tasks.ceph.mon.b:Started 2024-05-09T05:22:26.148 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:22:26.148 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:26.339 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.342+0000 7f5481eee640 1 Processor -- start 2024-05-09T05:22:26.340 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.342+0000 7f5481eee640 1 -- start start 2024-05-09T05:22:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.345+0000 7f5481eee640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f547c0c5960 0x7f547c06da90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:26.342 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.345+0000 7f5481eee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c070dd0 0x7f547c1530d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.345+0000 7f5481eee640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f547c1594d0 con 0x7f547c0c9d10 2024-05-09T05:22:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.346+0000 7f547bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c070dd0 0x7f547c1530d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.346+0000 7f547b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f547c0c9d10 0x7f547c0716c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:44124/0 (socket says 172.21.15.1:44124) 2024-05-09T05:22:26.343 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.346+0000 7f547b7fe640 1 -- 172.21.15.1:0/3695992434 learned_addr learned my addr 172.21.15.1:0/3695992434 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:22:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.346+0000 7f5481eee640 1 -- 172.21.15.1:0/3695992434 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f547c159da0 con 0x7f547c0c5960 2024-05-09T05:22:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.346+0000 7f5473fff640 1 --2- 172.21.15.1:0/3695992434 >> v2:172.21.15.1:3300/0 conn(0x7f547c0c5960 0x7f547c06da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.346+0000 7f5481eee640 1 -- 172.21.15.1:0/3695992434 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f547c070740 con 0x7f547c070dd0 2024-05-09T05:22:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.347+0000 7f5473fff640 1 -- 172.21.15.1:0/3695992434 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c070dd0 msgr2=0x7f547c1530d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:26.344 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.347+0000 7f5473fff640 1 --2- 172.21.15.1:0/3695992434 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c070dd0 0x7f547c1530d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.347+0000 7f5473fff640 1 -- 172.21.15.1:0/3695992434 >> v1:172.21.15.196:6789/0 conn(0x7f547c0c9d10 legacy=0x7f547c0716c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.347+0000 7f5473fff640 1 -- 172.21.15.1:0/3695992434 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c070a20 con 0x7f547c0c5960 2024-05-09T05:22:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.347+0000 7f547bfff640 1 --2- 172.21.15.1:0/3695992434 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c070dd0 0x7f547c1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:22:26.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.348+0000 7f5473fff640 1 --2- 172.21.15.1:0/3695992434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c0c5960 0x7f547c06da90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5464009a00 tx=0x7f546402a490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87ae25bfd5aa3e92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:26.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.349+0000 7f547affd640 1 -- 172.21.15.1:0/3695992434 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5464036070 con 0x7f547c0c5960 2024-05-09T05:22:26.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.349+0000 7f547affd640 1 -- 172.21.15.1:0/3695992434 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5464031600 con 0x7f547c0c5960 2024-05-09T05:22:26.347 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.349+0000 7f547affd640 1 -- 172.21.15.1:0/3695992434 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54640319d0 con 0x7f547c0c5960 2024-05-09T05:22:26.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.351+0000 7f5481eee640 1 -- 172.21.15.1:0/3695992434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c0c5960 msgr2=0x7f547c06da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:26.348 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.351+0000 7f5481eee640 1 --2- 172.21.15.1:0/3695992434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c0c5960 0x7f547c06da90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f5464009a00 tx=0x7f546402a490 comp rx=0 tx=0).stop 2024-05-09T05:22:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.351+0000 7f5481eee640 1 -- 172.21.15.1:0/3695992434 shutdown_connections 2024-05-09T05:22:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.351+0000 7f5481eee640 1 --2- 172.21.15.1:0/3695992434 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c070dd0 0x7f547c1530d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.351+0000 7f5481eee640 1 --2- 172.21.15.1:0/3695992434 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c0c5960 0x7f547c06da90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.351+0000 7f5481eee640 1 -- 172.21.15.1:0/3695992434 >> 172.21.15.1:0/3695992434 conn(0x7f547c0cec70 msgr2=0x7f547c0cf080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.352+0000 7f5481eee640 1 -- 172.21.15.1:0/3695992434 shutdown_connections 2024-05-09T05:22:26.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.352+0000 7f5481eee640 1 -- 172.21.15.1:0/3695992434 wait complete. 2024-05-09T05:22:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.355+0000 7f5481eee640 1 Processor -- start 2024-05-09T05:22:26.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.355+0000 7f5481eee640 1 -- start start 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.357+0000 7f5481eee640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c070dd0 0x7f547c14ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.358+0000 7f5481eee640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f547c0c5960 0x7f547c0cd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.358+0000 7f5481eee640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c0c9d10 0x7f547c147330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.358+0000 7f5481eee640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f547c06dfd0 con 0x7f547c070dd0 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.358+0000 7f5481eee640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f547c159a30 con 0x7f547c0c5960 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.358+0000 7f5481eee640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f547c06e150 con 0x7f547c0c9d10 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.358+0000 7f547b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c070dd0 0x7f547c14ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.358+0000 7f547b7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c070dd0 0x7f547c14ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47134/0 (socket says 172.21.15.1:47134) 2024-05-09T05:22:26.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.358+0000 7f547b7fe640 1 -- 172.21.15.1:0/3460275917 learned_addr learned my addr 172.21.15.1:0/3460275917 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f547b7fe640 1 -- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c0c9d10 msgr2=0x7f547c147330 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f547b7fe640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c0c9d10 0x7f547c147330 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f547b7fe640 1 -- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f547c0c5960 msgr2=0x7f547c0cd6b0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f547b7fe640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f547c0c5960 0x7f547c0cd6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f547b7fe640 1 -- 172.21.15.1:0/3460275917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f547c0cdbf0 con 0x7f547c070dd0 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f547b7fe640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c070dd0 0x7f547c14ed90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c3a0 tx=0x7f546c00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f5478ff9640 1 -- 172.21.15.1:0/3460275917 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c017070 con 0x7f547c070dd0 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f5478ff9640 1 -- 172.21.15.1:0/3460275917 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f546c005ce0 con 0x7f547c070dd0 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f5478ff9640 1 -- 172.21.15.1:0/3460275917 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f546c007310 con 0x7f547c070dd0 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.359+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f547c145b20 con 0x7f547c070dd0 2024-05-09T05:22:26.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.361+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f547c0ccbd0 con 0x7f547c070dd0 2024-05-09T05:22:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.363+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f547c050870 con 0x7f547c070dd0 2024-05-09T05:22:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.363+0000 7f5478ff9640 1 -- 172.21.15.1:0/3460275917 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f546c01a070 con 0x7f547c070dd0 2024-05-09T05:22:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.364+0000 7f5478ff9640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f545c076350 0x7f545c078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.364+0000 7f5478ff9640 1 -- 172.21.15.1:0/3460275917 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f546c05bb40 con 0x7f547c070dd0 2024-05-09T05:22:26.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.364+0000 7f5473fff640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f545c076350 0x7f545c078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:26.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.365+0000 7f5473fff640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f545c076350 0x7f545c078860 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5464033000 tx=0x7f5464008030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:26.369 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.371+0000 7f5478ff9640 1 -- 172.21.15.1:0/3460275917 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f547c1594d0 con 0x7f547c070dd0 2024-05-09T05:22:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.896+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f547c0cf3b0 con 0x7f547c070dd0 2024-05-09T05:22:26.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.896+0000 7f5478ff9640 1 -- 172.21.15.1:0/3460275917 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f546c05ffb0 con 0x7f547c070dd0 2024-05-09T05:22:26.894 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":22,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.900+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f545c076350 msgr2=0x7f545c078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.900+0000 7f5481eee640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f545c076350 0x7f545c078860 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f5464033000 tx=0x7f5464008030 comp rx=0 tx=0).stop 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.900+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c070dd0 msgr2=0x7f547c14ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.900+0000 7f5481eee640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c070dd0 0x7f547c14ed90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c3a0 tx=0x7f546c00c860 comp rx=0 tx=0).stop 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.900+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 shutdown_connections 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.900+0000 7f5481eee640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f545c076350 0x7f545c078860 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.900+0000 7f5481eee640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f547c0c5960 0x7f547c0cd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.901+0000 7f5481eee640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f547c0c9d10 0x7f547c147330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:26.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.901+0000 7f5481eee640 1 --2- 172.21.15.1:0/3460275917 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f547c070dd0 0x7f547c14ed90 secure :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f546c00c3a0 tx=0x7f546c00c860 comp rx=0 tx=0).stop 2024-05-09T05:22:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.901+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 >> 172.21.15.1:0/3460275917 conn(0x7f547c0cec70 msgr2=0x7f547c148620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.901+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 shutdown_connections 2024-05-09T05:22:26.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:26.902+0000 7f5481eee640 1 -- 172.21.15.1:0/3460275917 wait complete. 2024-05-09T05:22:29.924 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:30.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.111+0000 7fc721de0640 1 Processor -- start 2024-05-09T05:22:30.108 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.111+0000 7fc721de0640 1 -- start start 2024-05-09T05:22:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.114+0000 7fc721de0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c153660 0x7fc71c0c8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.114+0000 7fc720dde640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc71c0c9be0 0x7fc71c159be0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35026/0 (socket says 172.21.15.1:35026) 2024-05-09T05:22:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.114+0000 7fc721de0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 0x7fc71c0718e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.114+0000 7fc721de0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc71c153a70 con 0x7fc71c0c9be0 2024-05-09T05:22:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.114+0000 7fc721de0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc71c0c7d30 con 0x7fc71c0714d0 2024-05-09T05:22:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.114+0000 7fc721de0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc71c0c8b00 con 0x7fc71c153660 2024-05-09T05:22:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.114+0000 7fc720dde640 1 -- 172.21.15.1:0/3465717742 learned_addr learned my addr 172.21.15.1:0/3465717742 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:22:30.111 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.115+0000 7fc7215df640 1 --2- 172.21.15.1:0/3465717742 >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 0x7fc71c0718e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.115+0000 7fc71b7fe640 1 -- 172.21.15.1:0/3465717742 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2926116792 0 0) 0x7fc71c153a70 con 0x7fc71c0c9be0 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.115+0000 7fc71b7fe640 1 -- 172.21.15.1:0/3465717742 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc70c003520 con 0x7fc71c0c9be0 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.115+0000 7fc71bfff640 1 --2- 172.21.15.1:0/3465717742 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c153660 0x7fc71c0c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.115+0000 7fc7215df640 1 -- 172.21.15.1:0/3465717742 >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 msgr2=0x7fc71c0718e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.115+0000 7fc7215df640 1 -- 172.21.15.1:0/3465717742 >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 msgr2=0x7fc71c0718e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.115+0000 7fc7215df640 1 --2- 172.21.15.1:0/3465717742 >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 0x7fc71c0718e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.115+0000 7fc7215df640 1 --2- 172.21.15.1:0/3465717742 >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 0x7fc71c0718e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.116+0000 7fc71bfff640 1 -- 172.21.15.1:0/3465717742 >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 msgr2=0x7fc71c0718e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:30.113 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.116+0000 7fc71b7fe640 1 -- 172.21.15.1:0/3465717742 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2941990107 0 0) 0x7fc70c003520 con 0x7fc71c0c9be0 2024-05-09T05:22:30.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.116+0000 7fc71bfff640 1 --2- 172.21.15.1:0/3465717742 >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 0x7fc71c0718e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.116+0000 7fc71bfff640 1 -- 172.21.15.1:0/3465717742 >> v1:172.21.15.1:6789/0 conn(0x7fc71c0c9be0 legacy=0x7fc71c159be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:30.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.116+0000 7fc71bfff640 1 -- 172.21.15.1:0/3465717742 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71c0c9170 con 0x7fc71c153660 2024-05-09T05:22:30.114 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.117+0000 7fc71bfff640 1 --2- 172.21.15.1:0/3465717742 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c153660 0x7fc71c0c8380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7080097a0 tx=0x7fc70802a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=135926aa368ec249 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:30.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.118+0000 7fc71b7fe640 1 -- 172.21.15.1:0/3465717742 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc708037070 con 0x7fc71c153660 2024-05-09T05:22:30.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.118+0000 7fc71b7fe640 1 -- 172.21.15.1:0/3465717742 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc70802ac30 con 0x7fc71c153660 2024-05-09T05:22:30.115 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.118+0000 7fc71b7fe640 1 -- 172.21.15.1:0/3465717742 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc7100016f0 con 0x7fc71c153660 2024-05-09T05:22:30.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.119+0000 7fc721de0640 1 -- 172.21.15.1:0/3465717742 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c153660 msgr2=0x7fc71c0c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:30.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.119+0000 7fc721de0640 1 --2- 172.21.15.1:0/3465717742 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c153660 0x7fc71c0c8380 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7080097a0 tx=0x7fc70802a1a0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.119+0000 7fc721de0640 1 -- 172.21.15.1:0/3465717742 shutdown_connections 2024-05-09T05:22:30.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.119+0000 7fc721de0640 1 --2- 172.21.15.1:0/3465717742 >> v2:172.21.15.196:3300/0 conn(0x7fc71c0714d0 0x7fc71c0718e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.119+0000 7fc721de0640 1 --2- 172.21.15.1:0/3465717742 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c153660 0x7fc71c0c8380 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.116 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.119+0000 7fc721de0640 1 -- 172.21.15.1:0/3465717742 >> 172.21.15.1:0/3465717742 conn(0x7fc71c073b70 msgr2=0x7fc71c073f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:30.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.120+0000 7fc721de0640 1 -- 172.21.15.1:0/3465717742 shutdown_connections 2024-05-09T05:22:30.119 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.123+0000 7fc721de0640 1 -- 172.21.15.1:0/3465717742 wait complete. 2024-05-09T05:22:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.126+0000 7fc721de0640 1 Processor -- start 2024-05-09T05:22:30.122 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.126+0000 7fc721de0640 1 -- start start 2024-05-09T05:22:30.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc721de0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71c0714d0 0x7fc71c0cc5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:30.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc721de0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c0c9be0 0x7fc71c151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:30.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc720dde640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71c0714d0 0x7fc71c0cc5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:30.125 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc721de0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 0x7fc71c1478a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc721de0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c159750 con 0x7fc71c0714d0 2024-05-09T05:22:30.127 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc721de0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc71c0c88c0 con 0x7fc71c153660 2024-05-09T05:22:30.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc721de0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc71c0c9730 con 0x7fc71c0c9be0 2024-05-09T05:22:30.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc7215df640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 0x7fc71c1478a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:30.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc7215df640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 0x7fc71c1478a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33110/0 (socket says 172.21.15.1:33110) 2024-05-09T05:22:30.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc7215df640 1 -- 172.21.15.1:0/3847680362 learned_addr learned my addr 172.21.15.1:0/3847680362 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:30.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.130+0000 7fc7215df640 1 -- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 msgr2=0x7fc71c1478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:22:30.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.130+0000 7fc7215df640 1 -- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 msgr2=0x7fc71c1478a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:22:30.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.130+0000 7fc7215df640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 0x7fc71c1478a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:30.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.130+0000 7fc7215df640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 0x7fc71c1478a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:30.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.129+0000 7fc720dde640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71c0714d0 0x7fc71c0cc5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46082/0 (socket says 172.21.15.1:46082) 2024-05-09T05:22:30.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.131+0000 7fc720dde640 1 -- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c0c9be0 msgr2=0x7fc71c151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:30.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.131+0000 7fc720dde640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c0c9be0 0x7fc71c151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.131+0000 7fc720dde640 1 -- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 msgr2=0x7fc71c1478a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:30.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.131+0000 7fc720dde640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 0x7fc71c1478a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.131+0000 7fc720dde640 1 -- 172.21.15.1:0/3847680362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc71c1f0170 con 0x7fc71c0714d0 2024-05-09T05:22:30.129 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.131+0000 7fc720dde640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71c0714d0 0x7fc71c0cc5a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc71000c4d0 tx=0x7fc71000c990 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:30.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.132+0000 7fc7197fa640 1 -- 172.21.15.1:0/3847680362 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710017070 con 0x7fc71c0714d0 2024-05-09T05:22:30.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.132+0000 7fc7197fa640 1 -- 172.21.15.1:0/3847680362 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc710005d00 con 0x7fc71c0714d0 2024-05-09T05:22:30.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.132+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc71c0ccae0 con 0x7fc71c0714d0 2024-05-09T05:22:30.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.132+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc71c15bb70 con 0x7fc71c0714d0 2024-05-09T05:22:30.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.133+0000 7fc7197fa640 1 -- 172.21.15.1:0/3847680362 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc710007310 con 0x7fc71c0714d0 2024-05-09T05:22:30.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.133+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc71c076f80 con 0x7fc71c0714d0 2024-05-09T05:22:30.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.135+0000 7fc7197fa640 1 -- 172.21.15.1:0/3847680362 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc71001a070 con 0x7fc71c0714d0 2024-05-09T05:22:30.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.136+0000 7fc7197fa640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc700076300 0x7fc700078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:30.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.136+0000 7fc71bfff640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc700076300 0x7fc700078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:30.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.138+0000 7fc7197fa640 1 -- 172.21.15.1:0/3847680362 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc710096b00 con 0x7fc71c0714d0 2024-05-09T05:22:30.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.138+0000 7fc71bfff640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc700076300 0x7fc700078810 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc708034000 tx=0x7fc70802c000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:30.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.149+0000 7fc7197fa640 1 -- 172.21.15.1:0/3847680362 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc710060360 con 0x7fc71c0714d0 2024-05-09T05:22:30.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.689+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fc71c0c54a0 con 0x7fc71c0714d0 2024-05-09T05:22:30.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.690+0000 7fc7197fa640 1 -- 172.21.15.1:0/3847680362 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7fc710060360 con 0x7fc71c0714d0 2024-05-09T05:22:30.687 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":130,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":25,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:22:30.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc700076300 msgr2=0x7fc700078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:30.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc700076300 0x7fc700078810 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fc708034000 tx=0x7fc70802c000 comp rx=0 tx=0).stop 2024-05-09T05:22:30.690 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71c0714d0 msgr2=0x7fc71c0cc5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71c0714d0 0x7fc71c0cc5a0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc71000c4d0 tx=0x7fc71000c990 comp rx=0 tx=0).stop 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 shutdown_connections 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc700076300 0x7fc700078810 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc71c153660 0x7fc71c1478a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc71c0c9be0 0x7fc71c151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 --2- 172.21.15.1:0/3847680362 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc71c0714d0 0x7fc71c0cc5a0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 >> 172.21.15.1:0/3847680362 conn(0x7fc71c073b70 msgr2=0x7fc71c073f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 shutdown_connections 2024-05-09T05:22:30.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:30.694+0000 7fc721de0640 1 -- 172.21.15.1:0/3847680362 wait complete. 2024-05-09T05:22:33.711 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:33.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.892+0000 7ff1e35e0640 1 Processor -- start 2024-05-09T05:22:33.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.893+0000 7ff1e35e0640 1 -- start start 2024-05-09T05:22:33.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e35e0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 0x7ff1dc0c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e35e0640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff1dc15afd0 0x7ff1dc06e9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e35e0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff1dc0c3a10 con 0x7ff1dc0c4650 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e35e0640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff1dc0c8fd0 con 0x7ff1dc15afd0 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e35e0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff1dc0c3cd0 con 0x7ff1dc0c4ba0 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e137c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 0x7ff1dc0c8660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e137c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 0x7ff1dc0c8660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33128/0 (socket says 172.21.15.1:33128) 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e137c640 1 -- 172.21.15.1:0/252454312 learned_addr learned my addr 172.21.15.1:0/252454312 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.895+0000 7ff1e1b7d640 1 --2- 172.21.15.1:0/252454312 >> v2:172.21.15.1:3300/0 conn(0x7ff1dc15afd0 0x7ff1dc06e9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.896+0000 7ff1e137c640 1 -- 172.21.15.1:0/252454312 >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 msgr2=0x7ff1dc0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.896+0000 7ff1e137c640 1 -- 172.21.15.1:0/252454312 >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 msgr2=0x7ff1dc0c8660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.896+0000 7ff1e137c640 1 --2- 172.21.15.1:0/252454312 >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 0x7ff1dc0c8660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.896+0000 7ff1e137c640 1 --2- 172.21.15.1:0/252454312 >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 0x7ff1dc0c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:33.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.896+0000 7ff1e1b7d640 1 -- 172.21.15.1:0/252454312 >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 msgr2=0x7ff1dc0c8660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:33.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.896+0000 7ff1e1b7d640 1 --2- 172.21.15.1:0/252454312 >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 0x7ff1dc0c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:33.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.896+0000 7ff1e1b7d640 1 -- 172.21.15.1:0/252454312 >> v1:172.21.15.1:6789/0 conn(0x7ff1dc0c4650 legacy=0x7ff1dc0c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:33.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.896+0000 7ff1e1b7d640 1 -- 172.21.15.1:0/252454312 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc0c3fb0 con 0x7ff1dc15afd0 2024-05-09T05:22:33.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.898+0000 7ff1e1b7d640 1 --2- 172.21.15.1:0/252454312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc06e9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0cebb0 tx=0x7ff1cc02a0a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aee0d6b01adcdf72 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:33.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.899+0000 7ff1d3fff640 1 -- 172.21.15.1:0/252454312 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc037070 con 0x7ff1dc15afd0 2024-05-09T05:22:33.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.900+0000 7ff1d3fff640 1 -- 172.21.15.1:0/252454312 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1cc02ad30 con 0x7ff1dc15afd0 2024-05-09T05:22:33.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.900+0000 7ff1d3fff640 1 -- 172.21.15.1:0/252454312 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1cc032660 con 0x7ff1dc15afd0 2024-05-09T05:22:33.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.901+0000 7ff1e35e0640 1 -- 172.21.15.1:0/252454312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc15afd0 msgr2=0x7ff1dc06e9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:33.897 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.901+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/252454312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc06e9e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc0cebb0 tx=0x7ff1cc02a0a0 comp rx=0 tx=0).stop 2024-05-09T05:22:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.901+0000 7ff1e35e0640 1 -- 172.21.15.1:0/252454312 shutdown_connections 2024-05-09T05:22:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.901+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/252454312 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc06e9e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.901+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/252454312 >> v2:172.21.15.196:3300/0 conn(0x7ff1dc0c4ba0 0x7ff1dc0c8660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.901+0000 7ff1e35e0640 1 -- 172.21.15.1:0/252454312 >> 172.21.15.1:0/252454312 conn(0x7ff1dc0ce290 msgr2=0x7ff1dc0ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:33.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.902+0000 7ff1e35e0640 1 -- 172.21.15.1:0/252454312 shutdown_connections 2024-05-09T05:22:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.902+0000 7ff1e35e0640 1 -- 172.21.15.1:0/252454312 wait complete. 2024-05-09T05:22:33.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.905+0000 7ff1e35e0640 1 Processor -- start 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.905+0000 7ff1e35e0640 1 -- start start 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.907+0000 7ff1e35e0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1dc0c4650 0x7ff1dc163670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.907+0000 7ff1e35e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc0c4ba0 0x7ff1dc1f5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e137c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1dc0c4650 0x7ff1dc163670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e0b7b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc0c4ba0 0x7ff1dc1f5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e0b7b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc0c4ba0 0x7ff1dc1f5f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46104/0 (socket says 172.21.15.1:46104) 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e0b7b640 1 -- 172.21.15.1:0/827900659 learned_addr learned my addr 172.21.15.1:0/827900659 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc153e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:33.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc164100 con 0x7ff1dc0c4ba0 2024-05-09T05:22:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff1dc0c9220 con 0x7ff1dc15afd0 2024-05-09T05:22:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff1dc164280 con 0x7ff1dc0c4650 2024-05-09T05:22:33.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.908+0000 7ff1e1b7d640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc153e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:33.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e1b7d640 1 -- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 msgr2=0x7ff1dc153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 18 2024-05-09T05:22:33.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e1b7d640 1 -- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 msgr2=0x7ff1dc153e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:22:33.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e1b7d640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc153e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:33.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e1b7d640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc153e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:33.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e137c640 1 -- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 msgr2=0x7ff1dc153e90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e137c640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc153e90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e137c640 1 -- 172.21.15.1:0/827900659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc0c4ba0 msgr2=0x7ff1dc1f5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e137c640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc0c4ba0 0x7ff1dc1f5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.909+0000 7ff1e137c640 1 -- 172.21.15.1:0/827900659 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff1dc163bb0 con 0x7ff1dc0c4650 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.910+0000 7ff1e137c640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1dc0c4650 0x7ff1dc163670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4007b20 tx=0x7ff1d400d0d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.910+0000 7ff1d1ffb640 1 -- 172.21.15.1:0/827900659 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d401a070 con 0x7ff1dc0c4650 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.910+0000 7ff1d1ffb640 1 -- 172.21.15.1:0/827900659 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff1d400dcb0 con 0x7ff1dc0c4650 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.910+0000 7ff1d1ffb640 1 -- 172.21.15.1:0/827900659 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff1d4015640 con 0x7ff1dc0c4650 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.910+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff1dc0cbbe0 con 0x7ff1dc0c4650 2024-05-09T05:22:33.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.910+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7ff1dc156a10 con 0x7ff1dc0c4650 2024-05-09T05:22:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.912+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff1dc06a3c0 con 0x7ff1dc0c4650 2024-05-09T05:22:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.913+0000 7ff1d1ffb640 1 -- 172.21.15.1:0/827900659 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff1d40143a0 con 0x7ff1dc0c4650 2024-05-09T05:22:33.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.914+0000 7ff1d1ffb640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff1c00763d0 0x7ff1c00788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:33.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.914+0000 7ff1d1ffb640 1 -- 172.21.15.1:0/827900659 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff1d4097610 con 0x7ff1dc0c4650 2024-05-09T05:22:33.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.916+0000 7ff1e0b7b640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff1c00763d0 0x7ff1c00788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:33.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.918+0000 7ff1e0b7b640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff1c00763d0 0x7ff1c00788e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc161c60 tx=0x7ff1d8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:33.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:33.925+0000 7ff1d1ffb640 1 -- 172.21.15.1:0/827900659 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff1dc06a3c0 con 0x7ff1dc0c4650 2024-05-09T05:22:34.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.442+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7ff1dc050870 con 0x7ff1dc0c4650 2024-05-09T05:22:34.439 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.443+0000 7ff1d1ffb640 1 -- 172.21.15.1:0/827900659 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7ff1d4010070 con 0x7ff1dc0c4650 2024-05-09T05:22:34.440 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":3,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:22:34.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.446+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff1c00763d0 msgr2=0x7ff1c00788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff1c00763d0 0x7ff1c00788e0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc161c60 tx=0x7ff1d8009040 comp rx=0 tx=0).stop 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1dc0c4650 msgr2=0x7ff1dc163670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1dc0c4650 0x7ff1dc163670 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4007b20 tx=0x7ff1d400d0d0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 shutdown_connections 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff1c00763d0 0x7ff1c00788e0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff1dc15afd0 0x7ff1dc153e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff1dc0c4ba0 0x7ff1dc1f5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 --2- 172.21.15.1:0/827900659 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff1dc0c4650 0x7ff1dc163670 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 >> 172.21.15.1:0/827900659 conn(0x7ff1dc0ce290 msgr2=0x7ff1dc073210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 shutdown_connections 2024-05-09T05:22:34.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.447+0000 7ff1e35e0640 1 -- 172.21.15.1:0/827900659 wait complete. 2024-05-09T05:22:34.462 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:22:34.462 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:22:34.462 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:22:34.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.654+0000 7f355bfff640 1 Processor -- start 2024-05-09T05:22:34.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.654+0000 7f355bfff640 1 -- start start 2024-05-09T05:22:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.657+0000 7f355bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c0c8660 0x7f355c15cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.657+0000 7f355bfff640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355c0c3620 con 0x7f355c0c2c80 2024-05-09T05:22:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.657+0000 7f355bfff640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f355c0c3150 con 0x7f355c070260 2024-05-09T05:22:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.657+0000 7f355bfff640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f355c15cb40 con 0x7f355c0c8660 2024-05-09T05:22:34.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.658+0000 7f355a7fc640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c0c8660 0x7f355c15cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.658+0000 7f355b7fe640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f355c070260 0x7f355c0730b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:34450/0 (socket says 172.21.15.1:34450) 2024-05-09T05:22:34.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.658+0000 7f355b7fe640 1 -- 172.21.15.1:0/1252094225 learned_addr learned my addr 172.21.15.1:0/1252094225 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:22:34.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.661+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4256116813 0 0) 0x7f355c0c3620 con 0x7f355c0c2c80 2024-05-09T05:22:34.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.661+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3544003520 con 0x7f355c0c2c80 2024-05-09T05:22:34.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.662+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1915957158 0 0) 0x7f3544003520 con 0x7f355c0c2c80 2024-05-09T05:22:34.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.662+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c0c8660 msgr2=0x7f355c15cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:34.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.662+0000 7f3559ffb640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c0c8660 0x7f355c15cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.662+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 >> v1:172.21.15.196:6789/0 conn(0x7f355c070260 legacy=0x7f355c0730b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.662+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c15fd00 con 0x7f355c0c2c80 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.662+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f354c0016f0 con 0x7f355c0c2c80 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 >> v1:172.21.15.1:6789/0 conn(0x7f355c0c2c80 legacy=0x7f355c1535b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f3559ffb640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3544003a80 0x7f3544003ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f3559ffb640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544004800 0x7f3544004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f3559ffb640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3544005350 0x7f354400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f355c15cb40 con 0x7f3544004800 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f354400c250 con 0x7f3544005350 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f355affd640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3544003a80 0x7f3544003ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f354400c450 con 0x7f3544003a80 2024-05-09T05:22:34.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.663+0000 7f355b7fe640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3544005350 0x7f354400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.664+0000 7f355affd640 1 -- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3544005350 msgr2=0x7f354400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:34.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.664+0000 7f355affd640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3544005350 0x7f354400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.664+0000 7f355affd640 1 -- 172.21.15.1:0/1252094225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544004800 msgr2=0x7f3544004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:34.661 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.664+0000 7f355a7fc640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544004800 0x7f3544004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.664+0000 7f355affd640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544004800 0x7f3544004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.662 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.665+0000 7f355affd640 1 -- 172.21.15.1:0/1252094225 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3544003520 con 0x7f3544003a80 2024-05-09T05:22:34.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.667+0000 7f355affd640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3544003a80 0x7f3544003ef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f354c007fb0 tx=0x7f3544005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ed9cd1caeeb34b8c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.663 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.667+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c036070 con 0x7f3544003a80 2024-05-09T05:22:34.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.667+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f354c031600 con 0x7f3544003a80 2024-05-09T05:22:34.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.667+0000 7f355a7fc640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3544004800 0x7f3544004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:22:34.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.667+0000 7f3559ffb640 1 -- 172.21.15.1:0/1252094225 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c007c00 con 0x7f3544003a80 2024-05-09T05:22:34.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.668+0000 7f355bfff640 1 -- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3544003a80 msgr2=0x7f3544003ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.669+0000 7f355bfff640 1 --2- 172.21.15.1:0/1252094225 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3544003a80 0x7f3544003ef0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f354c007fb0 tx=0x7f3544005830 comp rx=0 tx=0).stop 2024-05-09T05:22:34.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.669+0000 7f355b7fe640 1 -- 172.21.15.1:0/1252094225 reap_dead start 2024-05-09T05:22:34.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.669+0000 7f355bfff640 1 -- 172.21.15.1:0/1252094225 shutdown_connections 2024-05-09T05:22:34.665 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.669+0000 7f355bfff640 1 -- 172.21.15.1:0/1252094225 >> 172.21.15.1:0/1252094225 conn(0x7f355c0cdc60 msgr2=0x7f355c0ce070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:34.666 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.670+0000 7f355bfff640 1 -- 172.21.15.1:0/1252094225 shutdown_connections 2024-05-09T05:22:34.672 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.676+0000 7f355bfff640 1 -- 172.21.15.1:0/1252094225 wait complete. 2024-05-09T05:22:34.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.680+0000 7f355bfff640 1 Processor -- start 2024-05-09T05:22:34.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.681+0000 7f355bfff640 1 -- start start 2024-05-09T05:22:34.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.684+0000 7f355bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c159f60 0x7f355c14ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.684+0000 7f355bfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f355c0c6500 0x7f355c0c6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.684+0000 7f355bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c155700 0x7f355c155b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.684+0000 7f355bfff640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f355c0c3ea0 con 0x7f355c155700 2024-05-09T05:22:34.681 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.684+0000 7f355bfff640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f355c0c3bc0 con 0x7f355c0c6500 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.684+0000 7f355bfff640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f355c0ca410 con 0x7f355c159f60 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.684+0000 7f355affd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c159f60 0x7f355c14ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355affd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c159f60 0x7f355c14ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:46520/0 (socket says 172.21.15.1:46520) 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355affd640 1 -- 172.21.15.1:0/1693193763 learned_addr learned my addr 172.21.15.1:0/1693193763 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355a7fc640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f355c0c6500 0x7f355c0c6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355b7fe640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c155700 0x7f355c155b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355affd640 1 -- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f355c0c6500 msgr2=0x7f355c0c6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355affd640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f355c0c6500 0x7f355c0c6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355affd640 1 -- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c155700 msgr2=0x7f355c155b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355affd640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c155700 0x7f355c155b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.682 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.685+0000 7f355affd640 1 -- 172.21.15.1:0/1693193763 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f355c156d70 con 0x7f355c159f60 2024-05-09T05:22:34.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.686+0000 7f355affd640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c159f60 0x7f355c14ba80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f354c007fb0 tx=0x7f355c0c3320 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.686+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c041020 con 0x7f355c159f60 2024-05-09T05:22:34.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.686+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f354c037040 con 0x7f355c159f60 2024-05-09T05:22:34.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.686+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f354c036070 con 0x7f355c159f60 2024-05-09T05:22:34.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.686+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f355c159c20 con 0x7f355c159f60 2024-05-09T05:22:34.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.686+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f355c156a90 con 0x7f355c159f60 2024-05-09T05:22:34.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.687+0000 7f355bfff640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c1571e0 0x7f355c157a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.687+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f355c156050 con 0x7f355c1571e0 2024-05-09T05:22:34.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.688+0000 7f355a7fc640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c1571e0 0x7f355c157a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.688+0000 7f355a7fc640 1 -- 172.21.15.1:0/1693193763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f355c050870 con 0x7f355c1571e0 2024-05-09T05:22:34.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.688+0000 7f355a7fc640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c1571e0 0x7f355c157a80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f355c0c9f60 tx=0x7f3550011a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.688+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f354c044020 con 0x7f355c159f60 2024-05-09T05:22:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.689+0000 7f35437fe640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f35480762d0 0x7f35480787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.689+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f354c0b79d0 con 0x7f355c159f60 2024-05-09T05:22:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.689+0000 7f355b7fe640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f35480762d0 0x7f35480787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.689+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550019020 con 0x7f355c1571e0 2024-05-09T05:22:34.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.690+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3550003610 con 0x7f355c1571e0 2024-05-09T05:22:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.690+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c1571e0 msgr2=0x7f355c157a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.690+0000 7f35437fe640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c1571e0 0x7f355c157a80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f355c0c9f60 tx=0x7f3550011a60 comp rx=0 tx=0).stop 2024-05-09T05:22:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.690+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c1571e0 msgr2=0x7f355c157a80 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.690+0000 7f35437fe640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c1571e0 0x7f355c157a80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.690+0000 7f355b7fe640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f35480762d0 0x7f35480787e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f355c071fe0 tx=0x7f3554001a50 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.702+0000 7f355bfff640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c157fc0 0x7f355c14c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.702+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f355c06d6b0 con 0x7f355c157fc0 2024-05-09T05:22:34.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.703+0000 7f355a7fc640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c157fc0 0x7f355c14c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.703+0000 7f355a7fc640 1 -- 172.21.15.1:0/1693193763 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f355c0c3580 con 0x7f355c157fc0 2024-05-09T05:22:34.700 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.704+0000 7f355a7fc640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c157fc0 0x7f355c14c360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3550017000 tx=0x7f3550004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.704+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3550019030 con 0x7f355c157fc0 2024-05-09T05:22:34.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.704+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7f3550003ce0 con 0x7f355c157fc0 2024-05-09T05:22:34.702 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.704+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c157fc0 msgr2=0x7f355c14c360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.704+0000 7f35437fe640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c157fc0 0x7f355c14c360 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3550017000 tx=0x7f3550004730 comp rx=0 tx=0).stop 2024-05-09T05:22:34.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.704+0000 7f35437fe640 1 -- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c157fc0 msgr2=0x7f355c14c360 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:34.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.704+0000 7f35437fe640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f355c157fc0 0x7f355c14c360 secure :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f3550017000 tx=0x7f3550004730 comp rx=0 tx=0).stop 2024-05-09T05:22:34.703 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:22:34.703 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 132, 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 143264, 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:22:34.704 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:34.705 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:34.705 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:34.705 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:34.705 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:34.705 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:34.705 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:34.705 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:34.705 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:34.706 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:34.707 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:34.708 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:34.709 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:22:34.710 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.711 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.712 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.713 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.714 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:22:34.715 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:22:34.716 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:22:34.717 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:22:34.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f35480762d0 msgr2=0x7f35480787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355bfff640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f35480762d0 0x7f35480787e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f355c071fe0 tx=0x7f3554001a50 comp rx=0 tx=0).stop 2024-05-09T05:22:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c159f60 msgr2=0x7f355c14ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355bfff640 1 --2- 172.21.15.1:0/1693193763 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f355c159f60 0x7f355c14ba80 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f354c007fb0 tx=0x7f355c0c3320 comp rx=0 tx=0).stop 2024-05-09T05:22:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355b7fe640 1 -- 172.21.15.1:0/1693193763 reap_dead start 2024-05-09T05:22:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355b7fe640 1 -- 172.21.15.1:0/1693193763 reap_dead start 2024-05-09T05:22:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 shutdown_connections 2024-05-09T05:22:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 >> 172.21.15.1:0/1693193763 conn(0x7f355c0cdc60 msgr2=0x7f355c0ccec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 shutdown_connections 2024-05-09T05:22:34.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.707+0000 7f355bfff640 1 -- 172.21.15.1:0/1693193763 wait complete. 2024-05-09T05:22:34.720 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:22:34.721 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:22:34.721 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:22:34.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.908+0000 7fc68e5d5640 1 Processor -- start 2024-05-09T05:22:34.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.908+0000 7fc68e5d5640 1 -- start start 2024-05-09T05:22:34.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.911+0000 7fc68e5d5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc688152af0 0x7fc6881592a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.911+0000 7fc68e5d5640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6880c3ed0 0x7fc6880c42e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.912+0000 7fc68cb72640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6880c3ed0 0x7fc6880c42e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.912+0000 7fc68cb72640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc6880c3ed0 0x7fc6880c42e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46144/0 (socket says 172.21.15.1:46144) 2024-05-09T05:22:34.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.912+0000 7fc687fff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc688069a80 0x7fc68806c6f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35046/0 (socket says 172.21.15.1:35046) 2024-05-09T05:22:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.912+0000 7fc68cb72640 1 -- 172.21.15.1:0/1518481735 learned_addr learned my addr 172.21.15.1:0/1518481735 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.912+0000 7fc68e5d5640 1 -- 172.21.15.1:0/1518481735 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc688152f00 con 0x7fc688069a80 2024-05-09T05:22:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.912+0000 7fc68e5d5640 1 -- 172.21.15.1:0/1518481735 --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc68806f010 con 0x7fc6880c3ed0 2024-05-09T05:22:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.912+0000 7fc68e5d5640 1 -- 172.21.15.1:0/1518481735 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc68806fb90 con 0x7fc688152af0 2024-05-09T05:22:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.912+0000 7fc6877fe640 1 --2- 172.21.15.1:0/1518481735 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc688152af0 0x7fc6881592a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.913+0000 7fc686ffd640 1 -- 172.21.15.1:0/1518481735 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 844752738 0 0) 0x7fc688152f00 con 0x7fc688069a80 2024-05-09T05:22:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.913+0000 7fc686ffd640 1 -- 172.21.15.1:0/1518481735 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fc670003520 con 0x7fc688069a80 2024-05-09T05:22:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.913+0000 7fc68cb72640 1 -- 172.21.15.1:0/1518481735 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc688152af0 msgr2=0x7fc6881592a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.913+0000 7fc68cb72640 1 --2- 172.21.15.1:0/1518481735 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc688152af0 0x7fc6881592a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.913+0000 7fc68cb72640 1 -- 172.21.15.1:0/1518481735 >> v1:172.21.15.1:6789/0 conn(0x7fc688069a80 legacy=0x7fc68806c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.913+0000 7fc68cb72640 1 -- 172.21.15.1:0/1518481735 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc688073a00 con 0x7fc6880c3ed0 2024-05-09T05:22:34.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.914+0000 7fc68cb72640 1 --2- 172.21.15.1:0/1518481735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6880c3ed0 0x7fc6880c42e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc68000f370 tx=0x7fc68002eb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f7c024def07ecad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.914+0000 7fc686ffd640 1 -- 172.21.15.1:0/1518481735 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680036070 con 0x7fc6880c3ed0 2024-05-09T05:22:34.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.915+0000 7fc686ffd640 1 -- 172.21.15.1:0/1518481735 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc68000d4e0 con 0x7fc6880c3ed0 2024-05-09T05:22:34.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.915+0000 7fc686ffd640 1 -- 172.21.15.1:0/1518481735 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68000ba00 con 0x7fc6880c3ed0 2024-05-09T05:22:34.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.916+0000 7fc68e5d5640 1 -- 172.21.15.1:0/1518481735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6880c3ed0 msgr2=0x7fc6880c42e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.916+0000 7fc68e5d5640 1 --2- 172.21.15.1:0/1518481735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6880c3ed0 0x7fc6880c42e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc68000f370 tx=0x7fc68002eb20 comp rx=0 tx=0).stop 2024-05-09T05:22:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.918+0000 7fc68e5d5640 1 -- 172.21.15.1:0/1518481735 shutdown_connections 2024-05-09T05:22:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.918+0000 7fc68e5d5640 1 --2- 172.21.15.1:0/1518481735 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc688152af0 0x7fc6881592a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.918+0000 7fc68e5d5640 1 --2- 172.21.15.1:0/1518481735 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc6880c3ed0 0x7fc6880c42e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.914 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.918+0000 7fc68e5d5640 1 -- 172.21.15.1:0/1518481735 >> 172.21.15.1:0/1518481735 conn(0x7fc6880cdcd0 msgr2=0x7fc6880ce0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:34.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.920+0000 7fc68e5d5640 1 -- 172.21.15.1:0/1518481735 shutdown_connections 2024-05-09T05:22:34.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.920+0000 7fc68e5d5640 1 -- 172.21.15.1:0/1518481735 wait complete. 2024-05-09T05:22:34.918 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.922+0000 7fc68e5d5640 1 Processor -- start 2024-05-09T05:22:34.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.923+0000 7fc68e5d5640 1 -- start start 2024-05-09T05:22:34.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.924+0000 7fc68e5d5640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc688069a80 0x7fc6881eeb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.924+0000 7fc68e5d5640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6880c3ed0 0x7fc6880c7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.924+0000 7fc68e5d5640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc688152af0 0x7fc688149a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.924+0000 7fc68e5d5640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc68806f820 con 0x7fc688152af0 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.924+0000 7fc68e5d5640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc68806f540 con 0x7fc688069a80 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.924+0000 7fc68e5d5640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc688073ea0 con 0x7fc6880c3ed0 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.925+0000 7fc68cb72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc688152af0 0x7fc688149a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.925+0000 7fc6877fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6880c3ed0 0x7fc6880c7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.925+0000 7fc68cb72640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc688152af0 0x7fc688149a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46152/0 (socket says 172.21.15.1:46152) 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.925+0000 7fc68cb72640 1 -- 172.21.15.1:0/3627347402 learned_addr learned my addr 172.21.15.1:0/3627347402 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.926+0000 7fc68cb72640 1 -- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6880c3ed0 msgr2=0x7fc6880c7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.926+0000 7fc68cb72640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc6880c3ed0 0x7fc6880c7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.926+0000 7fc68cb72640 1 -- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc688069a80 msgr2=0x7fc6881eeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.926+0000 7fc68cb72640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc688069a80 0x7fc6881eeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.926+0000 7fc68cb72640 1 -- 172.21.15.1:0/3627347402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc6881ef0a0 con 0x7fc688152af0 2024-05-09T05:22:34.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.926+0000 7fc68cb72640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc688152af0 0x7fc688149a10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc68002f110 tx=0x7fc680004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.928+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc68004a070 con 0x7fc688152af0 2024-05-09T05:22:34.924 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.928+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc68000d2e0 con 0x7fc688152af0 2024-05-09T05:22:34.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.928+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc680036050 con 0x7fc688152af0 2024-05-09T05:22:34.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.928+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc6881e2000 con 0x7fc688152af0 2024-05-09T05:22:34.925 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.928+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc6881e25c0 con 0x7fc688152af0 2024-05-09T05:22:34.926 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.930+0000 7fc68e5d5640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668002a70 0x7fc668004f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.930+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc668007f10 con 0x7fc668002a70 2024-05-09T05:22:34.927 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.931+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc68004d070 con 0x7fc688152af0 2024-05-09T05:22:34.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.932+0000 7fc684ff9640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc674076350 0x7fc674078860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.928 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.932+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc6800bc050 con 0x7fc688152af0 2024-05-09T05:22:34.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.932+0000 7fc6877fe640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc674076350 0x7fc674078860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.933+0000 7fc687fff640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668002a70 0x7fc668004f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.933+0000 7fc687fff640 1 -- 172.21.15.1:0/3627347402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc668008110 con 0x7fc668002a70 2024-05-09T05:22:34.929 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.933+0000 7fc6877fe640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc674076350 0x7fc674078860 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc678006fd0 tx=0x7fc678008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.930 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.934+0000 7fc687fff640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668002a70 0x7fc668004f80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c310 tx=0x7fc67c00c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.935+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c017070 con 0x7fc668002a70 2024-05-09T05:22:34.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.935+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc67c0074e0 con 0x7fc668002a70 2024-05-09T05:22:34.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.935+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668002a70 msgr2=0x7fc668004f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.935+0000 7fc684ff9640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668002a70 0x7fc668004f80 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c310 tx=0x7fc67c00c7d0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.935+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668002a70 msgr2=0x7fc668004f80 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:34.932 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.935+0000 7fc684ff9640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668002a70 0x7fc668004f80 secure :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7fc67c00c310 tx=0x7fc67c00c7d0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.950+0000 7fc68e5d5640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668006f80 0x7fc668007440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:34.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.950+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc66800a4f0 con 0x7fc668006f80 2024-05-09T05:22:34.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.950+0000 7fc687fff640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668006f80 0x7fc668007440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:34.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.951+0000 7fc687fff640 1 -- 172.21.15.1:0/3627347402 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc668007980 con 0x7fc668006f80 2024-05-09T05:22:34.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.951+0000 7fc687fff640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668006f80 0x7fc668007440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc67c014000 tx=0x7fc67c000fe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:34.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.951+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc67c017040 con 0x7fc668006f80 2024-05-09T05:22:34.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.952+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4550 (secure 0 0 0) 0x7fc67c012400 con 0x7fc668006f80 2024-05-09T05:22:34.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.952+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668006f80 msgr2=0x7fc668007440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.952+0000 7fc684ff9640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668006f80 0x7fc668007440 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc67c014000 tx=0x7fc67c000fe0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.952+0000 7fc684ff9640 1 -- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668006f80 msgr2=0x7fc668007440 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:34.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.952+0000 7fc684ff9640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc668006f80 0x7fc668007440 secure :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fc67c014000 tx=0x7fc67c000fe0 comp rx=0 tx=0).stop 2024-05-09T05:22:34.949 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:22:34.949 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:34.949 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:34.949 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:22:34.949 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 132, 2024-05-09T05:22:34.949 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 8667, 2024-05-09T05:22:34.949 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:22:34.949 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:34.950 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:34.951 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:22:34.952 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:22:34.953 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:34.954 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:34.955 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:34.955 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.955 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:22:34.955 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.955 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:22:34.955 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.955 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:22:34.956 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:34.957 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.958 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:34.959 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.960 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:22:34.961 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:34.962 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:34.963 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:34.963 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:22:34.963 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.953+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc674076350 msgr2=0x7fc674078860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.953+0000 7fc68e5d5640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc674076350 0x7fc674078860 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fc678006fd0 tx=0x7fc678008040 comp rx=0 tx=0).stop 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.953+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc688152af0 msgr2=0x7fc688149a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.953+0000 7fc68e5d5640 1 --2- 172.21.15.1:0/3627347402 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc688152af0 0x7fc688149a10 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc68002f110 tx=0x7fc680004a40 comp rx=0 tx=0).stop 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.953+0000 7fc68cb72640 1 -- 172.21.15.1:0/3627347402 reap_dead start 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.953+0000 7fc68cb72640 1 -- 172.21.15.1:0/3627347402 reap_dead start 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.954+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 shutdown_connections 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.954+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 >> 172.21.15.1:0/3627347402 conn(0x7fc6880cdcd0 msgr2=0x7fc6880c83c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:34.964 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.954+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 shutdown_connections 2024-05-09T05:22:34.965 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:34.954+0000 7fc68e5d5640 1 -- 172.21.15.1:0/3627347402 wait complete. 2024-05-09T05:22:34.967 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:22:34.967 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:22:34.967 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:22:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.156+0000 7ff597b06640 1 Processor -- start 2024-05-09T05:22:35.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.157+0000 7ff597b06640 1 -- start start 2024-05-09T05:22:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff597b06640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900cbde0 0x7ff590154500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff597b06640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff5900c49b0 0x7ff5900cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff597b06640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff5900d04f0 0x7ff5900cc7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff597b06640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff5900cfb20 con 0x7ff5900d04f0 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff597b06640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff590154a40 con 0x7ff5900c49b0 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff597b06640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff59015b170 con 0x7ff5900cbde0 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff5958a2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900cbde0 0x7ff590154500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff5950a1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff5900c49b0 0x7ff5900cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff5958a2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900cbde0 0x7ff590154500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:46538/0 (socket says 172.21.15.1:46538) 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff5958a2640 1 -- 172.21.15.1:0/2125044125 learned_addr learned my addr 172.21.15.1:0/2125044125 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.159+0000 7ff5960a3640 1 --2- 172.21.15.1:0/2125044125 >> v2:172.21.15.1:3300/0 conn(0x7ff5900d04f0 0x7ff5900cc7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.160+0000 7ff5958a2640 1 -- 172.21.15.1:0/2125044125 >> v2:172.21.15.196:3300/0 conn(0x7ff5900c49b0 msgr2=0x7ff5900cffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.160+0000 7ff5958a2640 1 --2- 172.21.15.1:0/2125044125 >> v2:172.21.15.196:3300/0 conn(0x7ff5900c49b0 0x7ff5900cffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.160+0000 7ff5958a2640 1 -- 172.21.15.1:0/2125044125 >> v2:172.21.15.1:3300/0 conn(0x7ff5900d04f0 msgr2=0x7ff5900cc7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:35.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.160+0000 7ff5958a2640 1 --2- 172.21.15.1:0/2125044125 >> v2:172.21.15.1:3300/0 conn(0x7ff5900d04f0 0x7ff5900cc7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.160+0000 7ff5958a2640 1 -- 172.21.15.1:0/2125044125 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5900cb120 con 0x7ff5900cbde0 2024-05-09T05:22:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.161+0000 7ff5958a2640 1 --2- 172.21.15.1:0/2125044125 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900cbde0 0x7ff590154500 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff590069630 tx=0x7ff58c02e4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e29c829e18185641 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.161+0000 7ff5948a0640 1 -- 172.21.15.1:0/2125044125 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c03a070 con 0x7ff5900cbde0 2024-05-09T05:22:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.161+0000 7ff5948a0640 1 -- 172.21.15.1:0/2125044125 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff58c00b650 con 0x7ff5900cbde0 2024-05-09T05:22:35.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.161+0000 7ff5948a0640 1 -- 172.21.15.1:0/2125044125 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c00ba60 con 0x7ff5900cbde0 2024-05-09T05:22:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.165+0000 7ff597b06640 1 -- 172.21.15.1:0/2125044125 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900cbde0 msgr2=0x7ff590154500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.165+0000 7ff597b06640 1 --2- 172.21.15.1:0/2125044125 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900cbde0 0x7ff590154500 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ff590069630 tx=0x7ff58c02e4f0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.165+0000 7ff597b06640 1 -- 172.21.15.1:0/2125044125 shutdown_connections 2024-05-09T05:22:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.165+0000 7ff597b06640 1 --2- 172.21.15.1:0/2125044125 >> v2:172.21.15.1:3300/0 conn(0x7ff5900d04f0 0x7ff5900cc7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.165+0000 7ff597b06640 1 --2- 172.21.15.1:0/2125044125 >> v2:172.21.15.196:3300/0 conn(0x7ff5900c49b0 0x7ff5900cffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.165+0000 7ff597b06640 1 --2- 172.21.15.1:0/2125044125 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900cbde0 0x7ff590154500 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.165+0000 7ff597b06640 1 -- 172.21.15.1:0/2125044125 >> 172.21.15.1:0/2125044125 conn(0x7ff590068830 msgr2=0x7ff590068c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.165+0000 7ff597b06640 1 -- 172.21.15.1:0/2125044125 shutdown_connections 2024-05-09T05:22:35.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.167+0000 7ff597b06640 1 -- 172.21.15.1:0/2125044125 wait complete. 2024-05-09T05:22:35.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.170+0000 7ff597b06640 1 Processor -- start 2024-05-09T05:22:35.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.170+0000 7ff597b06640 1 -- start start 2024-05-09T05:22:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.172+0000 7ff597b06640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900c49b0 0x7ff5901f2090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff597b06640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5900cbde0 0x7ff59015a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff597b06640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5900d04f0 0x7ff59014fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff597b06640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff5900cb8b0 con 0x7ff5900d04f0 2024-05-09T05:22:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff597b06640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff590154c80 con 0x7ff5900cbde0 2024-05-09T05:22:35.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff597b06640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff59015b430 con 0x7ff5900c49b0 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff5950a1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5900cbde0 0x7ff59015a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff5950a1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5900cbde0 0x7ff59015a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33222/0 (socket says 172.21.15.1:33222) 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff5950a1640 1 -- 172.21.15.1:0/939800963 learned_addr learned my addr 172.21.15.1:0/939800963 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff5960a3640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5900d04f0 0x7ff59014fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.173+0000 7ff5958a2640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900c49b0 0x7ff5901f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5950a1640 1 -- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900c49b0 msgr2=0x7ff5901f2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5950a1640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5900c49b0 0x7ff5901f2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5950a1640 1 -- 172.21.15.1:0/939800963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5900d04f0 msgr2=0x7ff59014fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5950a1640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5900d04f0 0x7ff59014fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5950a1640 1 -- 172.21.15.1:0/939800963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5901f25d0 con 0x7ff5900cbde0 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5950a1640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5900cbde0 0x7ff59015a420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff58000e840 tx=0x7ff58000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff580017070 con 0x7ff5900cbde0 2024-05-09T05:22:35.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff58000c730 con 0x7ff5900cbde0 2024-05-09T05:22:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff59015a960 con 0x7ff5900cbde0 2024-05-09T05:22:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.174+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58000cb20 con 0x7ff5900cbde0 2024-05-09T05:22:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.175+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5901eeeb0 con 0x7ff5900cbde0 2024-05-09T05:22:35.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.176+0000 7ff597b06640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff590150fd0 0x7ff590150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.176+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff590050870 con 0x7ff590150fd0 2024-05-09T05:22:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.177+0000 7ff5958a2640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff590150fd0 0x7ff590150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.177+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff58001a070 con 0x7ff5900cbde0 2024-05-09T05:22:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.178+0000 7ff5958a2640 1 -- 172.21.15.1:0/939800963 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff590157f00 con 0x7ff590150fd0 2024-05-09T05:22:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.178+0000 7ff5958a2640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff590150fd0 0x7ff590150020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff5900745d0 tx=0x7ff58c0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:35.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.178+0000 7ff5867fc640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5740762d0 0x7ff5740787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.179+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff5800974c0 con 0x7ff5900cbde0 2024-05-09T05:22:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.179+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c03a040 con 0x7ff590150fd0 2024-05-09T05:22:35.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.179+0000 7ff5960a3640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5740762d0 0x7ff5740787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.179+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7ff58c03b040 con 0x7ff590150fd0 2024-05-09T05:22:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.179+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff590150fd0 msgr2=0x7ff590150020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.179+0000 7ff5867fc640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff590150fd0 0x7ff590150020 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ff5900745d0 tx=0x7ff58c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.179+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff590150fd0 msgr2=0x7ff590150020 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:35.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.179+0000 7ff5867fc640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff590150fd0 0x7ff590150020 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.177 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.181+0000 7ff5960a3640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5740762d0 0x7ff5740787e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff59006f940 tx=0x7ff58800b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.195+0000 7ff597b06640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5901ef080 0x7ff5901e50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.195+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff5901ef490 con 0x7ff5901ef080 2024-05-09T05:22:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.195+0000 7ff5958a2640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5901ef080 0x7ff5901e50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.192 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.197+0000 7ff5958a2640 1 -- 172.21.15.1:0/939800963 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7ff5900b5470 con 0x7ff5901ef080 2024-05-09T05:22:35.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.197+0000 7ff5958a2640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5901ef080 0x7ff5901e50c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5900509f0 tx=0x7ff58c033000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.198+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff58c03a030 con 0x7ff5901ef080 2024-05-09T05:22:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.198+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5061 (secure 0 0 0) 0x7ff58c04f070 con 0x7ff5901ef080 2024-05-09T05:22:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.198+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5901ef080 msgr2=0x7ff5901e50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.194 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.198+0000 7ff5867fc640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5901ef080 0x7ff5901e50c0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7ff5900509f0 tx=0x7ff58c033000 comp rx=0 tx=0).stop 2024-05-09T05:22:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.198+0000 7ff5867fc640 1 -- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5901ef080 msgr2=0x7ff5901e50c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:35.195 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.198+0000 7ff5867fc640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5901ef080 0x7ff5901e50c0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.197 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:22:35.197 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:35.197 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:35.197 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:22:35.197 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 132, 2024-05-09T05:22:35.197 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 70023, 2024-05-09T05:22:35.197 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:22:35.197 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:35.198 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:35.199 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:22:35.200 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:22:35.201 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.202 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:22:35.203 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:35.204 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:22:35.205 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.206 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.207 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:35.208 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:22:35.209 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:35.210 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:35.210 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:35.210 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:35.210 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:35.210 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:35.210 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:35.210 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:22:35.210 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:22:35.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.202+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5740762d0 msgr2=0x7ff5740787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.202+0000 7ff597b06640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5740762d0 0x7ff5740787e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7ff59006f940 tx=0x7ff58800b040 comp rx=0 tx=0).stop 2024-05-09T05:22:35.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.203+0000 7ff5960a3640 1 -- 172.21.15.1:0/939800963 reap_dead start 2024-05-09T05:22:35.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.203+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5900cbde0 msgr2=0x7ff59015a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.203+0000 7ff597b06640 1 --2- 172.21.15.1:0/939800963 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5900cbde0 0x7ff59015a420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff58000e840 tx=0x7ff58000ed00 comp rx=0 tx=0).stop 2024-05-09T05:22:35.211 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.203+0000 7ff5960a3640 1 -- 172.21.15.1:0/939800963 reap_dead start 2024-05-09T05:22:35.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.203+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 shutdown_connections 2024-05-09T05:22:35.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.203+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 >> 172.21.15.1:0/939800963 conn(0x7ff590068830 msgr2=0x7ff59014df70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:35.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.203+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 shutdown_connections 2024-05-09T05:22:35.212 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.203+0000 7ff597b06640 1 -- 172.21.15.1:0/939800963 wait complete. 2024-05-09T05:22:35.217 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:22:35.217 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:22:35.217 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:22:35.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.409+0000 7fea9359e640 1 Processor -- start 2024-05-09T05:22:35.406 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.409+0000 7fea9359e640 1 -- start start 2024-05-09T05:22:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.412+0000 7fea9359e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fea940c95f0 0x7fea94158aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.412+0000 7fea9359e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea94073dc0 con 0x7fea940c8f60 2024-05-09T05:22:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.412+0000 7fea9359e640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fea940c9370 con 0x7fea940b2a80 2024-05-09T05:22:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.412+0000 7fea9359e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fea94068d60 con 0x7fea940c95f0 2024-05-09T05:22:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.412+0000 7fea92d9d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fea940c8f60 0x7fea940c4470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:35048/0 (socket says 172.21.15.1:35048) 2024-05-09T05:22:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.412+0000 7fea92d9d640 1 -- 172.21.15.1:0/2686710422 learned_addr learned my addr 172.21.15.1:0/2686710422 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:22:35.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.412+0000 7fea91d9b640 1 --2- 172.21.15.1:0/2686710422 >> v2:172.21.15.1:3300/0 conn(0x7fea940c95f0 0x7fea94158aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.413+0000 7fea91d9b640 1 -- 172.21.15.1:0/2686710422 >> v1:172.21.15.196:6789/0 conn(0x7fea940b2a80 legacy=0x7fea94068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.413+0000 7fea91d9b640 1 -- 172.21.15.1:0/2686710422 >> v1:172.21.15.1:6789/0 conn(0x7fea940c8f60 legacy=0x7fea940c4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.413+0000 7fea91d9b640 1 -- 172.21.15.1:0/2686710422 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea94074060 con 0x7fea940c95f0 2024-05-09T05:22:35.410 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.414+0000 7fea91d9b640 1 --2- 172.21.15.1:0/2686710422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940c95f0 0x7fea94158aa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea880097a0 tx=0x7fea8802a1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a081100a70ad6d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:35.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.415+0000 7fea9159a640 1 -- 172.21.15.1:0/2686710422 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88037070 con 0x7fea940c95f0 2024-05-09T05:22:35.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.415+0000 7fea9159a640 1 -- 172.21.15.1:0/2686710422 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8802ae50 con 0x7fea940c95f0 2024-05-09T05:22:35.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.415+0000 7fea9159a640 1 -- 172.21.15.1:0/2686710422 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea88032660 con 0x7fea940c95f0 2024-05-09T05:22:35.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.416+0000 7fea9359e640 1 -- 172.21.15.1:0/2686710422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940c95f0 msgr2=0x7fea94158aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.416+0000 7fea9359e640 1 --2- 172.21.15.1:0/2686710422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940c95f0 0x7fea94158aa0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fea880097a0 tx=0x7fea8802a1c0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.416+0000 7fea9359e640 1 -- 172.21.15.1:0/2686710422 shutdown_connections 2024-05-09T05:22:35.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.416+0000 7fea9359e640 1 --2- 172.21.15.1:0/2686710422 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940c95f0 0x7fea94158aa0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.416+0000 7fea9359e640 1 -- 172.21.15.1:0/2686710422 >> 172.21.15.1:0/2686710422 conn(0x7fea940ca860 msgr2=0x7fea940cac70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.417+0000 7fea9359e640 1 -- 172.21.15.1:0/2686710422 shutdown_connections 2024-05-09T05:22:35.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.417+0000 7fea9359e640 1 -- 172.21.15.1:0/2686710422 wait complete. 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.421+0000 7fea9359e640 1 Processor -- start 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.421+0000 7fea9359e640 1 -- start start 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.423+0000 7fea9359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940b2a80 0x7fea940c62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.423+0000 7fea9359e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fea940c8f60 0x7fea94151520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.423+0000 7fea9359e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea940c95f0 0x7fea940c7100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.423+0000 7fea9359e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fea940c8a40 con 0x7fea940b2a80 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.423+0000 7fea9359e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fea940c8760 con 0x7fea940c95f0 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.423+0000 7fea9359e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fea940cb6d0 con 0x7fea940c8f60 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.423+0000 7fea92d9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea940c95f0 0x7fea940c7100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.423+0000 7fea9259c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940b2a80 0x7fea940c62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea92d9d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea940c95f0 0x7fea940c7100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:33224/0 (socket says 172.21.15.1:33224) 2024-05-09T05:22:35.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea92d9d640 1 -- 172.21.15.1:0/1082354235 learned_addr learned my addr 172.21.15.1:0/1082354235 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea92d9d640 1 -- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fea940c8f60 msgr2=0x7fea94151520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea91d9b640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fea940c8f60 0x7fea94151520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea92d9d640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fea940c8f60 0x7fea94151520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea92d9d640 1 -- 172.21.15.1:0/1082354235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940b2a80 msgr2=0x7fea940c62a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea92d9d640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940b2a80 0x7fea940c62a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea92d9d640 1 -- 172.21.15.1:0/1082354235 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea940c67e0 con 0x7fea940c95f0 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea9259c640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940b2a80 0x7fea940c62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.424+0000 7fea91d9b640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fea940c8f60 0x7fea94151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.425+0000 7fea92d9d640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea940c95f0 0x7fea940c7100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00cc20 tx=0x7fea8c00a430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:35.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.425+0000 7fea82ffd640 1 -- 172.21.15.1:0/1082354235 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c017070 con 0x7fea940c95f0 2024-05-09T05:22:35.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.425+0000 7fea82ffd640 1 -- 172.21.15.1:0/1082354235 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fea8c00ce30 con 0x7fea940c95f0 2024-05-09T05:22:35.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.425+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea940c5200 con 0x7fea940c95f0 2024-05-09T05:22:35.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.425+0000 7fea82ffd640 1 -- 172.21.15.1:0/1082354235 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fea8c012620 con 0x7fea940c95f0 2024-05-09T05:22:35.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.425+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea940c57c0 con 0x7fea940c95f0 2024-05-09T05:22:35.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.426+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea94076f80 con 0x7fea940c95f0 2024-05-09T05:22:35.423 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.427+0000 7fea82ffd640 1 -- 172.21.15.1:0/1082354235 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fea8c01a070 con 0x7fea940c95f0 2024-05-09T05:22:35.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.428+0000 7fea82ffd640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fea74076300 0x7fea74078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:35.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.428+0000 7fea82ffd640 1 -- 172.21.15.1:0/1082354235 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fea8c097460 con 0x7fea940c95f0 2024-05-09T05:22:35.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.428+0000 7fea9259c640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fea74076300 0x7fea74078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:35.429 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.433+0000 7fea9259c640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fea74076300 0x7fea74078810 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fea84001b00 tx=0x7fea84009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:35.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.435+0000 7fea82ffd640 1 -- 172.21.15.1:0/1082354235 <== mon.1 v2:172.21.15.196:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fea8c060660 con 0x7fea940c95f0 2024-05-09T05:22:35.931 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.934+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fea940cbbd0 con 0x7fea940c95f0 2024-05-09T05:22:35.933 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.937+0000 7fea82ffd640 1 -- 172.21.15.1:0/1082354235 <== mon.1 v2:172.21.15.196:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fea8c064960 con 0x7fea940c95f0 2024-05-09T05:22:35.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fea74076300 msgr2=0x7fea74078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fea74076300 0x7fea74078810 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fea84001b00 tx=0x7fea84009040 comp rx=0 tx=0).stop 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea940c95f0 msgr2=0x7fea940c7100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea940c95f0 0x7fea940c7100 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fea8c00cc20 tx=0x7fea8c00a430 comp rx=0 tx=0).stop 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 shutdown_connections 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fea74076300 0x7fea74078810 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fea940c95f0 0x7fea940c7100 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fea940c8f60 0x7fea94151520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 --2- 172.21.15.1:0/1082354235 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fea940b2a80 0x7fea940c62a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 >> 172.21.15.1:0/1082354235 conn(0x7fea940ca860 msgr2=0x7fea94073320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 shutdown_connections 2024-05-09T05:22:35.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:35.944+0000 7fea9359e640 1 -- 172.21.15.1:0/1082354235 wait complete. 2024-05-09T05:22:35.963 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:22:36.520 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:22:36.524 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:22:45.964 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:22:45.964 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.156+0000 7f34f04a1640 1 Processor -- start 2024-05-09T05:22:46.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.156+0000 7f34f04a1640 1 -- start start 2024-05-09T05:22:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.159+0000 7f34f04a1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34e81538b0 0x7f34e80c92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.159+0000 7f34f04a1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e80c9810 0x7f34e815cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.159+0000 7f34f04a1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34e81594f0 con 0x7f34e80c87d0 2024-05-09T05:22:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.159+0000 7f34f04a1640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f34e815a380 con 0x7f34e81538b0 2024-05-09T05:22:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.159+0000 7f34eea3e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e80c9810 0x7f34e815cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.159+0000 7f34eea3e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e80c9810 0x7f34e815cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:55162/0 (socket says 172.21.15.1:55162) 2024-05-09T05:22:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.159+0000 7f34eea3e640 1 -- 172.21.15.1:0/1055113614 learned_addr learned my addr 172.21.15.1:0/1055113614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:46.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.159+0000 7f34f04a1640 1 -- 172.21.15.1:0/1055113614 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f34e8159770 con 0x7f34e80c9810 2024-05-09T05:22:46.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.162+0000 7f34eda3c640 1 --2- 172.21.15.1:0/1055113614 >> v2:172.21.15.1:3300/0 conn(0x7f34e81538b0 0x7f34e80c92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.162+0000 7f34eda3c640 1 -- 172.21.15.1:0/1055113614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e80c9810 msgr2=0x7f34e815cc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:46.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.162+0000 7f34ed23b640 1 -- 172.21.15.1:0/1055113614 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 718909084 0 0) 0x7f34e4001490 con 0x7f34e80c87d0 2024-05-09T05:22:46.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.162+0000 7f34eda3c640 1 --2- 172.21.15.1:0/1055113614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e80c9810 0x7f34e815cc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.162+0000 7f34eda3c640 1 -- 172.21.15.1:0/1055113614 >> v1:172.21.15.196:6789/0 conn(0x7f34e80c87d0 legacy=0x7f34e80c5ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.162+0000 7f34eda3c640 1 -- 172.21.15.1:0/1055113614 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e8159a50 con 0x7f34e81538b0 2024-05-09T05:22:46.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.164+0000 7f34eda3c640 1 --2- 172.21.15.1:0/1055113614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e81538b0 0x7f34e80c92d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f34d8006d10 tx=0x7f34d8032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b01acc9b1e7537a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:46.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.165+0000 7f34ed23b640 1 -- 172.21.15.1:0/1055113614 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d8040070 con 0x7f34e81538b0 2024-05-09T05:22:46.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.165+0000 7f34ed23b640 1 -- 172.21.15.1:0/1055113614 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34d8033b70 con 0x7f34e81538b0 2024-05-09T05:22:46.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.165+0000 7f34ed23b640 1 -- 172.21.15.1:0/1055113614 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d803b440 con 0x7f34e81538b0 2024-05-09T05:22:46.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.167+0000 7f34f04a1640 1 -- 172.21.15.1:0/1055113614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e81538b0 msgr2=0x7f34e80c92d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.167+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1055113614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e81538b0 0x7f34e80c92d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f34d8006d10 tx=0x7f34d8032e20 comp rx=0 tx=0).stop 2024-05-09T05:22:46.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.168+0000 7f34f04a1640 1 -- 172.21.15.1:0/1055113614 shutdown_connections 2024-05-09T05:22:46.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.168+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1055113614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e80c9810 0x7f34e815cc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.168+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1055113614 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e81538b0 0x7f34e80c92d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.168+0000 7f34f04a1640 1 -- 172.21.15.1:0/1055113614 >> 172.21.15.1:0/1055113614 conn(0x7f34e80745a0 msgr2=0x7f34e8071900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:46.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.169+0000 7f34f04a1640 1 -- 172.21.15.1:0/1055113614 shutdown_connections 2024-05-09T05:22:46.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.169+0000 7f34f04a1640 1 -- 172.21.15.1:0/1055113614 wait complete. 2024-05-09T05:22:46.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.173+0000 7f34f04a1640 1 Processor -- start 2024-05-09T05:22:46.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.173+0000 7f34f04a1640 1 -- start start 2024-05-09T05:22:46.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.174+0000 7f34f04a1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e80c87d0 0x7f34e815a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.174+0000 7f34f04a1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34e80c9810 0x7f34e806da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34f04a1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e81538b0 0x7f34e8143220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34ee23d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e80c87d0 0x7f34e815a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34ee23d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e80c87d0 0x7f34e815a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46188/0 (socket says 172.21.15.1:46188) 2024-05-09T05:22:46.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34ee23d640 1 -- 172.21.15.1:0/1461065017 learned_addr learned my addr 172.21.15.1:0/1461065017 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:46.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f34e80c70c0 con 0x7f34e80c87d0 2024-05-09T05:22:46.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f34e815a010 con 0x7f34e80c9810 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f34e80c7240 con 0x7f34e81538b0 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34eda3c640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34e80c9810 0x7f34e806da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34ee23d640 1 -- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e81538b0 msgr2=0x7f34e8143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34ee23d640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e81538b0 0x7f34e8143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34ee23d640 1 -- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34e80c9810 msgr2=0x7f34e806da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34ee23d640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34e80c9810 0x7f34e806da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.175+0000 7f34ee23d640 1 -- 172.21.15.1:0/1461065017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e81482e0 con 0x7f34e80c87d0 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.176+0000 7f34ee23d640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e80c87d0 0x7f34e815a6b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f34e400c730 tx=0x7f34e400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:46.173 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.176+0000 7f34deffd640 1 -- 172.21.15.1:0/1461065017 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4003da0 con 0x7f34e80c87d0 2024-05-09T05:22:46.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.176+0000 7f34deffd640 1 -- 172.21.15.1:0/1461065017 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34e400ce40 con 0x7f34e80c87d0 2024-05-09T05:22:46.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.177+0000 7f34deffd640 1 -- 172.21.15.1:0/1461065017 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34e4005270 con 0x7f34e80c87d0 2024-05-09T05:22:46.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.177+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e81eeb40 con 0x7f34e80c87d0 2024-05-09T05:22:46.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.178+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e81ef100 con 0x7f34e80c87d0 2024-05-09T05:22:46.176 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.180+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e81581d0 con 0x7f34e80c87d0 2024-05-09T05:22:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.192+0000 7f34deffd640 1 -- 172.21.15.1:0/1461065017 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f34e4005410 con 0x7f34e80c87d0 2024-05-09T05:22:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.193+0000 7f34deffd640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34cc0763d0 0x7f34cc0788e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.193+0000 7f34deffd640 1 -- 172.21.15.1:0/1461065017 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f34e4097e70 con 0x7f34e80c87d0 2024-05-09T05:22:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.193+0000 7f34eda3c640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34cc0763d0 0x7f34cc0788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.189 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.193+0000 7f34deffd640 1 -- 172.21.15.1:0/1461065017 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f34e81581d0 con 0x7f34e80c87d0 2024-05-09T05:22:46.191 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.195+0000 7f34eda3c640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34cc0763d0 0x7f34cc0788e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f34d803d000 tx=0x7f34d8037000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:46.664 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.668+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f34e8050870 con 0x7f34e80c87d0 2024-05-09T05:22:46.665 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":15,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:22:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.668+0000 7f34deffd640 1 -- 172.21.15.1:0/1461065017 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f34e81594f0 con 0x7f34e80c87d0 2024-05-09T05:22:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34cc0763d0 msgr2=0x7f34cc0788e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34cc0763d0 0x7f34cc0788e0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f34d803d000 tx=0x7f34d8037000 comp rx=0 tx=0).stop 2024-05-09T05:22:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e80c87d0 msgr2=0x7f34e815a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e80c87d0 0x7f34e815a6b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f34e400c730 tx=0x7f34e400cbf0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 shutdown_connections 2024-05-09T05:22:46.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34cc0763d0 0x7f34cc0788e0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34e80c9810 0x7f34e806da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34e81538b0 0x7f34e8143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 --2- 172.21.15.1:0/1461065017 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34e80c87d0 0x7f34e815a6b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 >> 172.21.15.1:0/1461065017 conn(0x7f34e80745a0 msgr2=0x7f34e80ca030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 shutdown_connections 2024-05-09T05:22:46.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.671+0000 7f34f04a1640 1 -- 172.21.15.1:0/1461065017 wait complete. 2024-05-09T05:22:46.686 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:22:46.686 INFO:tasks.mon_thrash.mon_thrasher:making sure all monitors are in the quorum 2024-05-09T05:22:46.686 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:22:46.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:22:46.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.887+0000 7f28c7b1f640 1 Processor -- start 2024-05-09T05:22:46.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.887+0000 7f28c7b1f640 1 -- start start 2024-05-09T05:22:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.889+0000 7f28c7b1f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28c00740c0 0x7f28c015ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.889+0000 7f28c7b1f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28c0074a20 0x7f28c00ca590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c7b1f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f28c0164f00 con 0x7f28c0074610 2024-05-09T05:22:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c7b1f640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f28c00c4e00 con 0x7f28c0074a20 2024-05-09T05:22:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c7b1f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f28c015b370 con 0x7f28c00740c0 2024-05-09T05:22:46.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c60bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28c0074a20 0x7f28c00ca590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c60bc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28c0074a20 0x7f28c00ca590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46196/0 (socket says 172.21.15.1:46196) 2024-05-09T05:22:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c60bc640 1 -- 172.21.15.1:0/152288667 learned_addr learned my addr 172.21.15.1:0/152288667 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c60bc640 1 -- 172.21.15.1:0/152288667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28c00740c0 msgr2=0x7f28c015ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c58bb640 1 --2- 172.21.15.1:0/152288667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28c00740c0 0x7f28c015ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.890+0000 7f28c60bc640 1 --2- 172.21.15.1:0/152288667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28c00740c0 0x7f28c015ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.891+0000 7f28c60bc640 1 -- 172.21.15.1:0/152288667 >> v1:172.21.15.196:6789/0 conn(0x7f28c0074610 legacy=0x7f28c00c9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:46.889 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.891+0000 7f28c60bc640 1 -- 172.21.15.1:0/152288667 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c00c97d0 con 0x7f28c0074a20 2024-05-09T05:22:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.892+0000 7f28c60bc640 1 --2- 172.21.15.1:0/152288667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0074a20 0x7f28c00ca590 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009e80 tx=0x7f28b802a730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=25d32ee4800b25a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.893+0000 7f28c48b9640 1 -- 172.21.15.1:0/152288667 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b8036070 con 0x7f28c0074a20 2024-05-09T05:22:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.893+0000 7f28c48b9640 1 -- 172.21.15.1:0/152288667 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b8031600 con 0x7f28c0074a20 2024-05-09T05:22:46.890 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.893+0000 7f28c48b9640 1 -- 172.21.15.1:0/152288667 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b80319d0 con 0x7f28c0074a20 2024-05-09T05:22:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.894+0000 7f28c7b1f640 1 -- 172.21.15.1:0/152288667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0074a20 msgr2=0x7f28c00ca590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.894+0000 7f28c7b1f640 1 --2- 172.21.15.1:0/152288667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0074a20 0x7f28c00ca590 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009e80 tx=0x7f28b802a730 comp rx=0 tx=0).stop 2024-05-09T05:22:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.895+0000 7f28c7b1f640 1 -- 172.21.15.1:0/152288667 shutdown_connections 2024-05-09T05:22:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.895+0000 7f28c7b1f640 1 --2- 172.21.15.1:0/152288667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0074a20 0x7f28c00ca590 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.891 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.895+0000 7f28c7b1f640 1 --2- 172.21.15.1:0/152288667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28c00740c0 0x7f28c015ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.895+0000 7f28c7b1f640 1 -- 172.21.15.1:0/152288667 >> 172.21.15.1:0/152288667 conn(0x7f28c0070f40 msgr2=0x7f28c006c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.895+0000 7f28c7b1f640 1 -- 172.21.15.1:0/152288667 shutdown_connections 2024-05-09T05:22:46.892 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.895+0000 7f28c7b1f640 1 -- 172.21.15.1:0/152288667 wait complete. 2024-05-09T05:22:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.898+0000 7f28c7b1f640 1 Processor -- start 2024-05-09T05:22:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.898+0000 7f28c7b1f640 1 -- start start 2024-05-09T05:22:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.900+0000 7f28c7b1f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c00740c0 0x7f28c015f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.900+0000 7f28c7b1f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28c0074610 0x7f28c01f7930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.901+0000 7f28c7b1f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28c0074a20 0x7f28c0150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.901+0000 7f28c7b1f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f28c0073610 con 0x7f28c00740c0 2024-05-09T05:22:46.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.901+0000 7f28c7b1f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f28c0164c10 con 0x7f28c0074a20 2024-05-09T05:22:46.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.901+0000 7f28c7b1f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f28c00caad0 con 0x7f28c0074610 2024-05-09T05:22:46.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.903+0000 7f28c58bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c00740c0 0x7f28c015f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.903+0000 7f28c58bb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c00740c0 0x7f28c015f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46210/0 (socket says 172.21.15.1:46210) 2024-05-09T05:22:46.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.903+0000 7f28c58bb640 1 -- 172.21.15.1:0/1158743102 learned_addr learned my addr 172.21.15.1:0/1158743102 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:46.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.904+0000 7f28c58bb640 1 -- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28c0074610 msgr2=0x7f28c01f7930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.904+0000 7f28c58bb640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28c0074610 0x7f28c01f7930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.904+0000 7f28c58bb640 1 -- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28c0074a20 msgr2=0x7f28c0150d40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:46.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.904+0000 7f28c58bb640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28c0074a20 0x7f28c0150d40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.904+0000 7f28c58bb640 1 -- 172.21.15.1:0/1158743102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c0161820 con 0x7f28c00740c0 2024-05-09T05:22:46.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.905+0000 7f28c58bb640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c00740c0 0x7f28c015f7f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f28b002e8a0 tx=0x7f28b002ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:46.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.905+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b0037070 con 0x7f28c00740c0 2024-05-09T05:22:46.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.905+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28b0004020 con 0x7f28c00740c0 2024-05-09T05:22:46.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.905+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b00043f0 con 0x7f28c00740c0 2024-05-09T05:22:46.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.905+0000 7f28c7b1f640 1 -- 172.21.15.1:0/1158743102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c015ea30 con 0x7f28c00740c0 2024-05-09T05:22:46.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.906+0000 7f28c7b1f640 1 -- 172.21.15.1:0/1158743102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c015eff0 con 0x7f28c00740c0 2024-05-09T05:22:46.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.908+0000 7f28a3fff640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0162d00 0x7f28c01fabe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.908+0000 7f28a3fff640 1 -- 172.21.15.1:0/1158743102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f28c0163140 con 0x7f28c0162d00 2024-05-09T05:22:46.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.909+0000 7f28c50ba640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0162d00 0x7f28c01fabe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.909+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f28b0007620 con 0x7f28c00740c0 2024-05-09T05:22:46.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.910+0000 7f28c50ba640 1 -- 172.21.15.1:0/1158743102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f28c0161ae0 con 0x7f28c0162d00 2024-05-09T05:22:46.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.910+0000 7f28c50ba640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0162d00 0x7f28c01fabe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00b310 tx=0x7f28bc00b7d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.910+0000 7f28b67fc640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28a4076320 0x7f28a4078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f28b00b8650 con 0x7f28c00740c0 2024-05-09T05:22:46.908 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28c60bc640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28a4076320 0x7f28a4078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc015020 con 0x7f28c0162d00 2024-05-09T05:22:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f28bc011830 con 0x7f28c0162d00 2024-05-09T05:22:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0162d00 msgr2=0x7f28c01fabe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28b67fc640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0162d00 0x7f28c01fabe0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00b310 tx=0x7f28bc00b7d0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0162d00 msgr2=0x7f28c01fabe0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28b67fc640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c0162d00 0x7f28c01fabe0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.911+0000 7f28c60bc640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28a4076320 0x7f28a4078830 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f28b8033000 tx=0x7f28b8009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:46.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.923+0000 7f28a3fff640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c007cc40 0x7f28c007f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:46.919 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.923+0000 7f28a3fff640 1 -- 172.21.15.1:0/1158743102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01f7e70 con 0x7f28c007cc40 2024-05-09T05:22:46.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.924+0000 7f28c50ba640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c007cc40 0x7f28c007f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:46.920 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.924+0000 7f28c50ba640 1 -- 172.21.15.1:0/1158743102 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f28c00733a0 con 0x7f28c007cc40 2024-05-09T05:22:46.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.924+0000 7f28c50ba640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c007cc40 0x7f28c007f060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00b310 tx=0x7f28bc0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:46.921 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.925+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc015040 con 0x7f28c007cc40 2024-05-09T05:22:46.922 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.925+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5065 (secure 0 0 0) 0x7f28bc024450 con 0x7f28c007cc40 2024-05-09T05:22:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.925+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c007cc40 msgr2=0x7f28c007f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.925+0000 7f28b67fc640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c007cc40 0x7f28c007f060 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f28bc00b310 tx=0x7f28bc0046b0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.925+0000 7f28b67fc640 1 -- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c007cc40 msgr2=0x7f28c007f060 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:46.923 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.925+0000 7f28b67fc640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c007cc40 0x7f28c007f060 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:46.923 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:22:46.923 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:46.924 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:46.924 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:22:46.924 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 132, 2024-05-09T05:22:46.924 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 155485, 2024-05-09T05:22:46.924 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:22:46.924 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:22:46.924 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 15, 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:46.925 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:22:46.926 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:46.927 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:46.928 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:46.929 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:46.930 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:46.931 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.932 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:22:46.933 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:22:46.933 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:46.933 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:46.933 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:46.933 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.933 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.933 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:46.933 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:46.934 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.935 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:46.936 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.937 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:46.938 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:46.939 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:46.940 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:22:46.941 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:22:46.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.929+0000 7f28a3fff640 1 -- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28a4076320 msgr2=0x7f28a4078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28a3fff640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28a4076320 0x7f28a4078830 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f28b8033000 tx=0x7f28b8009000 comp rx=0 tx=0).stop 2024-05-09T05:22:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28a3fff640 1 -- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c00740c0 msgr2=0x7f28c015f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28a3fff640 1 --2- 172.21.15.1:0/1158743102 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28c00740c0 0x7f28c015f7f0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f28b002e8a0 tx=0x7f28b002ed60 comp rx=0 tx=0).stop 2024-05-09T05:22:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28c60bc640 1 -- 172.21.15.1:0/1158743102 reap_dead start 2024-05-09T05:22:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28c60bc640 1 -- 172.21.15.1:0/1158743102 reap_dead start 2024-05-09T05:22:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28a3fff640 1 -- 172.21.15.1:0/1158743102 shutdown_connections 2024-05-09T05:22:46.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28a3fff640 1 -- 172.21.15.1:0/1158743102 >> 172.21.15.1:0/1158743102 conn(0x7f28c0070f40 msgr2=0x7f28c00cca70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:46.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28a3fff640 1 -- 172.21.15.1:0/1158743102 shutdown_connections 2024-05-09T05:22:46.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:46.930+0000 7f28a3fff640 1 -- 172.21.15.1:0/1158743102 wait complete. 2024-05-09T05:22:46.946 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:22:46.946 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:22:46.947 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:22:47.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.140+0000 7fb3a9805640 1 Processor -- start 2024-05-09T05:22:47.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.140+0000 7fb3a9805640 1 -- start start 2024-05-09T05:22:47.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.142+0000 7fb3a9805640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cbf60 0x7fb3a40cd4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.142+0000 7fb3a9805640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb3a40cc370 0x7fb3a40cda00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.142+0000 7fb3a9805640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb3a40c7970 con 0x7fb3a40747f0 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.142+0000 7fb3a9805640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb3a40c5600 con 0x7fb3a40cc370 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.142+0000 7fb3a9805640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a40c7be0 con 0x7fb3a40cbf60 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.142+0000 7fb3a2ffd640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb3a40747f0 0x7fb3a4152c00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57402/0 (socket says 172.21.15.1:57402) 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a2ffd640 1 -- 172.21.15.1:0/573898351 learned_addr learned my addr 172.21.15.1:0/573898351 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a27fc640 1 --2- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cbf60 0x7fb3a40cd4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a37fe640 1 --2- 172.21.15.1:0/573898351 >> v2:172.21.15.196:3300/0 conn(0x7fb3a40cc370 0x7fb3a40cda00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a37fe640 1 -- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cbf60 msgr2=0x7fb3a40cd4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a37fe640 1 --2- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cbf60 0x7fb3a40cd4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a37fe640 1 -- 172.21.15.1:0/573898351 >> v1:172.21.15.1:6789/0 conn(0x7fb3a40747f0 legacy=0x7fb3a4152c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a1ffb640 1 -- 172.21.15.1:0/573898351 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3252930571 0 0) 0x7fb3a40c7970 con 0x7fb3a40747f0 2024-05-09T05:22:47.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a37fe640 1 -- 172.21.15.1:0/573898351 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a40c7ec0 con 0x7fb3a40cc370 2024-05-09T05:22:47.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.143+0000 7fb3a27fc640 1 --2- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cbf60 0x7fb3a40cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:22:47.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.144+0000 7fb3a37fe640 1 --2- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40cc370 0x7fb3a40cda00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb394009d90 tx=0x7fb39402a5d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6336f80fced550fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.144+0000 7fb3a1ffb640 1 -- 172.21.15.1:0/573898351 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394031600 con 0x7fb3a40cc370 2024-05-09T05:22:47.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.144+0000 7fb3a1ffb640 1 -- 172.21.15.1:0/573898351 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb3940317a0 con 0x7fb3a40cc370 2024-05-09T05:22:47.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.145+0000 7fb3a1ffb640 1 -- 172.21.15.1:0/573898351 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb394031b70 con 0x7fb3a40cc370 2024-05-09T05:22:47.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.146+0000 7fb3a9805640 1 -- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40cc370 msgr2=0x7fb3a40cda00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.146+0000 7fb3a9805640 1 --2- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40cc370 0x7fb3a40cda00 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7fb394009d90 tx=0x7fb39402a5d0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.146+0000 7fb3a9805640 1 -- 172.21.15.1:0/573898351 shutdown_connections 2024-05-09T05:22:47.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.146+0000 7fb3a9805640 1 --2- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40cc370 0x7fb3a40cda00 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.146+0000 7fb3a9805640 1 --2- 172.21.15.1:0/573898351 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cbf60 0x7fb3a40cd4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.146+0000 7fb3a9805640 1 -- 172.21.15.1:0/573898351 >> 172.21.15.1:0/573898351 conn(0x7fb3a40ce180 msgr2=0x7fb3a40ce590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:47.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.146+0000 7fb3a9805640 1 -- 172.21.15.1:0/573898351 shutdown_connections 2024-05-09T05:22:47.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.147+0000 7fb3a9805640 1 -- 172.21.15.1:0/573898351 wait complete. 2024-05-09T05:22:47.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.150+0000 7fb3a9805640 1 Processor -- start 2024-05-09T05:22:47.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.151+0000 7fb3a9805640 1 -- start start 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.154+0000 7fb3a9805640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40747f0 0x7fb3a41542e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.155+0000 7fb3a9805640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40cbf60 0x7fb3a4143080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.155+0000 7fb3a9805640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cc370 0x7fb3a40c93a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.155+0000 7fb3a9805640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a40740c0 con 0x7fb3a40cbf60 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.155+0000 7fb3a27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40cbf60 0x7fb3a4143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.155+0000 7fb3a27fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40cbf60 0x7fb3a4143080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46236/0 (socket says 172.21.15.1:46236) 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.155+0000 7fb3a27fc640 1 -- 172.21.15.1:0/1366930601 learned_addr learned my addr 172.21.15.1:0/1366930601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.155+0000 7fb3a9805640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a40c8480 con 0x7fb3a40747f0 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.155+0000 7fb3a9805640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb3a4068650 con 0x7fb3a40cc370 2024-05-09T05:22:47.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.156+0000 7fb3a2ffd640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40747f0 0x7fb3a41542e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.156+0000 7fb3a2ffd640 1 -- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cc370 msgr2=0x7fb3a40c93a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.156+0000 7fb3a2ffd640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb3a40cc370 0x7fb3a40c93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.156+0000 7fb3a2ffd640 1 -- 172.21.15.1:0/1366930601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40cbf60 msgr2=0x7fb3a4143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.156+0000 7fb3a2ffd640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40cbf60 0x7fb3a4143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.156+0000 7fb3a2ffd640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb3a414d9e0 con 0x7fb3a40747f0 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.156+0000 7fb3a27fc640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb3a40cbf60 0x7fb3a4143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.157+0000 7fb3a2ffd640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40747f0 0x7fb3a41542e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00c390 tx=0x7fb39c00c850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.158+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c0074e0 con 0x7fb3a40747f0 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.158+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb39c0047e0 con 0x7fb3a40747f0 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.158+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb39c004c10 con 0x7fb3a40747f0 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.159+0000 7fb3a9805640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb3a414dcc0 con 0x7fb3a40747f0 2024-05-09T05:22:47.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.160+0000 7fb3a9805640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb3a414ee80 con 0x7fb3a40747f0 2024-05-09T05:22:47.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.163+0000 7fb3a9805640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4153c30 0x7fb3a4154820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.163+0000 7fb3a9805640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a406d220 con 0x7fb3a4153c30 2024-05-09T05:22:47.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.164+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb39c011070 con 0x7fb3a40747f0 2024-05-09T05:22:47.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.164+0000 7fb3a27fc640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4153c30 0x7fb3a4154820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.166+0000 7fb39bfff640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3840763a0 0x7fb3840788b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.167+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb39c097130 con 0x7fb3a40747f0 2024-05-09T05:22:47.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.168+0000 7fb3a27fc640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb3a4163260 con 0x7fb3a4153c30 2024-05-09T05:22:47.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.168+0000 7fb3a27fc640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4153c30 0x7fb3a4154820 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4167a60 tx=0x7fb38c012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.168+0000 7fb3a37fe640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3840763a0 0x7fb3840788b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.168+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c01b640 con 0x7fb3a4153c30 2024-05-09T05:22:47.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.169+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fb38c01b640 con 0x7fb3a4153c30 2024-05-09T05:22:47.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.169+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4153c30 msgr2=0x7fb3a4154820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.169+0000 7fb39bfff640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4153c30 0x7fb3a4154820 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4167a60 tx=0x7fb38c012820 comp rx=0 tx=0).stop 2024-05-09T05:22:47.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.169+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4153c30 msgr2=0x7fb3a4154820 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:47.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.169+0000 7fb39bfff640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4153c30 0x7fb3a4154820 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.166 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.170+0000 7fb3a37fe640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3840763a0 0x7fb3840788b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb394034000 tx=0x7fb39402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.181 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.184+0000 7fb3a9805640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4162ce0 0x7fb3a40c98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.184+0000 7fb3a9805640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb3a4050870 con 0x7fb3a4162ce0 2024-05-09T05:22:47.182 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.186+0000 7fb3a27fc640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4162ce0 0x7fb3a40c98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.187+0000 7fb3a27fc640 1 -- 172.21.15.1:0/1366930601 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fb3a4073990 con 0x7fb3a4162ce0 2024-05-09T05:22:47.184 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.188+0000 7fb3a27fc640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4162ce0 0x7fb3a40c98e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4167a60 tx=0x7fb38c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.188+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb38c0219c0 con 0x7fb3a4162ce0 2024-05-09T05:22:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4552 (secure 0 0 0) 0x7fb38c02b420 con 0x7fb3a4162ce0 2024-05-09T05:22:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4162ce0 msgr2=0x7fb3a40c98e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.185 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb39bfff640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4162ce0 0x7fb3a40c98e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4167a60 tx=0x7fb38c0046c0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.186 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb39bfff640 1 -- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4162ce0 msgr2=0x7fb3a40c98e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:47.186 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:22:47.186 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:47.186 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:47.186 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:22:47.186 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 132, 2024-05-09T05:22:47.186 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 20904, 2024-05-09T05:22:47.186 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 15, 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:47.187 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:47.188 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:22:47.189 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:47.190 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.191 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.192 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.193 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.194 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.195 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:47.196 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:47.196 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:47.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:47.196 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.196 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:47.196 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:22:47.196 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.196 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:47.197 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:22:47.198 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.199 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:47.199 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:47.199 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 2 2024-05-09T05:22:47.199 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.199 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.199 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.199 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:22:47.199 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:22:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb39bfff640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a4162ce0 0x7fb3a40c98e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb399ffb640 1 -- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3840763a0 msgr2=0x7fb3840788b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb399ffb640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb3840763a0 0x7fb3840788b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7fb394034000 tx=0x7fb39402e000 comp rx=0 tx=0).stop 2024-05-09T05:22:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb399ffb640 1 -- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40747f0 msgr2=0x7fb3a41542e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb399ffb640 1 --2- 172.21.15.1:0/1366930601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb3a40747f0 0x7fb3a41542e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fb39c00c390 tx=0x7fb39c00c850 comp rx=0 tx=0).stop 2024-05-09T05:22:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb3a37fe640 1 -- 172.21.15.1:0/1366930601 reap_dead start 2024-05-09T05:22:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb3a37fe640 1 -- 172.21.15.1:0/1366930601 reap_dead start 2024-05-09T05:22:47.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb399ffb640 1 -- 172.21.15.1:0/1366930601 shutdown_connections 2024-05-09T05:22:47.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.189+0000 7fb399ffb640 1 -- 172.21.15.1:0/1366930601 >> 172.21.15.1:0/1366930601 conn(0x7fb3a40ce180 msgr2=0x7fb3a414fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:47.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.190+0000 7fb399ffb640 1 -- 172.21.15.1:0/1366930601 shutdown_connections 2024-05-09T05:22:47.201 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.190+0000 7fb399ffb640 1 -- 172.21.15.1:0/1366930601 wait complete. 2024-05-09T05:22:47.207 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:22:47.207 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:22:47.207 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:22:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.399+0000 7f3d189b7640 1 Processor -- start 2024-05-09T05:22:47.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.400+0000 7f3d189b7640 1 -- start start 2024-05-09T05:22:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.402+0000 7f3d189b7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140cf810 0x7f3d140ce3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.402+0000 7f3d189b7640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f3d14160450 0x7f3d140ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.402+0000 7f3d189b7640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3d140d1640 con 0x7f3d14156a80 2024-05-09T05:22:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.402+0000 7f3d189b7640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f3d140d1410 con 0x7f3d14160450 2024-05-09T05:22:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.402+0000 7f3d189b7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d140cc390 con 0x7f3d140cf810 2024-05-09T05:22:47.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.402+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140cf810 0x7f3d140ce3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.402+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140cf810 0x7f3d140ce3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:55214/0 (socket says 172.21.15.1:55214) 2024-05-09T05:22:47.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.402+0000 7f3d137fe640 1 -- 172.21.15.1:0/2015188083 learned_addr learned my addr 172.21.15.1:0/2015188083 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.403+0000 7f3d13fff640 1 --2- 172.21.15.1:0/2015188083 >> v2:172.21.15.196:3300/0 conn(0x7f3d14160450 0x7f3d140ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.403+0000 7f3d137fe640 1 -- 172.21.15.1:0/2015188083 >> v2:172.21.15.196:3300/0 conn(0x7f3d14160450 msgr2=0x7f3d140ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.403+0000 7f3d137fe640 1 --2- 172.21.15.1:0/2015188083 >> v2:172.21.15.196:3300/0 conn(0x7f3d14160450 0x7f3d140ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.403+0000 7f3d137fe640 1 -- 172.21.15.1:0/2015188083 >> v1:172.21.15.196:6789/0 conn(0x7f3d14156a80 legacy=0x7f3d1415fd40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:22:47.400 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.403+0000 7f3d137fe640 1 -- 172.21.15.1:0/2015188083 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d140cc5e0 con 0x7f3d140cf810 2024-05-09T05:22:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.404+0000 7f3d137fe640 1 --2- 172.21.15.1:0/2015188083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140cf810 0x7f3d140ce3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009870 tx=0x7f3d0402a1e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=78b47ca73ffa96d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.405+0000 7f3d127fc640 1 -- 172.21.15.1:0/2015188083 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04037070 con 0x7f3d140cf810 2024-05-09T05:22:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.405+0000 7f3d127fc640 1 -- 172.21.15.1:0/2015188083 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d04032440 con 0x7f3d140cf810 2024-05-09T05:22:47.401 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.405+0000 7f3d127fc640 1 -- 172.21.15.1:0/2015188083 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04032850 con 0x7f3d140cf810 2024-05-09T05:22:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.406+0000 7f3d189b7640 1 -- 172.21.15.1:0/2015188083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140cf810 msgr2=0x7f3d140ce3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.406+0000 7f3d189b7640 1 --2- 172.21.15.1:0/2015188083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140cf810 0x7f3d140ce3c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3d04009870 tx=0x7f3d0402a1e0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.407+0000 7f3d189b7640 1 -- 172.21.15.1:0/2015188083 shutdown_connections 2024-05-09T05:22:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.407+0000 7f3d189b7640 1 --2- 172.21.15.1:0/2015188083 >> v2:172.21.15.196:3300/0 conn(0x7f3d14160450 0x7f3d140ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.407+0000 7f3d189b7640 1 --2- 172.21.15.1:0/2015188083 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140cf810 0x7f3d140ce3c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.407+0000 7f3d189b7640 1 -- 172.21.15.1:0/2015188083 >> 172.21.15.1:0/2015188083 conn(0x7f3d140b3520 msgr2=0x7f3d140b3930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.408+0000 7f3d189b7640 1 -- 172.21.15.1:0/2015188083 shutdown_connections 2024-05-09T05:22:47.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.408+0000 7f3d189b7640 1 -- 172.21.15.1:0/2015188083 wait complete. 2024-05-09T05:22:47.407 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.411+0000 7f3d189b7640 1 Processor -- start 2024-05-09T05:22:47.408 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.412+0000 7f3d189b7640 1 -- start start 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d189b7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d140cf810 0x7f3d140cb200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d189b7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d14156a80 0x7f3d141faf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d189b7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14160450 0x7f3d141592b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d189b7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3d140cce80 con 0x7f3d14160450 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d189b7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3d140ccba0 con 0x7f3d140cf810 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d189b7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d1406f230 con 0x7f3d14156a80 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d140cf810 0x7f3d140cb200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d137fe640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d140cf810 0x7f3d140cb200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:60736/0 (socket says 172.21.15.1:60736) 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d137fe640 1 -- 172.21.15.1:0/488758089 learned_addr learned my addr 172.21.15.1:0/488758089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d13fff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14160450 0x7f3d141592b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.411 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.414+0000 7f3d12ffd640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d14156a80 0x7f3d141faf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.415+0000 7f3d12ffd640 1 -- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d140cf810 msgr2=0x7f3d140cb200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.415+0000 7f3d12ffd640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3d140cf810 0x7f3d140cb200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.415+0000 7f3d12ffd640 1 -- 172.21.15.1:0/488758089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14160450 msgr2=0x7f3d141592b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.415+0000 7f3d12ffd640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14160450 0x7f3d141592b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.415+0000 7f3d12ffd640 1 -- 172.21.15.1:0/488758089 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d141fb460 con 0x7f3d14156a80 2024-05-09T05:22:47.412 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.415+0000 7f3d13fff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3d14160450 0x7f3d141592b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:22:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.415+0000 7f3d12ffd640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d14156a80 0x7f3d141faf20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3d1406cf30 tx=0x7f3d0000b7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.416+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00015020 con 0x7f3d14156a80 2024-05-09T05:22:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.416+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3d00011830 con 0x7f3d14156a80 2024-05-09T05:22:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.416+0000 7f3d189b7640 1 -- 172.21.15.1:0/488758089 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d141fa760 con 0x7f3d14156a80 2024-05-09T05:22:47.413 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.416+0000 7f3d189b7640 1 -- 172.21.15.1:0/488758089 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d140caf20 con 0x7f3d14156a80 2024-05-09T05:22:47.414 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.418+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d00011c40 con 0x7f3d14156a80 2024-05-09T05:22:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.419+0000 7f3d09ffb640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c4590 0x7f3d140c8d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.419+0000 7f3d09ffb640 1 -- 172.21.15.1:0/488758089 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d141f8780 con 0x7f3d140c4590 2024-05-09T05:22:47.415 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.419+0000 7f3d137fe640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c4590 0x7f3d140c8d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.416 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.420+0000 7f3d137fe640 1 -- 172.21.15.1:0/488758089 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3d14050870 con 0x7f3d140c4590 2024-05-09T05:22:47.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.421+0000 7f3d137fe640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c4590 0x7f3d140c8d10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3d040090b0 tx=0x7f3d040046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.417 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.421+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3d00018020 con 0x7f3d14156a80 2024-05-09T05:22:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.422+0000 7f3d0bfff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3cfc076320 0x7f3cfc078830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.422+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3d000973d0 con 0x7f3d14156a80 2024-05-09T05:22:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.422+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04043070 con 0x7f3d140c4590 2024-05-09T05:22:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.422+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f3d04009470 con 0x7f3d140c4590 2024-05-09T05:22:47.419 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.423+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c4590 msgr2=0x7f3d140c8d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.422+0000 7f3d13fff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3cfc076320 0x7f3cfc078830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.423+0000 7f3d0bfff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c4590 0x7f3d140c8d10 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f3d040090b0 tx=0x7f3d040046c0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.423+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c4590 msgr2=0x7f3d140c8d10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.423+0000 7f3d0bfff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c4590 0x7f3d140c8d10 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.420 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.423+0000 7f3d13fff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3cfc076320 0x7f3cfc078830 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3d140b3420 tx=0x7f3d0c00b040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.435+0000 7f3d09ffb640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c7f70 0x7f3d140c8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.436+0000 7f3d09ffb640 1 -- 172.21.15.1:0/488758089 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3d140cb740 con 0x7f3d140c7f70 2024-05-09T05:22:47.435 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.439+0000 7f3d137fe640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c7f70 0x7f3d140c8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.440 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.444+0000 7f3d137fe640 1 -- 172.21.15.1:0/488758089 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f3d140c88c0 con 0x7f3d140c7f70 2024-05-09T05:22:47.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.445+0000 7f3d137fe640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c7f70 0x7f3d140c8380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3d140d0bc0 tx=0x7f3d04008030 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.441 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.445+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d04037040 con 0x7f3d140c7f70 2024-05-09T05:22:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.447+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f3d04003c50 con 0x7f3d140c7f70 2024-05-09T05:22:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.447+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c7f70 msgr2=0x7f3d140c8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.447+0000 7f3d0bfff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c7f70 0x7f3d140c8380 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3d140d0bc0 tx=0x7f3d04008030 comp rx=0 tx=0).stop 2024-05-09T05:22:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.447+0000 7f3d0bfff640 1 -- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c7f70 msgr2=0x7f3d140c8380 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:22:47.443 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.447+0000 7f3d0bfff640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d140c7f70 0x7f3d140c8380 secure :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f3d140d0bc0 tx=0x7f3d04008030 comp rx=0 tx=0).stop 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 132, 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 82270, 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:22:47.444 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 16, 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:47.445 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:47.446 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:22:47.447 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:22:47.448 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.449 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:22:47.450 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:22:47.451 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.452 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:22:47.453 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.454 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:47.455 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:47.456 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:47.457 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:47.457 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.457 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.457 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:22:47.457 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.457 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:47.457 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:47.457 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:22:47.458 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.458 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:22:47.458 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:22:47.458 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:22:47.458 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:22:47.461 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:22:47.462 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:22:47.462 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:22:47.462 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:22:47.462 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:22:47.462 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:22:47.462 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:22:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d189b7640 1 -- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3cfc076320 msgr2=0x7f3cfc078830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d189b7640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3cfc076320 0x7f3cfc078830 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f3d140b3420 tx=0x7f3d0c00b040 comp rx=0 tx=0).stop 2024-05-09T05:22:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d189b7640 1 -- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d14156a80 msgr2=0x7f3d141faf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d189b7640 1 --2- 172.21.15.1:0/488758089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3d14156a80 0x7f3d141faf20 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3d1406cf30 tx=0x7f3d0000b7b0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d13fff640 1 -- 172.21.15.1:0/488758089 reap_dead start 2024-05-09T05:22:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d13fff640 1 -- 172.21.15.1:0/488758089 reap_dead start 2024-05-09T05:22:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d189b7640 1 -- 172.21.15.1:0/488758089 shutdown_connections 2024-05-09T05:22:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d189b7640 1 -- 172.21.15.1:0/488758089 >> 172.21.15.1:0/488758089 conn(0x7f3d140b3520 msgr2=0x7f3d14155020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d189b7640 1 -- 172.21.15.1:0/488758089 shutdown_connections 2024-05-09T05:22:47.466 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.448+0000 7f3d189b7640 1 -- 172.21.15.1:0/488758089 wait complete. 2024-05-09T05:22:47.466 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:22:47.467 INFO:tasks.mon_thrash.mon_thrasher:monitors to thrash: ['b'] 2024-05-09T05:22:47.467 INFO:tasks.mon_thrash.mon_thrasher:monitors to freeze: [] 2024-05-09T05:22:47.467 INFO:tasks.mon_thrash.mon_thrasher:thrashing mon.b 2024-05-09T05:22:47.467 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:22:47.468 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T05:22:47.468 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:22:47.481 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T05:22:47.481 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 2 2024-05-09T05:22:47.481 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.680+0000 7f599b7eb640 1 Processor -- start 2024-05-09T05:22:47.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.680+0000 7f599b7eb640 1 -- start start 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.682+0000 7f599b7eb640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f59940c4ba0 0x7f59940c8660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f599b7eb640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59940c3a10 con 0x7f59940c4650 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f599b7eb640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f59940c3cd0 con 0x7f599415afd0 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f599b7eb640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f59940c8fd0 con 0x7f59940c4ba0 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f5999587640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f59940c4ba0 msgr2=0x7f59940c8660 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f5999587640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f59940c4ba0 0x7f59940c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f5999d88640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f599415afd0 legacy=0x7f599406e9e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f5998d86640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f59940c4650 0x7f59940c9740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:57408/0 (socket says 172.21.15.1:57408) 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f5998d86640 1 -- 172.21.15.1:0/3723507087 learned_addr learned my addr 172.21.15.1:0/3723507087 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1896827327 0 0) 0x7f59940c3a10 con 0x7f59940c4650 2024-05-09T05:22:47.683 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.683+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f597c003520 con 0x7f59940c4650 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.684+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3094356348 0 0) 0x7f597c003520 con 0x7f59940c4650 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.684+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 >> v2:172.21.15.196:3300/0 conn(0x7f59940c4ba0 msgr2=0x7f59940c8660 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.684+0000 7f5993fff640 1 --2- 172.21.15.1:0/3723507087 >> v2:172.21.15.196:3300/0 conn(0x7f59940c4ba0 0x7f59940c8660 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.684+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 >> v1:172.21.15.196:6789/0 conn(0x7f599415afd0 legacy=0x7f599406e9e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.684+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f59940c3fb0 con 0x7f59940c4650 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.684+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f5984001a90 con 0x7f59940c4650 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.684+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 >> v1:172.21.15.1:6789/0 conn(0x7f59940c4650 legacy=0x7f59940c9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.685+0000 7f5993fff640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f597c003a00 0x7f597c003e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.685+0000 7f5993fff640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f597c004800 0x7f597c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.685+0000 7f5993fff640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c0053a0 0x7f597c00ba50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.685+0000 7f5999587640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f597c003a00 0x7f597c003e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.685+0000 7f5998d86640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f597c004800 0x7f597c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.685+0000 7f5999d88640 1 -- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c0053a0 msgr2=0x7f597c00ba50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:47.684 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.685+0000 7f5999d88640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c0053a0 0x7f597c00ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.685+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f59940c8fd0 con 0x7f597c004800 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.686+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f597c00c330 con 0x7f597c0053a0 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.686+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f597c00c530 con 0x7f597c003a00 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.686+0000 7f5998d86640 1 -- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f597c003a00 msgr2=0x7f597c003e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.686+0000 7f5998d86640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f597c003a00 0x7f597c003e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.686+0000 7f5998d86640 1 -- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c0053a0 msgr2=0x7f597c00ba50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.686+0000 7f5998d86640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c0053a0 0x7f597c00ba50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.686+0000 7f5999d88640 1 -- 172.21.15.1:0/3723507087 reap_dead start 2024-05-09T05:22:47.685 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.686+0000 7f5998d86640 1 -- 172.21.15.1:0/3723507087 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f597c003520 con 0x7f597c004800 2024-05-09T05:22:47.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.690+0000 7f5998d86640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f597c004800 0x7f597c004c10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5994074ca0 tx=0x7f597c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de1e94bedaf8308a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.690+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598400a430 con 0x7f597c004800 2024-05-09T05:22:47.686 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.690+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f598402f070 con 0x7f597c004800 2024-05-09T05:22:47.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.691+0000 7f5993fff640 1 -- 172.21.15.1:0/3723507087 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598402bb30 con 0x7f597c004800 2024-05-09T05:22:47.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.691+0000 7f599b7eb640 1 -- 172.21.15.1:0/3723507087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f597c004800 msgr2=0x7f597c004c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.687 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.691+0000 7f599b7eb640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f597c004800 0x7f597c004c10 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5994074ca0 tx=0x7f597c005830 comp rx=0 tx=0).stop 2024-05-09T05:22:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.691+0000 7f599b7eb640 1 -- 172.21.15.1:0/3723507087 shutdown_connections 2024-05-09T05:22:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.692+0000 7f599b7eb640 1 --2- 172.21.15.1:0/3723507087 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f597c004800 0x7f597c004c10 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.692+0000 7f599b7eb640 1 -- 172.21.15.1:0/3723507087 >> 172.21.15.1:0/3723507087 conn(0x7f59940ce290 msgr2=0x7f59940ca760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.692+0000 7f599b7eb640 1 -- 172.21.15.1:0/3723507087 shutdown_connections 2024-05-09T05:22:47.688 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.692+0000 7f599b7eb640 1 -- 172.21.15.1:0/3723507087 wait complete. 2024-05-09T05:22:47.691 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.695+0000 7f599b7eb640 1 Processor -- start 2024-05-09T05:22:47.694 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.697+0000 7f599b7eb640 1 -- start start 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f599b7eb640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c004800 0x7f59942026c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f599b7eb640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59940cbfe0 0x7f5994154020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f599b7eb640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f59941f49f0 0x7f59941f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f599b7eb640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5994164100 con 0x7f59940cbfe0 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f599b7eb640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f59940c9220 con 0x7f597c004800 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f599b7eb640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5994164280 con 0x7f59941f49f0 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f5999587640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c004800 msgr2=0x7f59942026c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f5999587640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c004800 0x7f59942026c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:47.697 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f5998d86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59940cbfe0 0x7f5994154020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f5998d86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59940cbfe0 0x7f5994154020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:46252/0 (socket says 172.21.15.1:46252) 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f5998d86640 1 -- 172.21.15.1:0/1887396251 learned_addr learned my addr 172.21.15.1:0/1887396251 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f5999d88640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f59941f49f0 0x7f59941f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.700+0000 7f5998d86640 1 -- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f59941f49f0 msgr2=0x7f59941f4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5998d86640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f59941f49f0 0x7f59941f4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5998d86640 1 -- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c004800 msgr2=0x7f59942026c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5998d86640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c004800 0x7f59942026c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5998d86640 1 -- 172.21.15.1:0/1887396251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5994155de0 con 0x7f59940cbfe0 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5999d88640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f59941f49f0 0x7f59941f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:22:47.698 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5998d86640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59940cbfe0 0x7f5994154020 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f598400a8d0 tx=0x7f5984001c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5991ffb640 1 -- 172.21.15.1:0/1887396251 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f598400a430 con 0x7f59940cbfe0 2024-05-09T05:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5991ffb640 1 -- 172.21.15.1:0/1887396251 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5984034470 con 0x7f59940cbfe0 2024-05-09T05:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f59941560c0 con 0x7f59940cbfe0 2024-05-09T05:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f5991ffb640 1 -- 172.21.15.1:0/1887396251 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5984034860 con 0x7f59940cbfe0 2024-05-09T05:22:47.699 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.701+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5994156d40 con 0x7f59940cbfe0 2024-05-09T05:22:47.703 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.707+0000 7f5991ffb640 1 -- 172.21.15.1:0/1887396251 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5984034a00 con 0x7f59940cbfe0 2024-05-09T05:22:47.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.708+0000 7f5991ffb640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5980076280 0x7f5980078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:47.704 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.708+0000 7f5991ffb640 1 -- 172.21.15.1:0/1887396251 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f59840b7ea0 con 0x7f59940cbfe0 2024-05-09T05:22:47.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.709+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5994050870 con 0x7f59940cbfe0 2024-05-09T05:22:47.705 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.709+0000 7f5999587640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5980076280 0x7f5980078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:47.707 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.711+0000 7f5999587640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5980076280 0x7f5980078790 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f598c003930 tx=0x7f598c0090e0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:47.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:47.721+0000 7f5991ffb640 1 -- 172.21.15.1:0/1887396251 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f59840815d0 con 0x7f59940cbfe0 2024-05-09T05:22:48.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.254+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f599415e4d0 con 0x7f59940cbfe0 2024-05-09T05:22:48.250 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.254+0000 7f5991ffb640 1 -- 172.21.15.1:0/1887396251 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f5984080ed0 con 0x7f59940cbfe0 2024-05-09T05:22:48.251 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":16,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:22:48.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5980076280 msgr2=0x7f5980078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:48.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5980076280 0x7f5980078790 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f598c003930 tx=0x7f598c0090e0 comp rx=0 tx=0).stop 2024-05-09T05:22:48.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59940cbfe0 msgr2=0x7f5994154020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:48.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59940cbfe0 0x7f5994154020 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f598400a8d0 tx=0x7f5984001c30 comp rx=0 tx=0).stop 2024-05-09T05:22:48.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 shutdown_connections 2024-05-09T05:22:48.254 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5980076280 0x7f5980078790 secure :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7f598c003930 tx=0x7f598c0090e0 comp rx=0 tx=0).stop 2024-05-09T05:22:48.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f59941f49f0 0x7f59941f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:48.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f59940cbfe0 0x7f5994154020 secure :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f598400a8d0 tx=0x7f5984001c30 comp rx=0 tx=0).stop 2024-05-09T05:22:48.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 --2- 172.21.15.1:0/1887396251 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f597c004800 0x7f59942026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:48.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 >> 172.21.15.1:0/1887396251 conn(0x7f59940ce290 msgr2=0x7f59941586d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:48.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.258+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 shutdown_connections 2024-05-09T05:22:48.255 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:48.259+0000 7f599b7eb640 1 -- 172.21.15.1:0/1887396251 wait complete. 2024-05-09T05:22:51.232 INFO:tasks.workunit.client.0.smithi001.stdout:Syncing()...1 sec 2024-05-09T05:22:51.232 INFO:tasks.workunit.client.0.smithi001.stdout:FFSB benchmark finished at: Thu May 9 05:22:51 2024 2024-05-09T05:22:51.232 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.232 INFO:tasks.workunit.client.0.smithi001.stdout:Results: 2024-05-09T05:22:51.232 INFO:tasks.workunit.client.0.smithi001.stdout:Benchmark took 305.21 sec 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout:Total Results 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout:=============== 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout: Op Name Transactions Trans/sec % Trans % Op Weight Throughput 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout: ======= ============ ========= ======= =========== ========== 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout: write : 1817600 5955.28 100.000% 100.000% 23.3MB/sec 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout:- 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout:5955.28 Transactions per Second 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout:Throughput Results 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout:=================== 2024-05-09T05:22:51.233 INFO:tasks.workunit.client.0.smithi001.stdout:Write Throughput: 23.3MB/sec 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout:System Call Latency statistics in millisecs 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout:===== 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout: Min Avg Max Total Calls 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout: ======== ======== ======== ============ 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout:[ open] 0.532000 19.282909 1446.785034 1420 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-05-09T05:22:51.242 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[2] 1000.000000 - 2000.000000 : 2 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[3] 500.000000 - 1000.000000 : 1 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[4] 200.000000 - 500.000000 : 2 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[5] 100.000000 - 200.000000 : 1 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[6] 50.000000 - 100.000000 : 13 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[7] 20.000000 - 50.000000 : 323 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[8] 10.000000 - 20.000000 : 818 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[9] 5.000000 - 10.000000 : 201 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[10] 2.000000 - 5.000000 : 50 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[11] 1.000000 - 2.000000 : 5 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[12] 0.500000 - 1.000000 : 4 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[14] 0.100000 - 0.200000 : 0 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-05-09T05:22:51.243 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout:[ write] 0.102000 5.303207 190.108002 1817600 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[5] 100.000000 - 200.000000 : 60 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[6] 50.000000 - 100.000000 : 285 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[7] 20.000000 - 50.000000 : 38699 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[8] 10.000000 - 20.000000 : 188444 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[9] 5.000000 - 10.000000 : 496329 2024-05-09T05:22:51.244 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[10] 2.000000 - 5.000000 : 664643 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[11] 1.000000 - 2.000000 : 179182 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[12] 0.500000 - 1.000000 : 23224 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[13] 0.200000 - 0.500000 : 132544 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[14] 0.100000 - 0.200000 : 94190 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[15] 0.050000 - 0.100000 : 0 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[16] 0.020000 - 0.050000 : 0 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout:[ lseek] 0.000000 0.001077 0.140000 1817600 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-05-09T05:22:51.245 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[7] 20.000000 - 50.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[8] 10.000000 - 20.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[9] 5.000000 - 10.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[10] 2.000000 - 5.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[11] 1.000000 - 2.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[12] 0.500000 - 1.000000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[13] 0.200000 - 0.500000 : 0 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[14] 0.100000 - 0.200000 : 2 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[15] 0.050000 - 0.100000 : 2 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[16] 0.020000 - 0.050000 : 219 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[17] 0.010000 - 0.020000 : 1166 2024-05-09T05:22:51.246 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[18] 0.000000 - 0.010000 : 1816211 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout:[ close] 0.021000 2.468089 46.630001 1420 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[0] 5000.000000 - 10000.000000 : 0 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[1] 2000.000000 - 5000.000000 : 0 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[2] 1000.000000 - 2000.000000 : 0 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[3] 500.000000 - 1000.000000 : 0 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[4] 200.000000 - 500.000000 : 0 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[5] 100.000000 - 200.000000 : 0 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[6] 50.000000 - 100.000000 : 0 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[7] 20.000000 - 50.000000 : 17 2024-05-09T05:22:51.247 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[8] 10.000000 - 20.000000 : 66 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[9] 5.000000 - 10.000000 : 114 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[10] 2.000000 - 5.000000 : 280 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[11] 1.000000 - 2.000000 : 251 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[12] 0.500000 - 1.000000 : 90 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[13] 0.200000 - 0.500000 : 18 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[14] 0.100000 - 0.200000 : 56 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[15] 0.050000 - 0.100000 : 115 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[16] 0.020000 - 0.050000 : 413 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[17] 0.010000 - 0.020000 : 0 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: msec_range[18] 0.000000 - 0.010000 : 0 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout: 2024-05-09T05:22:51.248 INFO:tasks.workunit.client.0.smithi001.stdout:1.5% User Time 2024-05-09T05:22:51.249 INFO:tasks.workunit.client.0.smithi001.stdout:15.2% System Time 2024-05-09T05:22:51.249 INFO:tasks.workunit.client.0.smithi001.stdout:16.7% CPU Utilization 2024-05-09T05:22:51.249 INFO:tasks.workunit.client.0.smithi001.stderr:+ cd .. 2024-05-09T05:22:51.253 INFO:tasks.workunit.client.0.smithi001.stderr:+ rm -r tmp ffsb 2024-05-09T05:22:51.273 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:51.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.426+0000 7f1cf607e640 1 Processor -- start 2024-05-09T05:22:51.422 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.426+0000 7f1cf607e640 1 -- start start 2024-05-09T05:22:51.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1cf607e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1cf0072120 0x7f1cf00c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:51.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1cf607e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1cf607e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1cf015b110 con 0x7f1cf00ca340 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1cf607e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f1cf00c53f0 con 0x7f1cf0072120 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1cf607e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf00c55b0 con 0x7f1cf00c62e0 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1cef7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1cf00ca340 0x7f1cf01547b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:41624/0 (socket says 172.21.15.1:41624) 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1cef7fe640 1 -- 172.21.15.1:0/1838631737 learned_addr learned my addr 172.21.15.1:0/1838631737 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1ceeffd640 1 -- 172.21.15.1:0/1838631737 >> v2:172.21.15.196:3300/0 conn(0x7f1cf0072120 msgr2=0x7f1cf00c4eb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.427+0000 7f1ceeffd640 1 --2- 172.21.15.1:0/1838631737 >> v2:172.21.15.196:3300/0 conn(0x7f1cf0072120 0x7f1cf00c4eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.428+0000 7f1ceffff640 1 --2- 172.21.15.1:0/1838631737 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.428+0000 7f1cee7fc640 1 -- 172.21.15.1:0/1838631737 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2977080365 0 0) 0x7f1cf015b110 con 0x7f1cf00ca340 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.428+0000 7f1cee7fc640 1 -- 172.21.15.1:0/1838631737 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1cd8003520 con 0x7f1cf00ca340 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.428+0000 7f1cee7fc640 1 -- 172.21.15.1:0/1838631737 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 2673398238 0 0) 0x7f1cd8003520 con 0x7f1cf00ca340 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.428+0000 7f1ceffff640 1 -- 172.21.15.1:0/1838631737 >> v2:172.21.15.196:3300/0 conn(0x7f1cf0072120 msgr2=0x7f1cf00c4eb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.428+0000 7f1ceffff640 1 --2- 172.21.15.1:0/1838631737 >> v2:172.21.15.196:3300/0 conn(0x7f1cf0072120 0x7f1cf00c4eb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.425 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.428+0000 7f1ceffff640 1 -- 172.21.15.1:0/1838631737 >> v1:172.21.15.1:6789/0 conn(0x7f1cf00ca340 legacy=0x7f1cf01547b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:51.426 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.428+0000 7f1ceffff640 1 -- 172.21.15.1:0/1838631737 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf015b3b0 con 0x7f1cf00c62e0 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.429+0000 7f1ceffff640 1 --2- 172.21.15.1:0/1838631737 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0070e20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009d90 tx=0x7f1ce802a570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f922bbcb4a273d03 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.429+0000 7f1cee7fc640 1 -- 172.21.15.1:0/1838631737 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce00016f0 con 0x7f1cf00c62e0 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.429+0000 7f1cee7fc640 1 -- 172.21.15.1:0/1838631737 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce8007540 con 0x7f1cf00c62e0 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.429+0000 7f1cee7fc640 1 -- 172.21.15.1:0/1838631737 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce80078f0 con 0x7f1cf00c62e0 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.430+0000 7f1cf607e640 1 -- 172.21.15.1:0/1838631737 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 msgr2=0x7f1cf0070e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.430+0000 7f1cf607e640 1 --2- 172.21.15.1:0/1838631737 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0070e20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1ce8009d90 tx=0x7f1ce802a570 comp rx=0 tx=0).stop 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.430+0000 7f1cf607e640 1 -- 172.21.15.1:0/1838631737 shutdown_connections 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.430+0000 7f1cf607e640 1 --2- 172.21.15.1:0/1838631737 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0070e20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.430+0000 7f1cf607e640 1 --2- 172.21.15.1:0/1838631737 >> v2:172.21.15.196:3300/0 conn(0x7f1cf0072120 0x7f1cf00c4eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.430+0000 7f1cf607e640 1 -- 172.21.15.1:0/1838631737 >> 172.21.15.1:0/1838631737 conn(0x7f1cf00731d0 msgr2=0x7f1cf00cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:51.427 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.430+0000 7f1cf607e640 1 -- 172.21.15.1:0/1838631737 shutdown_connections 2024-05-09T05:22:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.430+0000 7f1cf607e640 1 -- 172.21.15.1:0/1838631737 wait complete. 2024-05-09T05:22:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.432+0000 7f1cf607e640 1 Processor -- start 2024-05-09T05:22:51.428 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.432+0000 7f1cf607e640 1 -- start start 2024-05-09T05:22:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cf607e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cf0072120 0x7f1cf015bc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:51.430 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cf607e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0151680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cf607e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf00ca340 0x7f1cf00ca750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cf607e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf00c5dc0 con 0x7f1cf0072120 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cf607e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1cf00c5ae0 con 0x7f1cf00ca340 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cf607e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1cf00c66f0 con 0x7f1cf00c62e0 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cf0072120 0x7f1cf015bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1ceffff640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf00ca340 msgr2=0x7f1cf00ca750 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1ceffff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf00ca340 0x7f1cf00ca750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cef7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cf0072120 0x7f1cf015bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47986/0 (socket says 172.21.15.1:47986) 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1cef7fe640 1 -- 172.21.15.1:0/3269947478 learned_addr learned my addr 172.21.15.1:0/3269947478 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.433+0000 7f1ceeffd640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0151680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cef7fe640 1 -- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 msgr2=0x7f1cf0151680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cef7fe640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0151680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cef7fe640 1 -- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf00ca340 msgr2=0x7f1cf00ca750 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:51.431 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cef7fe640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf00ca340 0x7f1cf00ca750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cef7fe640 1 -- 172.21.15.1:0/3269947478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1cf015e820 con 0x7f1cf0072120 2024-05-09T05:22:51.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cef7fe640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cf0072120 0x7f1cf015bc30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00013c0 tx=0x7f1ce000cb10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:51.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cd7fff640 1 -- 172.21.15.1:0/3269947478 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0005ad0 con 0x7f1cf0072120 2024-05-09T05:22:51.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cd7fff640 1 -- 172.21.15.1:0/3269947478 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1ce000ce50 con 0x7f1cf0072120 2024-05-09T05:22:51.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cd7fff640 1 -- 172.21.15.1:0/3269947478 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ce0007070 con 0x7f1cf0072120 2024-05-09T05:22:51.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1cf015e260 con 0x7f1cf0072120 2024-05-09T05:22:51.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.434+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1cf015d2a0 con 0x7f1cf0072120 2024-05-09T05:22:51.432 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.435+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1cf0050870 con 0x7f1cf0072120 2024-05-09T05:22:51.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.437+0000 7f1cd7fff640 1 -- 172.21.15.1:0/3269947478 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1ce0007210 con 0x7f1cf0072120 2024-05-09T05:22:51.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.437+0000 7f1cd7fff640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1cdc0762b0 0x7f1cdc0787c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:51.433 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.437+0000 7f1cd7fff640 1 -- 172.21.15.1:0/3269947478 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1ce0097250 con 0x7f1cf0072120 2024-05-09T05:22:51.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.438+0000 7f1ceeffd640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1cdc0762b0 0x7f1cdc0787c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:51.434 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.438+0000 7f1ceeffd640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1cdc0762b0 0x7f1cdc0787c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1cf00b49f0 tx=0x7f1ce4003b90 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:51.438 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.442+0000 7f1cd7fff640 1 -- 172.21.15.1:0/3269947478 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f1ce0060b30 con 0x7f1cf0072120 2024-05-09T05:22:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.716+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f1cf015ea40 con 0x7f1cf0072120 2024-05-09T05:22:51.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.716+0000 7f1cd7fff640 1 -- 172.21.15.1:0/3269947478 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f1ce0060430 con 0x7f1cf0072120 2024-05-09T05:22:51.713 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":20,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1cdc0762b0 msgr2=0x7f1cdc0787c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1cdc0762b0 0x7f1cdc0787c0 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7f1cf00b49f0 tx=0x7f1ce4003b90 comp rx=0 tx=0).stop 2024-05-09T05:22:51.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cf0072120 msgr2=0x7f1cf015bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cf0072120 0x7f1cf015bc30 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1ce00013c0 tx=0x7f1ce000cb10 comp rx=0 tx=0).stop 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 shutdown_connections 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1cdc0762b0 0x7f1cdc0787c0 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1cf00ca340 0x7f1cf00ca750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1cf00c62e0 0x7f1cf0151680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 --2- 172.21.15.1:0/3269947478 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1cf0072120 0x7f1cf015bc30 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 >> 172.21.15.1:0/3269947478 conn(0x7f1cf00731d0 msgr2=0x7f1cf00b4a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 shutdown_connections 2024-05-09T05:22:51.716 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:51.719+0000 7f1cf607e640 1 -- 172.21.15.1:0/3269947478 wait complete. 2024-05-09T05:22:53.420 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-09T05:22:53.421 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-09T05:22:53.459 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-05-09T05:22:53.459 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-09T05:22:53.771 DEBUG:teuthology.parallel:result is None 2024-05-09T05:22:53.771 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T05:22:53.807 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T05:22:53.807 DEBUG:teuthology.run_tasks:Unwinding manager mon_thrash 2024-05-09T05:22:53.820 INFO:tasks.mon_thrash:joining mon_thrasher 2024-05-09T05:22:54.728 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:54.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.888+0000 7f5fd4fdd640 1 Processor -- start 2024-05-09T05:22:54.885 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.888+0000 7f5fd4fdd640 1 -- start start 2024-05-09T05:22:54.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.891+0000 7f5fd4fdd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0152a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:54.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.891+0000 7f5fd4fdd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f5fd0072e20 0x7f5fd0159290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:54.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.891+0000 7f5fd4fdd640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f5fd01597d0 con 0x7f5fd00cd760 2024-05-09T05:22:54.887 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.891+0000 7f5fd4fdd640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f5fd00c8de0 con 0x7f5fd0072e20 2024-05-09T05:22:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.891+0000 7f5fd4fdd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd0152fb0 con 0x7f5fd00c47c0 2024-05-09T05:22:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.891+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0152a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.891+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0152a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59292/0 (socket says 172.21.15.1:59292) 2024-05-09T05:22:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.891+0000 7f5fce59c640 1 -- 172.21.15.1:0/1826852772 learned_addr learned my addr 172.21.15.1:0/1826852772 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.892+0000 7f5fce59c640 1 -- 172.21.15.1:0/1826852772 >> v2:172.21.15.196:3300/0 conn(0x7f5fd0072e20 msgr2=0x7f5fd0159290 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:22:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.892+0000 7f5fce59c640 1 --2- 172.21.15.1:0/1826852772 >> v2:172.21.15.196:3300/0 conn(0x7f5fd0072e20 0x7f5fd0159290 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.892+0000 7f5fce59c640 1 -- 172.21.15.1:0/1826852772 >> v1:172.21.15.196:6789/0 conn(0x7f5fd00cd760 legacy=0x7f5fd00c9270 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T05:22:54.888 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.892+0000 7f5fce59c640 1 -- 172.21.15.1:0/1826852772 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd00b35e0 con 0x7f5fd00c47c0 2024-05-09T05:22:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.893+0000 7f5fce59c640 1 --2- 172.21.15.1:0/1826852772 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0152a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00719b0 tx=0x7f5fc002a250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d25eac4441884194 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.895+0000 7f5fcdd9b640 1 -- 172.21.15.1:0/1826852772 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0037070 con 0x7f5fd00c47c0 2024-05-09T05:22:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.895+0000 7f5fcdd9b640 1 -- 172.21.15.1:0/1826852772 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc0032440 con 0x7f5fd00c47c0 2024-05-09T05:22:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.896+0000 7f5fcdd9b640 1 -- 172.21.15.1:0/1826852772 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc0032810 con 0x7f5fd00c47c0 2024-05-09T05:22:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.897+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/1826852772 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 msgr2=0x7f5fd0152a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.897+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/1826852772 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0152a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5fd00719b0 tx=0x7f5fc002a250 comp rx=0 tx=0).stop 2024-05-09T05:22:54.893 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.897+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/1826852772 shutdown_connections 2024-05-09T05:22:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.897+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/1826852772 >> v2:172.21.15.196:3300/0 conn(0x7f5fd0072e20 0x7f5fd0159290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.897+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/1826852772 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0152a70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.897+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/1826852772 >> 172.21.15.1:0/1826852772 conn(0x7f5fd0070dd0 msgr2=0x7f5fd00711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.897+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/1826852772 shutdown_connections 2024-05-09T05:22:54.894 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.897+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/1826852772 wait complete. 2024-05-09T05:22:54.896 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.900+0000 7f5fd4fdd640 1 Processor -- start 2024-05-09T05:22:54.898 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.901+0000 7f5fd4fdd640 1 -- start start 2024-05-09T05:22:54.899 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.903+0000 7f5fd4fdd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0072e20 0x7f5fd00c5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.903+0000 7f5fce59c640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0072e20 msgr2=0x7f5fd00c5b00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:22:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.903+0000 7f5fce59c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0072e20 0x7f5fd00c5b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.903+0000 7f5fd4fdd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.903+0000 7f5fd4fdd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00cd760 0x7f5fd0155870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.904+0000 7f5fc5d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.904+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00cd760 0x7f5fd0155870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:54.900 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.904+0000 7f5fc5d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59304/0 (socket says 172.21.15.1:59304) 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.904+0000 7f5fced9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00cd760 0x7f5fd0155870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47994/0 (socket says 172.21.15.1:47994) 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.904+0000 7f5fd4fdd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd00b3ba0 con 0x7f5fd00cd760 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.904+0000 7f5fd4fdd640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f5fd00c82f0 con 0x7f5fd0072e20 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.904+0000 7f5fd4fdd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f5fd00c2b30 con 0x7f5fd00c47c0 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.904+0000 7f5fc5d9b640 1 -- 172.21.15.1:0/2267463018 learned_addr learned my addr 172.21.15.1:0/2267463018 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fced9d640 1 -- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 msgr2=0x7f5fd0155330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fced9d640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0155330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fced9d640 1 -- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0072e20 msgr2=0x7f5fd00c5b00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fced9d640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0072e20 0x7f5fd00c5b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fced9d640 1 -- 172.21.15.1:0/2267463018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fd00c6040 con 0x7f5fd00cd760 2024-05-09T05:22:54.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fced9d640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00cd760 0x7f5fd0155870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800bdf0 tx=0x7f5fc8009ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fc77fe640 1 -- 172.21.15.1:0/2267463018 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8019070 con 0x7f5fd00cd760 2024-05-09T05:22:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fc77fe640 1 -- 172.21.15.1:0/2267463018 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f5fc8014440 con 0x7f5fd00cd760 2024-05-09T05:22:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fc77fe640 1 -- 172.21.15.1:0/2267463018 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8014830 con 0x7f5fd00cd760 2024-05-09T05:22:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fd0151c20 con 0x7f5fd00cd760 2024-05-09T05:22:54.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.905+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fd015b800 con 0x7f5fd00cd760 2024-05-09T05:22:54.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.907+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fd00696d0 con 0x7f5fd00cd760 2024-05-09T05:22:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.908+0000 7f5fc77fe640 1 -- 172.21.15.1:0/2267463018 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5fc801c070 con 0x7f5fd00cd760 2024-05-09T05:22:54.905 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.909+0000 7f5fc77fe640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5fb0076280 0x7f5fb0078790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:54.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.909+0000 7f5fc77fe640 1 -- 172.21.15.1:0/2267463018 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f5fc8097610 con 0x7f5fd00cd760 2024-05-09T05:22:54.909 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.913+0000 7f5fce59c640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5fb0076280 0x7f5fb0078790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:54.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.915+0000 7f5fce59c640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5fb0076280 0x7f5fb0078790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0034000 tx=0x7f5fc002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:22:54.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:54.917+0000 7f5fc77fe640 1 -- 172.21.15.1:0/2267463018 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f5fc8060780 con 0x7f5fd00cd760 2024-05-09T05:22:55.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.379+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f5fd0050870 con 0x7f5fd00cd760 2024-05-09T05:22:55.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.379+0000 7f5fc77fe640 1 -- 172.21.15.1:0/2267463018 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7f5fc8064a80 con 0x7f5fd00cd760 2024-05-09T05:22:55.376 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":132,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":24,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:22:55.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5fb0076280 msgr2=0x7f5fb0078790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:55.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5fb0076280 0x7f5fb0078790 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0034000 tx=0x7f5fc002e000 comp rx=0 tx=0).stop 2024-05-09T05:22:55.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00cd760 msgr2=0x7f5fd0155870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:22:55.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00cd760 0x7f5fd0155870 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800bdf0 tx=0x7f5fc8009ef0 comp rx=0 tx=0).stop 2024-05-09T05:22:55.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 shutdown_connections 2024-05-09T05:22:55.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f5fb0076280 0x7f5fb0078790 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:55.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f5fd00cd760 0x7f5fd0155870 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:55.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f5fd00c47c0 0x7f5fd0155330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:55.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 --2- 172.21.15.1:0/2267463018 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f5fd0072e20 0x7f5fd00c5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:22:55.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 >> 172.21.15.1:0/2267463018 conn(0x7f5fd0070dd0 msgr2=0x7f5fd00711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:22:55.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 shutdown_connections 2024-05-09T05:22:55.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:55.382+0000 7f5fd4fdd640 1 -- 172.21.15.1:0/2267463018 wait complete. 2024-05-09T05:22:58.396 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:22:58.566 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.570+0000 7f08d2ecd640 1 Processor -- start 2024-05-09T05:22:58.567 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.570+0000 7f08d2ecd640 1 -- start start 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.572+0000 7f08d2ecd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.572+0000 7f08d2ecd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.572+0000 7f08d2ecd640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08cc160c90 con 0x7f08cc072120 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.572+0000 7f08d2ecd640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f08cc161b80 con 0x7f08cc15ad00 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.572+0000 7f08d2ecd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f08cc160f70 con 0x7f08cc071360 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08cbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08cbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47996/0 (socket says 172.21.15.1:47996) 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 learned_addr learned my addr 172.21.15.1:0/662566045 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:22:58.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:22:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:22:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:58.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.573+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:22:58.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:58.771 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:58.774+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:22:59.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:59.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.175+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:22:59.973 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:22:59.976+0000 7f08d146a640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:23:01.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 msgr2=0x7f08cc0ca620 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08ca7fc640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08cc071360 0x7f08cc0ca620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 msgr2=0x7f08cc070e20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08ca7fc640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.1:3300/0 conn(0x7f08cc15ad00 0x7f08cc070e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.1:6789/0 conn(0x7f08cc072120 legacy=0x7f08cc06dbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08ca7fc640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 0x7f08b8009940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08cc161250 con 0x7f08b80036c0 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f08cc160f70 con 0x7f08b8007430 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 msgr2=0x7f08b8009940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08d0c69640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.196:6789/0 conn(0x7f08b80036c0 legacy=0x7f08b8005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:23:01.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.573+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 0x7f08b8009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:01.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.774+0000 7f08d0c69640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.196:6789/0 conn(0x7f08b80036c0 legacy=0x7f08b8005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:23:01.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.774+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 msgr2=0x7f08b8009940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:01.770 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:01.774+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 0x7f08b8009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T05:23:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:02.175+0000 7f08d0c69640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.196:6789/0 conn(0x7f08b80036c0 legacy=0x7f08b8005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:23:02.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:02.175+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 msgr2=0x7f08b8009940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:02.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:02.175+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 0x7f08b8009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T05:23:02.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:02.976+0000 7f08cbfff640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 msgr2=0x7f08b8009940 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:02.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:02.976+0000 7f08d0c69640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.196:6789/0 conn(0x7f08b80036c0 legacy=0x7f08b8005b90 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:23:02.972 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:02.976+0000 7f08cbfff640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 0x7f08b8009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T05:23:04.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 msgr2=0x7f08b8009940 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:04.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08ca7fc640 1 --2- 172.21.15.1:0/662566045 >> v2:172.21.15.196:3300/0 conn(0x7f08b8007430 0x7f08b8009940 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.196:6789/0 conn(0x7f08b80036c0 legacy=0x7f08b8005b90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:04.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 reap_dead start 2024-05-09T05:23:04.570 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08ca7fc640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8009fe0 0x7f08b8009870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08b800b0c0 con 0x7f08b800a3f0 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f08cc161250 con 0x7f08b8006c90 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08ca7fc640 1 -- 172.21.15.1:0/662566045 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f08cc160f70 con 0x7f08b8009fe0 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08d146a640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.196:6789/0 conn(0x7f08b8006c90 legacy=0x7f08b800d4e0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08d0c69640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8009fe0 0x7f08b8009870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08cb7fe640 1 -- 172.21.15.1:0/662566045 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3584069214 0 0) 0x7f08b800b0c0 con 0x7f08b800a3f0 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.573+0000 7f08cb7fe640 1 -- 172.21.15.1:0/662566045 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f08b4003520 con 0x7f08b800a3f0 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.574+0000 7f08d0c69640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.196:6789/0 conn(0x7f08b8006c90 legacy=0x7f08b800d4e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.574+0000 7f08d0c69640 1 -- 172.21.15.1:0/662566045 >> v1:172.21.15.1:6789/0 conn(0x7f08b800a3f0 legacy=0x7f08b800a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:04.571 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.574+0000 7f08d0c69640 1 -- 172.21.15.1:0/662566045 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08b8010e40 con 0x7f08b8009fe0 2024-05-09T05:23:04.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.575+0000 7f08d0c69640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8009fe0 0x7f08b8009870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08c0001970 tx=0x7f08c002a370 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ef42b6844cc5d4b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.575+0000 7f08cb7fe640 1 -- 172.21.15.1:0/662566045 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c0007270 con 0x7f08b8009fe0 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.575+0000 7f08cb7fe640 1 -- 172.21.15.1:0/662566045 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08c002e070 con 0x7f08b8009fe0 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.575+0000 7f08cb7fe640 1 -- 172.21.15.1:0/662566045 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08c00057c0 con 0x7f08b8009fe0 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.576+0000 7f08d2ecd640 1 -- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8009fe0 msgr2=0x7f08b8009870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.576+0000 7f08d2ecd640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8009fe0 0x7f08b8009870 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f08c0001970 tx=0x7f08c002a370 comp rx=0 tx=0).stop 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.576+0000 7f08d2ecd640 1 -- 172.21.15.1:0/662566045 shutdown_connections 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.576+0000 7f08d2ecd640 1 --2- 172.21.15.1:0/662566045 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8009fe0 0x7f08b8009870 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.576+0000 7f08d2ecd640 1 -- 172.21.15.1:0/662566045 >> 172.21.15.1:0/662566045 conn(0x7f08cc0731d0 msgr2=0x7f08cc0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.576+0000 7f08d2ecd640 1 -- 172.21.15.1:0/662566045 shutdown_connections 2024-05-09T05:23:04.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.576+0000 7f08d2ecd640 1 -- 172.21.15.1:0/662566045 wait complete. 2024-05-09T05:23:04.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.578+0000 7f08d2ecd640 1 Processor -- start 2024-05-09T05:23:04.574 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.578+0000 7f08d2ecd640 1 -- start start 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d2ecd640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8006c90 0x7f08cc1645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d2ecd640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08b8009fe0 0x7f08cc155090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d2ecd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08b800a3f0 0x7f08cc152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d2ecd640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc15b110 con 0x7f08b800a3f0 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d2ecd640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f08cc161810 con 0x7f08b8009fe0 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d2ecd640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f08cc15b290 con 0x7f08b8006c90 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08cbfff640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08b8009fe0 msgr2=0x7f08cc155090 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08cbfff640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08b8009fe0 0x7f08cc155090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d146a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08b800a3f0 0x7f08cc152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d0c69640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8006c90 0x7f08cc1645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d146a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08b800a3f0 0x7f08cc152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57780/0 (socket says 172.21.15.1:57780) 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.580+0000 7f08d146a640 1 -- 172.21.15.1:0/150505592 learned_addr learned my addr 172.21.15.1:0/150505592 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:04.578 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d146a640 1 -- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8006c90 msgr2=0x7f08cc1645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d146a640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8006c90 0x7f08cc1645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d146a640 1 -- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08b8009fe0 msgr2=0x7f08cc155090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d146a640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08b8009fe0 0x7f08cc155090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d146a640 1 -- 172.21.15.1:0/150505592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08cc1555d0 con 0x7f08b800a3f0 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d0c69640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8006c90 0x7f08cc1645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d146a640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08b800a3f0 0x7f08cc152210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08bc006fd0 tx=0x7f08bc013a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08c97fa640 1 -- 172.21.15.1:0/150505592 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc015690 con 0x7f08b800a3f0 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08c97fa640 1 -- 172.21.15.1:0/150505592 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f08bc01f070 con 0x7f08b800a3f0 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08cc0b2b40 con 0x7f08b800a3f0 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08c97fa640 1 -- 172.21.15.1:0/150505592 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08bc015a90 con 0x7f08b800a3f0 2024-05-09T05:23:04.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.581+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08cc0b2e20 con 0x7f08b800a3f0 2024-05-09T05:23:04.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.582+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08cc050870 con 0x7f08b800a3f0 2024-05-09T05:23:04.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.583+0000 7f08c97fa640 1 -- 172.21.15.1:0/150505592 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f08bc015c30 con 0x7f08b800a3f0 2024-05-09T05:23:04.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.584+0000 7f08c97fa640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08b8077fb0 0x7f08b807a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.584+0000 7f08c97fa640 1 -- 172.21.15.1:0/150505592 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f08bc0975e0 con 0x7f08b800a3f0 2024-05-09T05:23:04.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.584+0000 7f08d0c69640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08b8077fb0 0x7f08b807a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:04.581 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.585+0000 7f08d0c69640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08b8077fb0 0x7f08b807a4c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f08c0032000 tx=0x7f08c0007000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:04.585 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.589+0000 7f08c97fa640 1 -- 172.21.15.1:0/150505592 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f08bc060ec0 con 0x7f08b800a3f0 2024-05-09T05:23:04.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.864+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f08cc0cba10 con 0x7f08b800a3f0 2024-05-09T05:23:04.861 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.865+0000 7f08c97fa640 1 -- 172.21.15.1:0/150505592 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1555 (secure 0 0 0) 0x7f08bc0607c0 con 0x7f08b800a3f0 2024-05-09T05:23:04.861 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":2,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:23:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.866+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08b8077fb0 msgr2=0x7f08b807a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.866+0000 7f08d2ecd640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08b8077fb0 0x7f08b807a4c0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f08c0032000 tx=0x7f08c0007000 comp rx=0 tx=0).stop 2024-05-09T05:23:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.866+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08b800a3f0 msgr2=0x7f08cc152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.866+0000 7f08d2ecd640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08b800a3f0 0x7f08cc152210 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f08bc006fd0 tx=0x7f08bc013a20 comp rx=0 tx=0).stop 2024-05-09T05:23:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.867+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 shutdown_connections 2024-05-09T05:23:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.867+0000 7f08d2ecd640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f08b8077fb0 0x7f08b807a4c0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.867+0000 7f08d2ecd640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f08b8009fe0 0x7f08cc155090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.863 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.867+0000 7f08d2ecd640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f08b800a3f0 0x7f08cc152210 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.867+0000 7f08d2ecd640 1 --2- 172.21.15.1:0/150505592 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f08b8006c90 0x7f08cc1645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.867+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 >> 172.21.15.1:0/150505592 conn(0x7f08cc0731d0 msgr2=0x7f08cc06e300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:04.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.867+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 shutdown_connections 2024-05-09T05:23:04.864 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.867+0000 7f08d2ecd640 1 -- 172.21.15.1:0/150505592 wait complete. 2024-05-09T05:23:04.875 INFO:tasks.mon_thrash.ceph_manager:quorum is size 2 2024-05-09T05:23:04.875 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon a 2024-05-09T05:23:04.875 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:23:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.994+0000 7fc563cb1640 1 Processor -- start 2024-05-09T05:23:04.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.994+0000 7fc563cb1640 1 -- start start 2024-05-09T05:23:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.995+0000 7fc563cb1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0cf360 0x7fc55c0b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.995+0000 7fc563cb1640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc55c15c9a0 0x7fc55c15cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.995+0000 7fc563cb1640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc55c0ce5c0 con 0x7fc55c0b4570 2024-05-09T05:23:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.995+0000 7fc563cb1640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc55c15d4c0 con 0x7fc55c15c9a0 2024-05-09T05:23:04.992 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.995+0000 7fc563cb1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc55c15d7a0 con 0x7fc55c0cf360 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc56124c640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7fc55c0b4570 legacy=0x7fc55c156b50 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc56224e640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7fc55c15c9a0 msgr2=0x7fc55c15cdb0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc56224e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc55c15c9a0 0x7fc55c15cdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc561a4d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0cf360 0x7fc55c0b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc561a4d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0cf360 0x7fc55c0b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:41968/0 (socket says 172.21.15.1:41968) 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc561a4d640 1 -- 172.21.15.1:0/3068225276 learned_addr learned my addr 172.21.15.1:0/3068225276 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc561a4d640 1 -- 172.21.15.1:0/3068225276 >> v2:172.21.15.196:3300/0 conn(0x7fc55c15c9a0 msgr2=0x7fc55c15cdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc561a4d640 1 --2- 172.21.15.1:0/3068225276 >> v2:172.21.15.196:3300/0 conn(0x7fc55c15c9a0 0x7fc55c15cdb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc561a4d640 1 -- 172.21.15.1:0/3068225276 >> v1:172.21.15.196:6789/0 conn(0x7fc55c0b4570 legacy=0x7fc55c156b50 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:04.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.996+0000 7fc561a4d640 1 -- 172.21.15.1:0/3068225276 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55c0ce880 con 0x7fc55c0cf360 2024-05-09T05:23:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.997+0000 7fc561a4d640 1 --2- 172.21.15.1:0/3068225276 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0cf360 0x7fc55c0b4b50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc558009870 tx=0x7fc55802a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=159e158924b7ac6d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:04.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.997+0000 7fc560a4b640 1 -- 172.21.15.1:0/3068225276 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558037070 con 0x7fc55c0cf360 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.997+0000 7fc560a4b640 1 -- 172.21.15.1:0/3068225276 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc558032440 con 0x7fc55c0cf360 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.997+0000 7fc560a4b640 1 -- 172.21.15.1:0/3068225276 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558032810 con 0x7fc55c0cf360 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.998+0000 7fc563cb1640 1 -- 172.21.15.1:0/3068225276 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0cf360 msgr2=0x7fc55c0b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.998+0000 7fc563cb1640 1 --2- 172.21.15.1:0/3068225276 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0cf360 0x7fc55c0b4b50 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc558009870 tx=0x7fc55802a270 comp rx=0 tx=0).stop 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.998+0000 7fc563cb1640 1 -- 172.21.15.1:0/3068225276 shutdown_connections 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.998+0000 7fc563cb1640 1 --2- 172.21.15.1:0/3068225276 >> v2:172.21.15.196:3300/0 conn(0x7fc55c15c9a0 0x7fc55c15cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.998+0000 7fc563cb1640 1 --2- 172.21.15.1:0/3068225276 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0cf360 0x7fc55c0b4b50 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.998+0000 7fc563cb1640 1 -- 172.21.15.1:0/3068225276 >> 172.21.15.1:0/3068225276 conn(0x7fc55c06de10 msgr2=0x7fc55c06e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.998+0000 7fc563cb1640 1 -- 172.21.15.1:0/3068225276 shutdown_connections 2024-05-09T05:23:04.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:04.998+0000 7fc563cb1640 1 -- 172.21.15.1:0/3068225276 wait complete. 2024-05-09T05:23:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.000+0000 7fc563cb1640 1 Processor -- start 2024-05-09T05:23:04.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.000+0000 7fc563cb1640 1 -- start start 2024-05-09T05:23:04.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc563cb1640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0b4570 0x7fc55c1e6620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc563cb1640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc55c15c9a0 0x7fc55c1f30d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc563cb1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f5db0 0x7fc55c0ca0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc563cb1640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c0b5090 con 0x7fc55c1f5db0 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc563cb1640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c0cee40 con 0x7fc55c15c9a0 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc563cb1640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc55c0cd3f0 con 0x7fc55c0b4570 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc56124c640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc55c15c9a0 msgr2=0x7fc55c1f30d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc56124c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc55c15c9a0 0x7fc55c1f30d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc56224e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f5db0 0x7fc55c0ca0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc56224e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f5db0 0x7fc55c0ca0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57788/0 (socket says 172.21.15.1:57788) 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.001+0000 7fc56224e640 1 -- 172.21.15.1:0/1955028626 learned_addr learned my addr 172.21.15.1:0/1955028626 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc561a4d640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0b4570 0x7fc55c1e6620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:04.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc56224e640 1 -- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0b4570 msgr2=0x7fc55c1e6620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc56224e640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc55c0b4570 0x7fc55c1e6620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc56224e640 1 -- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc55c15c9a0 msgr2=0x7fc55c1f30d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc56224e640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc55c15c9a0 0x7fc55c1f30d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc56224e640 1 -- 172.21.15.1:0/1955028626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc55c1f3610 con 0x7fc55c1f5db0 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc56224e640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f5db0 0x7fc55c0ca0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc55400be90 tx=0x7fc55400bf80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc554015470 con 0x7fc55c1f5db0 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc554010070 con 0x7fc55c1f5db0 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc55c1f6660 con 0x7fc55c1f5db0 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc554015890 con 0x7fc55c1f5db0 2024-05-09T05:23:05.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.002+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc55c1f28d0 con 0x7fc55c1f5db0 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.003+0000 7fc563cb1640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f2aa0 0x7fc55c156090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.003+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c1f63d0 con 0x7fc55c1f2aa0 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.003+0000 7fc561a4d640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f2aa0 0x7fc55c156090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.003+0000 7fc561a4d640 1 -- 172.21.15.1:0/1955028626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc55c0d2890 con 0x7fc55c1f2aa0 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.004+0000 7fc561a4d640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f2aa0 0x7fc55c156090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0cf7d0 tx=0x7fc5580046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.004+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558043070 con 0x7fc55c1f2aa0 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.004+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc554005070 con 0x7fc55c1f5db0 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.005+0000 7fc5527fc640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5400761d0 0x7fc5400786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.005+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7fc55803c640 con 0x7fc55c1f2aa0 2024-05-09T05:23:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.005+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f2aa0 msgr2=0x7fc55c156090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.005+0000 7fc5527fc640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f2aa0 0x7fc55c156090 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0cf7d0 tx=0x7fc5580046c0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.005+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f2aa0 msgr2=0x7fc55c156090 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.005+0000 7fc5527fc640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f2aa0 0x7fc55c156090 secure :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fc55c0cf7d0 tx=0x7fc5580046c0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.005+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc554097c40 con 0x7fc55c1f5db0 2024-05-09T05:23:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.005+0000 7fc56124c640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5400761d0 0x7fc5400786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.003 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.006+0000 7fc56124c640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5400761d0 0x7fc5400786e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc54c006fd0 tx=0x7fc54c006ce0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.011+0000 7fc563cb1640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1565d0 0x7fc55c157310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.011+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc55c1569e0 con 0x7fc55c1565d0 2024-05-09T05:23:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc561a4d640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1565d0 0x7fc55c157310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.008 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc561a4d640 1 -- 172.21.15.1:0/1955028626 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7fc55c050870 con 0x7fc55c1565d0 2024-05-09T05:23:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc561a4d640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1565d0 0x7fc55c157310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc55802a2b0 tx=0x7fc558008030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc558037040 con 0x7fc55c1565d0 2024-05-09T05:23:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5053 (secure 0 0 0) 0x7fc558003a20 con 0x7fc55c1565d0 2024-05-09T05:23:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1565d0 msgr2=0x7fc55c157310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc5527fc640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1565d0 0x7fc55c157310 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fc55802a2b0 tx=0x7fc558008030 comp rx=0 tx=0).stop 2024-05-09T05:23:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc5527fc640 1 -- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1565d0 msgr2=0x7fc55c157310 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:05.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.012+0000 7fc5527fc640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1565d0 0x7fc55c157310 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.009 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 136, 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 173572, 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:23:05.010 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:23:05.011 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.012 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:23:05.013 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:05.014 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:23:05.015 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:05.016 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:23:05.017 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:05.018 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:23:05.019 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.020 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.021 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.022 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.023 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:23:05.023 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:05.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5400761d0 msgr2=0x7fc5400786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc5400761d0 0x7fc5400786e0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fc54c006fd0 tx=0x7fc54c006ce0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f5db0 msgr2=0x7fc55c0ca0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f5db0 0x7fc55c0ca0f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc55400be90 tx=0x7fc55400bf80 comp rx=0 tx=0).stop 2024-05-09T05:23:05.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc56224e640 1 -- 172.21.15.1:0/1955028626 reap_dead start 2024-05-09T05:23:05.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 shutdown_connections 2024-05-09T05:23:05.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 --2- 172.21.15.1:0/1955028626 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc55c1f5db0 0x7fc55c0ca0f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 >> 172.21.15.1:0/1955028626 conn(0x7fc55c06de10 msgr2=0x7fc55c0cb680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 shutdown_connections 2024-05-09T05:23:05.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.013+0000 7fc563cb1640 1 -- 172.21.15.1:0/1955028626 wait complete. 2024-05-09T05:23:05.025 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:23:05.025 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 2 for mon c 2024-05-09T05:23:05.025 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:23:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.145+0000 7f18fcf8f640 1 Processor -- start 2024-05-09T05:23:05.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.145+0000 7f18fcf8f640 1 -- start start 2024-05-09T05:23:05.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18fcf8f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f815ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18fcf8f640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f18f80cdaf0 0x7f18f815d350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18fcf8f640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18f8160160 con 0x7f18f8156c20 2024-05-09T05:23:05.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18fcf8f640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f18f8160ce0 con 0x7f18f80cdaf0 2024-05-09T05:23:05.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18fcf8f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f18f8160440 con 0x7f18f80cd610 2024-05-09T05:23:05.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18f659c640 1 -- >> v1:172.21.15.196:6789/0 conn(0x7f18f8156c20 legacy=0x7f18f80cea20 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v1:172.21.15.196:6789/0 2024-05-09T05:23:05.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18f6d9d640 1 -- >> v2:172.21.15.196:3300/0 conn(0x7f18f80cdaf0 msgr2=0x7f18f815d350 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:05.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18f6d9d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f18f80cdaf0 0x7f18f815d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18f5d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f815ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18f5d9b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f815ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:41996/0 (socket says 172.21.15.1:41996) 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.146+0000 7f18f5d9b640 1 -- 172.21.15.1:0/199823950 learned_addr learned my addr 172.21.15.1:0/199823950 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.147+0000 7f18f5d9b640 1 -- 172.21.15.1:0/199823950 >> v2:172.21.15.196:3300/0 conn(0x7f18f80cdaf0 msgr2=0x7f18f815d350 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.147+0000 7f18f5d9b640 1 --2- 172.21.15.1:0/199823950 >> v2:172.21.15.196:3300/0 conn(0x7f18f80cdaf0 0x7f18f815d350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.147+0000 7f18f5d9b640 1 -- 172.21.15.1:0/199823950 >> v1:172.21.15.196:6789/0 conn(0x7f18f8156c20 legacy=0x7f18f80cea20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.147+0000 7f18f5d9b640 1 -- 172.21.15.1:0/199823950 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f8160720 con 0x7f18f80cd610 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.148+0000 7f18f5d9b640 1 --2- 172.21.15.1:0/199823950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f815ce10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009870 tx=0x7f18e002a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=723c1aa1c7025640 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.148+0000 7f18f559a640 1 -- 172.21.15.1:0/199823950 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0037070 con 0x7f18f80cd610 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.148+0000 7f18f559a640 1 -- 172.21.15.1:0/199823950 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18e0032440 con 0x7f18f80cd610 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.148+0000 7f18f559a640 1 -- 172.21.15.1:0/199823950 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e0032810 con 0x7f18f80cd610 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.148+0000 7f18fcf8f640 1 -- 172.21.15.1:0/199823950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 msgr2=0x7f18f815ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.148+0000 7f18fcf8f640 1 --2- 172.21.15.1:0/199823950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f815ce10 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f18e0009870 tx=0x7f18e002a270 comp rx=0 tx=0).stop 2024-05-09T05:23:05.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.149+0000 7f18fcf8f640 1 -- 172.21.15.1:0/199823950 shutdown_connections 2024-05-09T05:23:05.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.149+0000 7f18fcf8f640 1 --2- 172.21.15.1:0/199823950 >> v2:172.21.15.196:3300/0 conn(0x7f18f80cdaf0 0x7f18f815d350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.149+0000 7f18fcf8f640 1 --2- 172.21.15.1:0/199823950 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f815ce10 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.149+0000 7f18fcf8f640 1 -- 172.21.15.1:0/199823950 >> 172.21.15.1:0/199823950 conn(0x7f18f8068830 msgr2=0x7f18f8068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:05.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.149+0000 7f18fcf8f640 1 -- 172.21.15.1:0/199823950 shutdown_connections 2024-05-09T05:23:05.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.149+0000 7f18fcf8f640 1 -- 172.21.15.1:0/199823950 wait complete. 2024-05-09T05:23:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.150+0000 7f18fcf8f640 1 Processor -- start 2024-05-09T05:23:05.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.151+0000 7f18fcf8f640 1 -- start start 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18fcf8f640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f80c7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18fcf8f640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18f80cdaf0 0x7f18f815eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18fcf8f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18f814c450 0x7f18f814c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18fcf8f640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f18f80ce2a0 con 0x7f18f814c450 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18fcf8f640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f18f815c8f0 con 0x7f18f80cdaf0 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18fcf8f640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f18f80cf130 con 0x7f18f80cd610 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f5d9b640 1 -- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18f80cdaf0 msgr2=0x7f18f815eb40 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.196:3300/0 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f5d9b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18f80cdaf0 0x7f18f815eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18f814c450 0x7f18f814c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18f814c450 0x7f18f814c860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:57808/0 (socket says 172.21.15.1:57808) 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f6d9d640 1 -- 172.21.15.1:0/1311733633 learned_addr learned my addr 172.21.15.1:0/1311733633 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:05.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f659c640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f80c7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f6d9d640 1 -- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 msgr2=0x7f18f80c7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f6d9d640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f80cd610 0x7f18f80c7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f6d9d640 1 -- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18f80cdaf0 msgr2=0x7f18f815eb40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f6d9d640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18f80cdaf0 0x7f18f815eb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.152+0000 7f18f6d9d640 1 -- 172.21.15.1:0/1311733633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18f80cb000 con 0x7f18f814c450 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.153+0000 7f18f6d9d640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18f814c450 0x7f18f814c860 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f18f000b440 tx=0x7f18f000b900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.153+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f0017070 con 0x7f18f814c450 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.153+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18f0003d40 con 0x7f18f814c450 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.153+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18f815c190 con 0x7f18f814c450 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.153+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18f00062f0 con 0x7f18f814c450 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.153+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18f8156020 con 0x7f18f814c450 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.153+0000 7f18fcf8f640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f5ff0 0x7f18f81f6400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.153+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f18f80cb4d0 con 0x7f18f81f5ff0 2024-05-09T05:23:05.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.154+0000 7f18f659c640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f5ff0 0x7f18f81f6400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.154+0000 7f18f659c640 1 -- 172.21.15.1:0/1311733633 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f18f8156390 con 0x7f18f81f5ff0 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.154+0000 7f18f659c640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f5ff0 0x7f18f81f6400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f40 tx=0x7f18e8007720 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.154+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e801a070 con 0x7f18f81f5ff0 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f18f001a070 con 0x7f18f814c450 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18eeffd640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18d8076200 0x7f18d8078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f18f0096640 con 0x7f18f814c450 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f18e80054e0 con 0x7f18f81f5ff0 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f5ff0 msgr2=0x7f18f81f6400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18f5d9b640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18d8076200 0x7f18d8078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18eeffd640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f5ff0 0x7f18f81f6400 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f40 tx=0x7f18e8007720 comp rx=0 tx=0).stop 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f5ff0 msgr2=0x7f18f81f6400 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:05.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.155+0000 7f18eeffd640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f5ff0 0x7f18f81f6400 secure :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f40 tx=0x7f18e8007720 comp rx=0 tx=0).stop 2024-05-09T05:23:05.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.156+0000 7f18f5d9b640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18d8076200 0x7f18d8078710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f18e0034000 tx=0x7f18e002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.162+0000 7f18fcf8f640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f6940 0x7f18f81549b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:05.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.162+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f18f815bf60 con 0x7f18f81f6940 2024-05-09T05:23:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.162+0000 7f18f659c640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f6940 0x7f18f81549b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.163+0000 7f18f659c640 1 -- 172.21.15.1:0/1311733633 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f18f80c7880 con 0x7f18f81f6940 2024-05-09T05:23:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.163+0000 7f18f659c640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f6940 0x7f18f81549b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f40 tx=0x7f18e8005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:05.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.163+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18e801dad0 con 0x7f18f81f6940 2024-05-09T05:23:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.163+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5050 (secure 0 0 0) 0x7f18e801dad0 con 0x7f18f81f6940 2024-05-09T05:23:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.163+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f6940 msgr2=0x7f18f81549b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.163+0000 7f18eeffd640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f6940 0x7f18f81549b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f40 tx=0x7f18e8005d50 comp rx=0 tx=0).stop 2024-05-09T05:23:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.163+0000 7f18eeffd640 1 -- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f6940 msgr2=0x7f18f81549b0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:05.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.163+0000 7f18eeffd640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18f81f6940 0x7f18f81549b0 secure :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f18e8009f40 tx=0x7f18e8005d50 comp rx=0 tx=0).stop 2024-05-09T05:23:05.160 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:23:05.160 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 136, 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 99988, 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 3, 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:05.161 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:05.162 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:23:05.163 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:05.164 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.165 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.166 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:05.167 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:05.168 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:23:05.169 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:23:05.170 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:23:05.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:05.172 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:05.173 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:05.173 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:05.173 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:05.173 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:05.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:05.173 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:23:05.173 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18d8076200 msgr2=0x7f18d8078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.174 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18fcf8f640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18d8076200 0x7f18d8078710 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f18e0034000 tx=0x7f18e002e000 comp rx=0 tx=0).stop 2024-05-09T05:23:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18f814c450 msgr2=0x7f18f814c860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18fcf8f640 1 --2- 172.21.15.1:0/1311733633 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18f814c450 0x7f18f814c860 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f18f000b440 tx=0x7f18f000b900 comp rx=0 tx=0).stop 2024-05-09T05:23:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18f6d9d640 1 -- 172.21.15.1:0/1311733633 reap_dead start 2024-05-09T05:23:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18f6d9d640 1 -- 172.21.15.1:0/1311733633 reap_dead start 2024-05-09T05:23:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 shutdown_connections 2024-05-09T05:23:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 >> 172.21.15.1:0/1311733633 conn(0x7f18f8068830 msgr2=0x7f18f81516a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 shutdown_connections 2024-05-09T05:23:05.175 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:05.164+0000 7f18fcf8f640 1 -- 172.21.15.1:0/1311733633 wait complete. 2024-05-09T05:23:05.175 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 2 2024-05-09T05:23:05.175 INFO:tasks.mon_thrash.mon_thrasher:waiting for 20.0 secs before reviving monitors 2024-05-09T05:23:06.614 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:23:06.618 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:23:25.176 INFO:tasks.mon_thrash.mon_thrasher:killing mon.b 2024-05-09T05:23:25.176 INFO:tasks.mon_thrash.mon_thrasher:reviving mon.b 2024-05-09T05:23:25.176 INFO:tasks.ceph.mon.b:Restarting daemon 2024-05-09T05:23:25.176 DEBUG:teuthology.orchestra.run.smithi196:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2024-05-09T05:23:25.179 INFO:tasks.ceph.mon.b:Started 2024-05-09T05:23:25.179 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:23:25.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:23:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.312+0000 7f26f44f0640 1 Processor -- start 2024-05-09T05:23:25.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.312+0000 7f26f44f0640 1 -- start start 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.313+0000 7f26f44f0640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 0x7f26ec156900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.313+0000 7f26f44f0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec0cc970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.313+0000 7f26f44f0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f26ec06de20 con 0x7f26ec15fdf0 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.313+0000 7f26f44f0640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f26ec15c9a0 con 0x7f26ec06efa0 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.313+0000 7f26f44f0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec0b51a0 con 0x7f26ec0cc560 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f228c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 0x7f26ec156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f1a8b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec0cc970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f2a8d640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f26ec15fdf0 0x7f26ec160200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:41046/0 (socket says 172.21.15.1:41046) 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f2a8d640 1 -- 172.21.15.1:0/922701802 learned_addr learned my addr 172.21.15.1:0/922701802 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f228c640 1 -- 172.21.15.1:0/922701802 >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 msgr2=0x7f26ec156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f228c640 1 -- 172.21.15.1:0/922701802 >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 msgr2=0x7f26ec156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f228c640 1 --2- 172.21.15.1:0/922701802 >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 0x7f26ec156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:23:25.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f228c640 1 --2- 172.21.15.1:0/922701802 >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 0x7f26ec156900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f1a8b640 1 -- 172.21.15.1:0/922701802 >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 msgr2=0x7f26ec156900 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f1a8b640 1 --2- 172.21.15.1:0/922701802 >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 0x7f26ec156900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f1a8b640 1 -- 172.21.15.1:0/922701802 >> v1:172.21.15.196:6789/0 conn(0x7f26ec15fdf0 legacy=0x7f26ec160200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.314+0000 7f26f1a8b640 1 -- 172.21.15.1:0/922701802 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec15d720 con 0x7f26ec0cc560 2024-05-09T05:23:25.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.315+0000 7f26f1a8b640 1 --2- 172.21.15.1:0/922701802 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec0cc970 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0097a0 tx=0x7f26dc02a210 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4cef4bbc30ee6ddb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.315+0000 7f26f128a640 1 -- 172.21.15.1:0/922701802 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc037070 con 0x7f26ec0cc560 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.315+0000 7f26f128a640 1 -- 172.21.15.1:0/922701802 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26dc032440 con 0x7f26ec0cc560 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.315+0000 7f26f128a640 1 -- 172.21.15.1:0/922701802 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc032810 con 0x7f26ec0cc560 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.316+0000 7f26f44f0640 1 -- 172.21.15.1:0/922701802 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 msgr2=0x7f26ec0cc970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.316+0000 7f26f44f0640 1 --2- 172.21.15.1:0/922701802 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec0cc970 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f26dc0097a0 tx=0x7f26dc02a210 comp rx=0 tx=0).stop 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.316+0000 7f26f44f0640 1 -- 172.21.15.1:0/922701802 shutdown_connections 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.316+0000 7f26f44f0640 1 --2- 172.21.15.1:0/922701802 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec0cc970 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.316+0000 7f26f44f0640 1 --2- 172.21.15.1:0/922701802 >> v2:172.21.15.196:3300/0 conn(0x7f26ec06efa0 0x7f26ec156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.316+0000 7f26f44f0640 1 -- 172.21.15.1:0/922701802 >> 172.21.15.1:0/922701802 conn(0x7f26ec068d90 msgr2=0x7f26ec0d1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.316+0000 7f26f44f0640 1 -- 172.21.15.1:0/922701802 shutdown_connections 2024-05-09T05:23:25.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.316+0000 7f26f44f0640 1 -- 172.21.15.1:0/922701802 wait complete. 2024-05-09T05:23:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.318+0000 7f26f44f0640 1 Processor -- start 2024-05-09T05:23:25.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.318+0000 7f26f44f0640 1 -- start start 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f44f0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 0x7f26ec1f3b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f44f0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec1f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f44f0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec15fdf0 0x7f26ec0b6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f44f0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec15cf20 con 0x7f26ec15fdf0 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f44f0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f26ec15cc40 con 0x7f26ec06efa0 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f44f0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f26ec15d200 con 0x7f26ec0cc560 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f2a8d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec15fdf0 0x7f26ec0b6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f2a8d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec15fdf0 0x7f26ec0b6060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47254/0 (socket says 172.21.15.1:47254) 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f1a8b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec1f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f2a8d640 1 -- 172.21.15.1:0/3365705096 learned_addr learned my addr 172.21.15.1:0/3365705096 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.319+0000 7f26f228c640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 0x7f26ec1f3b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f228c640 1 -- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 msgr2=0x7f26ec1f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 16 2024-05-09T05:23:25.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f228c640 1 -- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 msgr2=0x7f26ec1f3b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f228c640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 0x7f26ec1f3b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f228c640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 0x7f26ec1f3b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f1a8b640 1 -- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 msgr2=0x7f26ec1f3b40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f1a8b640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 0x7f26ec1f3b40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f1a8b640 1 -- 172.21.15.1:0/3365705096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec15fdf0 msgr2=0x7f26ec0b6060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f1a8b640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec15fdf0 0x7f26ec0b6060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f1a8b640 1 -- 172.21.15.1:0/3365705096 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26ec1545a0 con 0x7f26ec0cc560 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f2a8d640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec15fdf0 0x7f26ec0b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f1a8b640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec1f6ed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0b5370 tx=0x7f26dc0046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26e2ffd640 1 -- 172.21.15.1:0/3365705096 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc032e60 con 0x7f26ec0cc560 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26e2ffd640 1 -- 172.21.15.1:0/3365705096 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f26dc030770 con 0x7f26ec0cc560 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26ec0cfef0 con 0x7f26ec0cc560 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26e2ffd640 1 -- 172.21.15.1:0/3365705096 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26dc037040 con 0x7f26ec0cc560 2024-05-09T05:23:25.318 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.320+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({osdmap=0}) -- 0x7f26ec1f3340 con 0x7f26ec0cc560 2024-05-09T05:23:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.321+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f26ec050870 con 0x7f26ec0cc560 2024-05-09T05:23:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.322+0000 7f26e2ffd640 1 -- 172.21.15.1:0/3365705096 <== mon.2 v2:172.21.15.196:3301/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f26dc044020 con 0x7f26ec0cc560 2024-05-09T05:23:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.322+0000 7f26e2ffd640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26d0076300 0x7f26d0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:25.319 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.323+0000 7f26e2ffd640 1 -- 172.21.15.1:0/3365705096 <== mon.2 v2:172.21.15.196:3301/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f26dc0b8400 con 0x7f26ec0cc560 2024-05-09T05:23:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.323+0000 7f26f228c640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26d0076300 0x7f26d0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:25.320 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.323+0000 7f26f228c640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26d0076300 0x7f26d0078810 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f26e8006fd0 tx=0x7f26e8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:25.323 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.327+0000 7f26e2ffd640 1 -- 172.21.15.1:0/3365705096 <== mon.2 v2:172.21.15.196:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f26dc081c70 con 0x7f26ec0cc560 2024-05-09T05:23:25.588 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.592+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f26ec0c8cd0 con 0x7f26ec0cc560 2024-05-09T05:23:25.589 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.593+0000 7f26e2ffd640 1 -- 172.21.15.1:0/3365705096 <== mon.2 v2:172.21.15.196:3301/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1556 (secure 0 0 0) 0x7f26dc081570 con 0x7f26ec0cc560 2024-05-09T05:23:25.589 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":136,"quorum":[0,2],"quorum_names":["a","c"],"quorum_leader_name":"a","quorum_age":23,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:23:25.591 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26d0076300 msgr2=0x7f26d0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26d0076300 0x7f26d0078810 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f26e8006fd0 tx=0x7f26e8008040 comp rx=0 tx=0).stop 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 msgr2=0x7f26ec1f6ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec1f6ed0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f26ec0b5370 tx=0x7f26dc0046c0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 shutdown_connections 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f26d0076300 0x7f26d0078810 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f26ec15fdf0 0x7f26ec0b6060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f26ec0cc560 0x7f26ec1f6ed0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 --2- 172.21.15.1:0/3365705096 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f26ec06efa0 0x7f26ec1f3b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 >> 172.21.15.1:0/3365705096 conn(0x7f26ec068d90 msgr2=0x7f26ec15e290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 shutdown_connections 2024-05-09T05:23:25.592 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:25.595+0000 7f26f44f0640 1 -- 172.21.15.1:0/3365705096 wait complete. 2024-05-09T05:23:28.604 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:23:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.736+0000 7f76e2bce640 1 Processor -- start 2024-05-09T05:23:28.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.736+0000 7f76e2bce640 1 -- start start 2024-05-09T05:23:28.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.737+0000 7f76e2bce640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 0x7f76dc0ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:28.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.737+0000 7f76e2bce640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76dc15ca90 0x7f76dc15cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:28.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.737+0000 7f76e2bce640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f76dc06f6d0 con 0x7f76dc0d16b0 2024-05-09T05:23:28.734 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.737+0000 7f76e2bce640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f76dc0cd0e0 con 0x7f76dc15ca90 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.737+0000 7f76e2bce640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f76dc0cc4d0 con 0x7f76dc06f1b0 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76e116b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76dc15ca90 0x7f76dc15cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76e116b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f76dc15ca90 0x7f76dc15cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47262/0 (socket says 172.21.15.1:47262) 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76e116b640 1 -- 172.21.15.1:0/1787479845 learned_addr learned my addr 172.21.15.1:0/1787479845 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76dbfff640 1 --2- 172.21.15.1:0/1787479845 >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 0x7f76dc0ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76dbfff640 1 -- 172.21.15.1:0/1787479845 >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 msgr2=0x7f76dc0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76dbfff640 1 -- 172.21.15.1:0/1787479845 >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 msgr2=0x7f76dc0ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76dbfff640 1 --2- 172.21.15.1:0/1787479845 >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 0x7f76dc0ce740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76dbfff640 1 --2- 172.21.15.1:0/1787479845 >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 0x7f76dc0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76e116b640 1 -- 172.21.15.1:0/1787479845 >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 msgr2=0x7f76dc0ce740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76e116b640 1 --2- 172.21.15.1:0/1787479845 >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 0x7f76dc0ce740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76e116b640 1 -- 172.21.15.1:0/1787479845 >> v1:172.21.15.196:6789/0 conn(0x7f76dc0d16b0 legacy=0x7f76dc0d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:28.735 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.738+0000 7f76e116b640 1 -- 172.21.15.1:0/1787479845 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76dc0cc7b0 con 0x7f76dc15ca90 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.739+0000 7f76e116b640 1 --2- 172.21.15.1:0/1787479845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc15ca90 0x7f76dc15cea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f76dc06d0d0 tx=0x7f76d402a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a16af69a6545c3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.739+0000 7f76db7fe640 1 -- 172.21.15.1:0/1787479845 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d4036070 con 0x7f76dc15ca90 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.739+0000 7f76db7fe640 1 -- 172.21.15.1:0/1787479845 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76d4031600 con 0x7f76dc15ca90 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.739+0000 7f76db7fe640 1 -- 172.21.15.1:0/1787479845 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d40319f0 con 0x7f76dc15ca90 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.740+0000 7f76e2bce640 1 -- 172.21.15.1:0/1787479845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc15ca90 msgr2=0x7f76dc15cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.740+0000 7f76e2bce640 1 --2- 172.21.15.1:0/1787479845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc15ca90 0x7f76dc15cea0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f76dc06d0d0 tx=0x7f76d402a7c0 comp rx=0 tx=0).stop 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.740+0000 7f76e2bce640 1 -- 172.21.15.1:0/1787479845 shutdown_connections 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.740+0000 7f76e2bce640 1 --2- 172.21.15.1:0/1787479845 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc15ca90 0x7f76dc15cea0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.740+0000 7f76e2bce640 1 --2- 172.21.15.1:0/1787479845 >> v2:172.21.15.196:3300/0 conn(0x7f76dc06f1b0 0x7f76dc0ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.740+0000 7f76e2bce640 1 -- 172.21.15.1:0/1787479845 >> 172.21.15.1:0/1787479845 conn(0x7f76dc0b3660 msgr2=0x7f76dc0b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.740+0000 7f76e2bce640 1 -- 172.21.15.1:0/1787479845 shutdown_connections 2024-05-09T05:23:28.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.740+0000 7f76e2bce640 1 -- 172.21.15.1:0/1787479845 wait complete. 2024-05-09T05:23:28.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.742+0000 7f76e2bce640 1 Processor -- start 2024-05-09T05:23:28.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.742+0000 7f76e2bce640 1 -- start start 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e2bce640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc06f1b0 0x7f76dc0c5ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e2bce640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76dc0d16b0 0x7f76dc151620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e2bce640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc15ca90 0x7f76dc1e90d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e2bce640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc15d3e0 con 0x7f76dc06f1b0 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e2bce640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f76dc0ccd70 con 0x7f76dc15ca90 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e2bce640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f76dc0cec80 con 0x7f76dc0d16b0 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc06f1b0 0x7f76dc0c5ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc06f1b0 0x7f76dc0c5ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47278/0 (socket says 172.21.15.1:47278) 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 -- 172.21.15.1:0/950793522 learned_addr learned my addr 172.21.15.1:0/950793522 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e116b640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc15ca90 0x7f76dc1e90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:28.741 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76dbfff640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76dc0d16b0 0x7f76dc151620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 -- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76dc0d16b0 msgr2=0x7f76dc151620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76dc0d16b0 0x7f76dc151620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 -- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc15ca90 msgr2=0x7f76dc1e90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc15ca90 0x7f76dc1e90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 -- 172.21.15.1:0/950793522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76dc0c5300 con 0x7f76dc06f1b0 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.744+0000 7f76e096a640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc06f1b0 0x7f76dc0c5ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f76d000c3a0 tx=0x7f76d000c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.745+0000 7f76d97fa640 1 -- 172.21.15.1:0/950793522 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0017070 con 0x7f76dc06f1b0 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.745+0000 7f76d97fa640 1 -- 172.21.15.1:0/950793522 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f76d00052d0 con 0x7f76dc06f1b0 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.745+0000 7f76d97fa640 1 -- 172.21.15.1:0/950793522 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0003830 con 0x7f76dc06f1b0 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.745+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76dc0cba60 con 0x7f76dc06f1b0 2024-05-09T05:23:28.742 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.745+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76dc0cc020 con 0x7f76dc06f1b0 2024-05-09T05:23:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.746+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76dc050870 con 0x7f76dc06f1b0 2024-05-09T05:23:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.747+0000 7f76d97fa640 1 -- 172.21.15.1:0/950793522 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f76d001a070 con 0x7f76dc06f1b0 2024-05-09T05:23:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.747+0000 7f76d97fa640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76c0076300 0x7f76c0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.747+0000 7f76d97fa640 1 -- 172.21.15.1:0/950793522 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f76d00968d0 con 0x7f76dc06f1b0 2024-05-09T05:23:28.743 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.747+0000 7f76dbfff640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76c0076300 0x7f76c0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:28.744 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.748+0000 7f76dbfff640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76c0076300 0x7f76c0078810 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f76dc15afa0 tx=0x7f76c8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:28.748 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:28.752+0000 7f76d97fa640 1 -- 172.21.15.1:0/950793522 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f76d005fb60 con 0x7f76dc06f1b0 2024-05-09T05:23:29.014 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.018+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7f76dc0c7ea0 con 0x7f76dc06f1b0 2024-05-09T05:23:29.015 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.019+0000 7f76d97fa640 1 -- 172.21.15.1:0/950793522 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1561 (secure 0 0 0) 0x7f76d00681a0 con 0x7f76dc06f1b0 2024-05-09T05:23:29.015 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":0,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:23:29.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76c0076300 msgr2=0x7f76c0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76c0076300 0x7f76c0078810 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f76dc15afa0 tx=0x7f76c8008040 comp rx=0 tx=0).stop 2024-05-09T05:23:29.017 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc06f1b0 msgr2=0x7f76dc0c5ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc06f1b0 0x7f76dc0c5ac0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f76d000c3a0 tx=0x7f76d000c860 comp rx=0 tx=0).stop 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 shutdown_connections 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f76c0076300 0x7f76c0078810 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f76dc15ca90 0x7f76dc1e90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f76dc0d16b0 0x7f76dc151620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 --2- 172.21.15.1:0/950793522 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f76dc06f1b0 0x7f76dc0c5ac0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 >> 172.21.15.1:0/950793522 conn(0x7f76dc0b3660 msgr2=0x7f76dc153660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 shutdown_connections 2024-05-09T05:23:29.018 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.021+0000 7f76e2bce640 1 -- 172.21.15.1:0/950793522 wait complete. 2024-05-09T05:23:29.029 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:23:29.029 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon a 2024-05-09T05:23:29.029 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.a mon_status 2024-05-09T05:23:29.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.149+0000 7f1157c43640 1 Processor -- start 2024-05-09T05:23:29.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.149+0000 7f1157c43640 1 -- start start 2024-05-09T05:23:29.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.150+0000 7f1157c43640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11500c69a0 0x7f1150160a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.150+0000 7f1157c43640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f11500cb610 con 0x7f115006e1d0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.150+0000 7f1157c43640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1150156ef0 con 0x7f1150156ae0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.150+0000 7f1157c43640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f11500cb050 con 0x7f11500c69a0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.150+0000 7f11551de640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f115006e1d0 0x7f11500cc800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53390/0 (socket says 172.21.15.1:53390) 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.150+0000 7f11551de640 1 -- 172.21.15.1:0/4127233911 learned_addr learned my addr 172.21.15.1:0/4127233911 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11559df640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11500c69a0 0x7f1150160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4020360614 0 0) 0x7f11500cb610 con 0x7f115006e1d0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f113c003520 con 0x7f115006e1d0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 175302391 0 0) 0x7f113c003520 con 0x7f115006e1d0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11500c69a0 msgr2=0x7f1150160a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f11500c69a0 0x7f1150160a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 >> v1:172.21.15.196:6789/0 conn(0x7f1150156ae0 legacy=0x7f1150167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11500cb330 con 0x7f115006e1d0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f11400016f0 con 0x7f115006e1d0 2024-05-09T05:23:29.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 >> v1:172.21.15.1:6789/0 conn(0x7f115006e1d0 legacy=0x7f11500cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f113c0039e0 0x7f113c003df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c0043a0 0x7f113c0048c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f113c0050c0 0x7f113c0141d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11500cb050 con 0x7f113c0043a0 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f113c014790 con 0x7f113c0050c0 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f113c014990 con 0x7f113c0039e0 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11551de640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c0043a0 0x7f113c0048c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11559df640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f113c0039e0 0x7f113c003df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.151+0000 7f11561e0640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f113c0050c0 0x7f113c0141d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11551de640 1 -- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f113c0039e0 msgr2=0x7f113c003df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11551de640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f113c0039e0 0x7f113c003df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11551de640 1 -- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f113c0050c0 msgr2=0x7f113c0141d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11551de640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f113c0050c0 0x7f113c0141d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11551de640 1 -- 172.21.15.1:0/4127233911 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f113c003520 con 0x7f113c0043a0 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11551de640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c0043a0 0x7f113c0048c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1140007fd0 tx=0x7f113c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1c6578e822691f2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140007ca0 con 0x7f113c0043a0 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1140005ad0 con 0x7f113c0043a0 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.152+0000 7f11549dd640 1 -- 172.21.15.1:0/4127233911 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1140007070 con 0x7f113c0043a0 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.153+0000 7f1157c43640 1 -- 172.21.15.1:0/4127233911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c0043a0 msgr2=0x7f113c0048c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.153+0000 7f1157c43640 1 --2- 172.21.15.1:0/4127233911 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f113c0043a0 0x7f113c0048c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f1140007fd0 tx=0x7f113c005830 comp rx=0 tx=0).stop 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.153+0000 7f11561e0640 1 -- 172.21.15.1:0/4127233911 reap_dead start 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.153+0000 7f1157c43640 1 -- 172.21.15.1:0/4127233911 shutdown_connections 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.153+0000 7f1157c43640 1 -- 172.21.15.1:0/4127233911 >> 172.21.15.1:0/4127233911 conn(0x7f1150068830 msgr2=0x7f1150068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.151 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.153+0000 7f1157c43640 1 -- 172.21.15.1:0/4127233911 shutdown_connections 2024-05-09T05:23:29.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.153+0000 7f1157c43640 1 -- 172.21.15.1:0/4127233911 wait complete. 2024-05-09T05:23:29.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.155+0000 7f1157c43640 1 Processor -- start 2024-05-09T05:23:29.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.155+0000 7f1157c43640 1 -- start start 2024-05-09T05:23:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.156+0000 7f1157c43640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f113c0043a0 0x7f1150157a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.156+0000 7f1157c43640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11501ee430 0x7f11500c9100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.156+0000 7f1157c43640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150150410 0x7f1150150820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.156+0000 7f1157c43640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11501604f0 con 0x7f1150150410 2024-05-09T05:23:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.156+0000 7f1157c43640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1150160210 con 0x7f11501ee430 2024-05-09T05:23:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.156+0000 7f1157c43640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f11500ccf10 con 0x7f113c0043a0 2024-05-09T05:23:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.156+0000 7f11561e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150150410 0x7f1150150820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11561e0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150150410 0x7f1150150820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47298/0 (socket says 172.21.15.1:47298) 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11561e0640 1 -- 172.21.15.1:0/3926901755 learned_addr learned my addr 172.21.15.1:0/3926901755 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11551de640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11501ee430 0x7f11500c9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11559df640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f113c0043a0 0x7f1150157a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11561e0640 1 -- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f113c0043a0 msgr2=0x7f1150157a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11561e0640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f113c0043a0 0x7f1150157a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11561e0640 1 -- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11501ee430 msgr2=0x7f11500c9100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11561e0640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11501ee430 0x7f11500c9100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11561e0640 1 -- 172.21.15.1:0/3926901755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11500c70a0 con 0x7f1150150410 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11551de640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f11501ee430 0x7f11500c9100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11561e0640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150150410 0x7f1150150820 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1150072890 tx=0x7f114800ff70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1148003b80 con 0x7f1150150410 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1148018070 con 0x7f1150150410 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11500c7380 con 0x7f1150150410 2024-05-09T05:23:29.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114800d770 con 0x7f1150150410 2024-05-09T05:23:29.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.157+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11500c7940 con 0x7f1150150410 2024-05-09T05:23:29.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.158+0000 7f1157c43640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501ef8a0 0x7f11501f19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.158+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1150150d60 con 0x7f11501ef8a0 2024-05-09T05:23:29.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.158+0000 7f11559df640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501ef8a0 0x7f11501f19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.158+0000 7f11559df640 1 -- 172.21.15.1:0/3926901755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f11500c6e10 con 0x7f11501ef8a0 2024-05-09T05:23:29.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.159+0000 7f11559df640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501ef8a0 0x7f11501f19a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f114c014920 tx=0x7f114c014de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.159+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c004240 con 0x7f11501ef8a0 2024-05-09T05:23:29.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.159+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f114800d910 con 0x7f1150150410 2024-05-09T05:23:29.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.159+0000 7f11467fc640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1134076200 0x7f1134078710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.159+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f11480976d0 con 0x7f1150150410 2024-05-09T05:23:29.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.160+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f114c0043e0 con 0x7f11501ef8a0 2024-05-09T05:23:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.160+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501ef8a0 msgr2=0x7f11501f19a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.160+0000 7f11467fc640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501ef8a0 0x7f11501f19a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f114c014920 tx=0x7f114c014de0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.160+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501ef8a0 msgr2=0x7f11501f19a0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.160+0000 7f11467fc640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501ef8a0 0x7f11501f19a0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f114c014920 tx=0x7f114c014de0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.160+0000 7f11551de640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1134076200 0x7f1134078710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.158 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.160+0000 7f11551de640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1134076200 0x7f1134078710 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1140032000 tx=0x7f1140009030 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.166+0000 7f1157c43640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501f1ee0 0x7f11501f22f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.166+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f11501f2830 con 0x7f11501f1ee0 2024-05-09T05:23:29.162 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.166+0000 7f11559df640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501f1ee0 0x7f11501f22f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11559df640 1 -- 172.21.15.1:0/3926901755 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f115006a3c0 con 0x7f11501f1ee0 2024-05-09T05:23:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11559df640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501f1ee0 0x7f11501f22f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f114c014920 tx=0x7f114c004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f114c003a20 con 0x7f11501f1ee0 2024-05-09T05:23:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 <== mon.0 v2:172.21.15.1:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5064 (secure 0 0 0) 0x7f114c003a20 con 0x7f11501f1ee0 2024-05-09T05:23:29.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501f1ee0 msgr2=0x7f11501f22f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11467fc640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501f1ee0 0x7f11501f22f0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f114c014920 tx=0x7f114c004730 comp rx=0 tx=0).stop 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11467fc640 1 -- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501f1ee0 msgr2=0x7f11501f22f0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11467fc640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f11501f1ee0 0x7f11501f22f0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "leader", 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 197726, 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:23:29.164 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.165 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.166 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:23:29.167 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.168 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.169 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.170 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.171 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:23:29.172 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.173 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:29.174 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:23:29.175 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 5 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:29.176 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.177 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.177 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.177 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:23:29.177 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1134076200 msgr2=0x7f1134078710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f1157c43640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1134076200 0x7f1134078710 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f1140032000 tx=0x7f1140009030 comp rx=0 tx=0).stop 2024-05-09T05:23:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150150410 msgr2=0x7f1150150820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.178 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f1157c43640 1 --2- 172.21.15.1:0/3926901755 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1150150410 0x7f1150150820 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f1150072890 tx=0x7f114800ff70 comp rx=0 tx=0).stop 2024-05-09T05:23:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11561e0640 1 -- 172.21.15.1:0/3926901755 reap_dead start 2024-05-09T05:23:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f11561e0640 1 -- 172.21.15.1:0/3926901755 reap_dead start 2024-05-09T05:23:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 shutdown_connections 2024-05-09T05:23:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.167+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 >> 172.21.15.1:0/3926901755 conn(0x7f1150068830 msgr2=0x7f1150072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.168+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 shutdown_connections 2024-05-09T05:23:29.179 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.168+0000 7f1157c43640 1 -- 172.21.15.1:0/3926901755 wait complete. 2024-05-09T05:23:29.179 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:23:29.179 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon b 2024-05-09T05:23:29.179 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.b mon_status 2024-05-09T05:23:29.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.299+0000 7f259b1e8640 1 Processor -- start 2024-05-09T05:23:29.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.299+0000 7f259b1e8640 1 -- start start 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f259b1e8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c49e0 0x7f25941549c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f259b1e8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f25940c5190 0x7f259415dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f259b1e8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f25940b2860 con 0x7f25940cbde0 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f259b1e8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f25940c6970 con 0x7f25940c5190 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f259b1e8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f25940c57b0 con 0x7f25940c49e0 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2598f84640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f25940cbde0 0x7f25940cb060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53394/0 (socket says 172.21.15.1:53394) 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2598f84640 1 -- 172.21.15.1:0/260347356 learned_addr learned my addr 172.21.15.1:0/260347356 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2599785640 1 --2- 172.21.15.1:0/260347356 >> v2:172.21.15.1:3300/0 conn(0x7f25940c5190 0x7f259415dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2593fff640 1 --2- 172.21.15.1:0/260347356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c49e0 0x7f25941549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2599785640 1 -- 172.21.15.1:0/260347356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c49e0 msgr2=0x7f25941549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2599785640 1 --2- 172.21.15.1:0/260347356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c49e0 0x7f25941549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2599785640 1 -- 172.21.15.1:0/260347356 >> v1:172.21.15.1:6789/0 conn(0x7f25940cbde0 legacy=0x7f25940cb060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2599785640 1 -- 172.21.15.1:0/260347356 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25940c5d60 con 0x7f25940c5190 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.301+0000 7f2593fff640 1 --2- 172.21.15.1:0/260347356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c49e0 0x7f25941549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.302+0000 7f2599785640 1 --2- 172.21.15.1:0/260347356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 0x7f259415dc00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f25940c6af0 tx=0x7f257c02a180 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aacae1a8b0213096 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.302+0000 7f25937fe640 1 -- 172.21.15.1:0/260347356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c037070 con 0x7f25940c5190 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.302+0000 7f25937fe640 1 -- 172.21.15.1:0/260347356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f257c032440 con 0x7f25940c5190 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.302+0000 7f25937fe640 1 -- 172.21.15.1:0/260347356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f257c032810 con 0x7f25940c5190 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.303+0000 7f259b1e8640 1 -- 172.21.15.1:0/260347356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 msgr2=0x7f259415dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.303+0000 7f259b1e8640 1 --2- 172.21.15.1:0/260347356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 0x7f259415dc00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f25940c6af0 tx=0x7f257c02a180 comp rx=0 tx=0).stop 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.303+0000 7f259b1e8640 1 -- 172.21.15.1:0/260347356 shutdown_connections 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.303+0000 7f259b1e8640 1 --2- 172.21.15.1:0/260347356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940c49e0 0x7f25941549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.303+0000 7f259b1e8640 1 --2- 172.21.15.1:0/260347356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 0x7f259415dc00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.303+0000 7f259b1e8640 1 -- 172.21.15.1:0/260347356 >> 172.21.15.1:0/260347356 conn(0x7f2594068830 msgr2=0x7f2594068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.303+0000 7f259b1e8640 1 -- 172.21.15.1:0/260347356 shutdown_connections 2024-05-09T05:23:29.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.303+0000 7f259b1e8640 1 -- 172.21.15.1:0/260347356 wait complete. 2024-05-09T05:23:29.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.305+0000 7f259b1e8640 1 Processor -- start 2024-05-09T05:23:29.302 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.305+0000 7f259b1e8640 1 -- start start 2024-05-09T05:23:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f259b1e8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25940c49e0 0x7f25941f3090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f259b1e8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 0x7f25940b0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f259b1e8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940af720 0x7f25940cea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f259b1e8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f25940c6600 con 0x7f25940c5190 2024-05-09T05:23:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f259b1e8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f25940c6320 con 0x7f25940c49e0 2024-05-09T05:23:29.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f259b1e8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f25940cb770 con 0x7f25940af720 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f2593fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 0x7f25940b0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f2593fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 0x7f25940b0c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47332/0 (socket says 172.21.15.1:47332) 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.306+0000 7f2593fff640 1 -- 172.21.15.1:0/3064629069 learned_addr learned my addr 172.21.15.1:0/3064629069 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f2598f84640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25940c49e0 0x7f25941f3090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f2599785640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940af720 0x7f25940cea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f2593fff640 1 -- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940af720 msgr2=0x7f25940cea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f2593fff640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f25940af720 0x7f25940cea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f2593fff640 1 -- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25940c49e0 msgr2=0x7f25941f3090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f2593fff640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25940c49e0 0x7f25941f3090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f2593fff640 1 -- 172.21.15.1:0/3064629069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25940b11a0 con 0x7f25940c5190 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f2593fff640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 0x7f25940b0c60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2584007f20 tx=0x7f258400d560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258401a070 con 0x7f25940c5190 2024-05-09T05:23:29.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2584003b90 con 0x7f25940c5190 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f259406d800 con 0x7f25940c5190 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2584004030 con 0x7f25940c5190 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.307+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25941f25b0 con 0x7f25940c5190 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.308+0000 7f259b1e8640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25941f27f0 0x7f25941452c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.308+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f259406dcd0 con 0x7f25941f27f0 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.308+0000 7f2598f84640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25941f27f0 0x7f25941452c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.309+0000 7f2598f84640 1 -- 172.21.15.1:0/3064629069 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f25940b1480 con 0x7f25941f27f0 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.309+0000 7f2598f84640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25941f27f0 0x7f25941452c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f258c007fd0 tx=0x7f258c00f9b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.309+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f258401d070 con 0x7f25940c5190 2024-05-09T05:23:29.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f25917fa640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2574076300 0x7f2574078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.308 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f25840977c0 con 0x7f25940c5190 2024-05-09T05:23:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f2599785640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2574076300 0x7f2574078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c005ac0 con 0x7f25941f27f0 2024-05-09T05:23:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f258c018070 con 0x7f25941f27f0 2024-05-09T05:23:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25941f27f0 msgr2=0x7f25941452c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f25917fa640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25941f27f0 0x7f25941452c0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f258c007fd0 tx=0x7f258c00f9b0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25941f27f0 msgr2=0x7f25941452c0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f25917fa640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f25941f27f0 0x7f25941452c0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.309 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.310+0000 7f2599785640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2574076300 0x7f2574078810 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f257c034000 tx=0x7f257c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.316+0000 7f259b1e8640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594145800 0x7f25941f59c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.316+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2594145c10 con 0x7f2594145800 2024-05-09T05:23:29.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.317+0000 7f2598f84640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594145800 0x7f25941f59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.317+0000 7f2598f84640 1 -- 172.21.15.1:0/3064629069 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f2594050870 con 0x7f2594145800 2024-05-09T05:23:29.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.317+0000 7f2598f84640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594145800 0x7f25941f59c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f258c007fd0 tx=0x7f258c000ed0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.317+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f258c027410 con 0x7f2594145800 2024-05-09T05:23:29.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 <== mon.1 v2:172.21.15.196:3300/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+4550 (secure 0 0 0) 0x7f258c027410 con 0x7f2594145800 2024-05-09T05:23:29.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594145800 msgr2=0x7f25941f59c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f25917fa640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594145800 0x7f25941f59c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f258c007fd0 tx=0x7f258c000ed0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f25917fa640 1 -- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594145800 msgr2=0x7f25941f59c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:29.314 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f25917fa640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2594145800 0x7f25941f59c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 4032, 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:23:29.315 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 0, 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.316 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.317 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:23:29.318 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.319 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.320 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.321 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.322 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:23:29.323 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.324 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:29.325 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.326 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.326 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.326 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:23:29.326 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:29.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2574076300 msgr2=0x7f2574078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f259b1e8640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2574076300 0x7f2574078810 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7f257c034000 tx=0x7f257c02e000 comp rx=0 tx=0).stop 2024-05-09T05:23:29.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 msgr2=0x7f25940b0c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.327 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f259b1e8640 1 --2- 172.21.15.1:0/3064629069 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f25940c5190 0x7f25940b0c60 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2584007f20 tx=0x7f258400d560 comp rx=0 tx=0).stop 2024-05-09T05:23:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f2599785640 1 -- 172.21.15.1:0/3064629069 reap_dead start 2024-05-09T05:23:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.318+0000 7f2599785640 1 -- 172.21.15.1:0/3064629069 reap_dead start 2024-05-09T05:23:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.319+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 shutdown_connections 2024-05-09T05:23:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.319+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 >> 172.21.15.1:0/3064629069 conn(0x7f2594068830 msgr2=0x7f25940b3f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.319+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 shutdown_connections 2024-05-09T05:23:29.328 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.319+0000 7f259b1e8640 1 -- 172.21.15.1:0/3064629069 wait complete. 2024-05-09T05:23:29.328 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:23:29.328 INFO:tasks.mon_thrash.mon_thrasher:waiting for quorum size 3 for mon c 2024-05-09T05:23:29.328 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mon.c mon_status 2024-05-09T05:23:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.448+0000 7f8dbce15640 1 Processor -- start 2024-05-09T05:23:29.444 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.448+0000 7f8dbce15640 1 -- start start 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8dbce15640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f8db8072390 0x7f8db81569e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8dbce15640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8db806d800 0x7f8db80ca7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8dbce15640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f8db815ca00 con 0x7f8db80c6900 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8dbce15640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f8db8156f20 con 0x7f8db806d800 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8dbce15640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f8db815cca0 con 0x7f8db8072390 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8db6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8db806d800 0x7f8db80ca7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8db6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8db806d800 0x7f8db80ca7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47348/0 (socket says 172.21.15.1:47348) 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8db6d9d640 1 -- 172.21.15.1:0/1635815464 learned_addr learned my addr 172.21.15.1:0/1635815464 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:29.446 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.449+0000 7f8db5d9b640 1 --2- 172.21.15.1:0/1635815464 >> v2:172.21.15.196:3300/0 conn(0x7f8db8072390 0x7f8db81569e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.450+0000 7f8db6d9d640 1 -- 172.21.15.1:0/1635815464 >> v2:172.21.15.196:3300/0 conn(0x7f8db8072390 msgr2=0x7f8db81569e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.450+0000 7f8db6d9d640 1 --2- 172.21.15.1:0/1635815464 >> v2:172.21.15.196:3300/0 conn(0x7f8db8072390 0x7f8db81569e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.450+0000 7f8db6d9d640 1 -- 172.21.15.1:0/1635815464 >> v1:172.21.15.196:6789/0 conn(0x7f8db80c6900 legacy=0x7f8db8070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:29.447 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.450+0000 7f8db6d9d640 1 -- 172.21.15.1:0/1635815464 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db815cf80 con 0x7f8db806d800 2024-05-09T05:23:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.450+0000 7f8db6d9d640 1 --2- 172.21.15.1:0/1635815464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db80ca7a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8db80cceb0 tx=0x7f8da802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae2d7f34214120e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8db559a640 1 -- 172.21.15.1:0/1635815464 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da8036070 con 0x7f8db806d800 2024-05-09T05:23:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8db559a640 1 -- 172.21.15.1:0/1635815464 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8da8031600 con 0x7f8db806d800 2024-05-09T05:23:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8db559a640 1 -- 172.21.15.1:0/1635815464 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da80319d0 con 0x7f8db806d800 2024-05-09T05:23:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8dbce15640 1 -- 172.21.15.1:0/1635815464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 msgr2=0x7f8db80ca7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8dbce15640 1 --2- 172.21.15.1:0/1635815464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db80ca7a0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f8db80cceb0 tx=0x7f8da802a6a0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8dbce15640 1 -- 172.21.15.1:0/1635815464 shutdown_connections 2024-05-09T05:23:29.448 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8dbce15640 1 --2- 172.21.15.1:0/1635815464 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db80ca7a0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8dbce15640 1 --2- 172.21.15.1:0/1635815464 >> v2:172.21.15.196:3300/0 conn(0x7f8db8072390 0x7f8db81569e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8dbce15640 1 -- 172.21.15.1:0/1635815464 >> 172.21.15.1:0/1635815464 conn(0x7f8db80d0470 msgr2=0x7f8db80cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8dbce15640 1 -- 172.21.15.1:0/1635815464 shutdown_connections 2024-05-09T05:23:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.451+0000 7f8dbce15640 1 -- 172.21.15.1:0/1635815464 wait complete. 2024-05-09T05:23:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.453+0000 7f8dbce15640 1 Processor -- start 2024-05-09T05:23:29.449 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.453+0000 7f8dbce15640 1 -- start start 2024-05-09T05:23:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8dbce15640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db81f2be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8dbce15640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8db8072390 0x7f8db80cef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8dbce15640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c6900 0x7f8db8148620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8dbce15640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f8db8071610 con 0x7f8db806d800 2024-05-09T05:23:29.454 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8dbce15640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f8db815d540 con 0x7f8db8072390 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8dbce15640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8db8071790 con 0x7f8db80c6900 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db81f2be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db81f2be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47352/0 (socket says 172.21.15.1:47352) 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 -- 172.21.15.1:0/319681443 learned_addr learned my addr 172.21.15.1:0/319681443 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db5d9b640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8db8072390 0x7f8db80cef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db6d9d640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c6900 0x7f8db8148620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 -- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c6900 msgr2=0x7f8db8148620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c6900 0x7f8db8148620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 -- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8db8072390 msgr2=0x7f8db80cef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8db8072390 0x7f8db80cef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 -- 172.21.15.1:0/319681443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8db81f3120 con 0x7f8db806d800 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db5d9b640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f8db8072390 0x7f8db80cef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.455+0000 7f8db659c640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db81f2be0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8db000b420 tx=0x7f8db000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.455 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.456+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0011830 con 0x7f8db806d800 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.456+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8db0016070 con 0x7f8db806d800 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.456+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8db81f6130 con 0x7f8db806d800 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.456+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8db0011c10 con 0x7f8db806d800 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.456+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8db81f66f0 con 0x7f8db806d800 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.456+0000 7f8dbce15640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db81f32f0 0x7f8db81f68f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.456+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8db81f5ee0 con 0x7f8db81f32f0 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.457+0000 7f8db5d9b640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db81f32f0 0x7f8db81f68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.457+0000 7f8db5d9b640 1 -- 172.21.15.1:0/319681443 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8db8050870 con 0x7f8db81f32f0 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.457+0000 7f8db5d9b640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db81f32f0 0x7f8db81f68f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8db8160f00 tx=0x7f8da0007f30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.457+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da0018020 con 0x7f8db81f32f0 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.457+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8db001b460 con 0x7f8db806d800 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.458+0000 7f8daeffd640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d980761d0 0x7f8d980786e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.458+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f8db0096fc0 con 0x7f8db806d800 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.458+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 1: 0 ) ==== 8+0+9324 (secure 0 0 0) 0x7f8da0004300 con 0x7f8db81f32f0 2024-05-09T05:23:29.456 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.458+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db81f32f0 msgr2=0x7f8db81f68f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.458+0000 7f8daeffd640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db81f32f0 0x7f8db81f68f0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8db8160f00 tx=0x7f8da0007f30 comp rx=0 tx=0).stop 2024-05-09T05:23:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.458+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db81f32f0 msgr2=0x7f8db81f68f0 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.458+0000 7f8daeffd640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db81f32f0 0x7f8db81f68f0 secure :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f8db8160f00 tx=0x7f8da0007f30 comp rx=0 tx=0).stop 2024-05-09T05:23:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.458+0000 7f8db6d9d640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d980761d0 0x7f8d980786e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.457 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.459+0000 7f8db6d9d640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d980761d0 0x7f8d980786e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8da8033000 tx=0x7f8da8009000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.465+0000 7f8dbce15640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c9510 0x7f8db8148b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.465+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f8db815c710 con 0x7f8db80c9510 2024-05-09T05:23:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.465+0000 7f8db5d9b640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c9510 0x7f8db8148b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.465+0000 7f8db5d9b640 1 -- 172.21.15.1:0/319681443 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- command(tid 2: {"prefix": "mon_status"}) -- 0x7f8db80cbbf0 con 0x7f8db80c9510 2024-05-09T05:23:29.461 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.465+0000 7f8db5d9b640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c9510 0x7f8db8148b60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8db8160f00 tx=0x7f8da00046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8da001f0d0 con 0x7f8db80c9510 2024-05-09T05:23:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 <== mon.2 v2:172.21.15.196:3301/0 2 ==== command_reply(tid 2: 0 ) ==== 8+0+5062 (secure 0 0 0) 0x7f8da001f0d0 con 0x7f8db80c9510 2024-05-09T05:23:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c9510 msgr2=0x7f8db8148b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.462 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8daeffd640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c9510 0x7f8db8148b60 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8db8160f00 tx=0x7f8da00046c0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8daeffd640 1 -- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c9510 msgr2=0x7f8db8148b60 secure :-1 s=STATE_CLOSED l=1).mark_down 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8daeffd640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f8db80c9510 0x7f8db8148b60 secure :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8db8160f00 tx=0x7f8da00046c0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stdout: "state": "peon", 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stdout: "election_epoch": 138, 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stdout: "uptime": 124291, 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum": [ 2024-05-09T05:23:29.463 INFO:teuthology.orchestra.run.smithi001.stdout: 0, 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: 1, 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: 2 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_age": 1, 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "required_con": "2450521708007866372", 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "required_mon": [ 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.464 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_con": "4540701547738038271", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "quorum_mon": [ 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.465 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "outside_quorum": [], 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "extra_probe_peers": [], 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "sync_provider": [], 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "monmap": { 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "epoch": 1, 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "fsid": "aa0bbdea-07dc-4c3a-9697-b556ae629556", 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "modified": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "created": "2024-05-09T05:05:03.309340Z", 2024-05-09T05:23:29.466 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release": 19, 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "min_mon_release_name": "squid", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "election_strategy": 1, 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "disallowed_leaders: ": "", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false, 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "tiebreaker_mon": "", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "removed_ranks: ": "", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "features": { 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "persistent": [ 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "kraken", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "luminous", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "mimic", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "osdmap-prune", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "nautilus", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "octopus", 2024-05-09T05:23:29.467 INFO:teuthology.orchestra.run.smithi001.stdout: "pacific", 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "elector-pinging", 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "quincy", 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "reef", 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "squid" 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "optional": [] 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "mons": [ 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 0, 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "a", 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.468 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:3300", 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789", 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.1:6789/0", 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.1:6789/0", 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.469 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 1, 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "b", 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3300", 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.470 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789", 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6789/0", 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6789/0", 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "rank": 2, 2024-05-09T05:23:29.471 INFO:teuthology.orchestra.run.smithi001.stdout: "name": "c", 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addrs": { 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addrvec": [ 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v2", 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:3301", 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "type": "v1", 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790", 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: "nonce": 0 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.472 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "addr": "172.21.15.196:6790/0", 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "public_addr": "172.21.15.196:6790/0", 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "priority": 0, 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "weight": 0, 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "crush_location": "{}" 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "feature_map": { 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "mon": [ 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.473 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "mds": [ 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "osd": [ 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.474 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 4 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "client": [ 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 3 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: ], 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "mgr": [ 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: { 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "features": "0x3f03cffffffdffff", 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "release": "luminous", 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: "num": 1 2024-05-09T05:23:29.475 INFO:teuthology.orchestra.run.smithi001.stdout: } 2024-05-09T05:23:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: ] 2024-05-09T05:23:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: }, 2024-05-09T05:23:29.476 INFO:teuthology.orchestra.run.smithi001.stdout: "stretch_mode": false 2024-05-09T05:23:29.476 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d980761d0 msgr2=0x7f8d980786e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f8d980761d0 0x7f8d980786e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7f8da8033000 tx=0x7f8da8009000 comp rx=0 tx=0).stop 2024-05-09T05:23:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 msgr2=0x7f8db81f2be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db81f2be0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f8db000b420 tx=0x7f8db000b8e0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8db6d9d640 1 -- 172.21.15.1:0/319681443 reap_dead start 2024-05-09T05:23:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 shutdown_connections 2024-05-09T05:23:29.477 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 --2- 172.21.15.1:0/319681443 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8db806d800 0x7f8db81f2be0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 >> 172.21.15.1:0/319681443 conn(0x7f8db80d0470 msgr2=0x7f8db8075ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 shutdown_connections 2024-05-09T05:23:29.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.466+0000 7f8dbce15640 1 -- 172.21.15.1:0/319681443 wait complete. 2024-05-09T05:23:29.478 INFO:tasks.mon_thrash.mon_thrasher:final quorum is size 3 2024-05-09T05:23:29.478 INFO:tasks.mon_thrash.mon_thrasher:triggering scrub 2024-05-09T05:23:29.478 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mon scrub 2024-05-09T05:23:29.593 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.597+0000 7fb70ca53640 1 Processor -- start 2024-05-09T05:23:29.594 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.597+0000 7fb70ca53640 1 -- start start 2024-05-09T05:23:29.597 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70ca53640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7080cebc0 0x7fb7080d1530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70ca53640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fb708156b30 0x7fb70815ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70ca53640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb70815ff70 con 0x7fb70806f440 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70ca53640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fb7080cd180 con 0x7fb708156b30 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70ca53640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb708160250 con 0x7fb7080cebc0 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70659c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb70806f440 0x7fb7080ce4b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:53404/0 (socket says 172.21.15.1:53404) 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70659c640 1 -- 172.21.15.1:0/3226394776 learned_addr learned my addr 172.21.15.1:0/3226394776 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb706d9d640 1 --2- 172.21.15.1:0/3226394776 >> v2:172.21.15.196:3300/0 conn(0x7fb708156b30 0x7fb70815ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb705d9b640 1 --2- 172.21.15.1:0/3226394776 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7080cebc0 0x7fb7080d1530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70559a640 1 -- 172.21.15.1:0/3226394776 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4067960088 0 0) 0x7fb70815ff70 con 0x7fb70806f440 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb70559a640 1 -- 172.21.15.1:0/3226394776 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7fb6f0003520 con 0x7fb70806f440 2024-05-09T05:23:29.598 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb706d9d640 1 -- 172.21.15.1:0/3226394776 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7080cebc0 msgr2=0x7fb7080d1530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb706d9d640 1 --2- 172.21.15.1:0/3226394776 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7080cebc0 0x7fb7080d1530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb706d9d640 1 -- 172.21.15.1:0/3226394776 >> v1:172.21.15.1:6789/0 conn(0x7fb70806f440 legacy=0x7fb7080ce4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.599+0000 7fb706d9d640 1 -- 172.21.15.1:0/3226394776 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb708160530 con 0x7fb708156b30 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.600+0000 7fb706d9d640 1 --2- 172.21.15.1:0/3226394776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb708156b30 0x7fb70815ca30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb700009d90 tx=0x7fb70002a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da3b9482923b2590 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.600+0000 7fb70559a640 1 -- 172.21.15.1:0/3226394776 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700007740 con 0x7fb708156b30 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.600+0000 7fb70559a640 1 -- 172.21.15.1:0/3226394776 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb7000078e0 con 0x7fb708156b30 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.600+0000 7fb70559a640 1 -- 172.21.15.1:0/3226394776 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb700007cb0 con 0x7fb708156b30 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.601+0000 7fb70ca53640 1 -- 172.21.15.1:0/3226394776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb708156b30 msgr2=0x7fb70815ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.601+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3226394776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb708156b30 0x7fb70815ca30 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb700009d90 tx=0x7fb70002a660 comp rx=0 tx=0).stop 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.601+0000 7fb70ca53640 1 -- 172.21.15.1:0/3226394776 shutdown_connections 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.601+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3226394776 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb708156b30 0x7fb70815ca30 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.601+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3226394776 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb7080cebc0 0x7fb7080d1530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.599 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.601+0000 7fb70ca53640 1 -- 172.21.15.1:0/3226394776 >> 172.21.15.1:0/3226394776 conn(0x7fb7080b3670 msgr2=0x7fb7080b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.601+0000 7fb70ca53640 1 -- 172.21.15.1:0/3226394776 shutdown_connections 2024-05-09T05:23:29.600 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.601+0000 7fb70ca53640 1 -- 172.21.15.1:0/3226394776 wait complete. 2024-05-09T05:23:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.603+0000 7fb70ca53640 1 Processor -- start 2024-05-09T05:23:29.602 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.603+0000 7fb70ca53640 1 -- start start 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb70ca53640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb70806f440 0x7fb7081f3e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb70ca53640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708156b30 0x7fb70814f250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb70ca53640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb708154f50 0x7fb7081f58e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb70ca53640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb7080cf940 con 0x7fb708156b30 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb70ca53640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb708156f40 con 0x7fb70806f440 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb70ca53640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb7080cfac0 con 0x7fb708154f50 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb705d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708156b30 0x7fb70814f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb705d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708156b30 0x7fb70814f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:47362/0 (socket says 172.21.15.1:47362) 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.604+0000 7fb705d9b640 1 -- 172.21.15.1:0/3729409050 learned_addr learned my addr 172.21.15.1:0/3729409050 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb70659c640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb70806f440 0x7fb7081f3e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb706d9d640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb708154f50 0x7fb7081f58e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb705d9b640 1 -- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb708154f50 msgr2=0x7fb7081f58e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.603 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb705d9b640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb708154f50 0x7fb7081f58e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb705d9b640 1 -- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb70806f440 msgr2=0x7fb7081f3e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb705d9b640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb70806f440 0x7fb7081f3e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb705d9b640 1 -- 172.21.15.1:0/3729409050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb7081f4370 con 0x7fb708156b30 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb70659c640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb70806f440 0x7fb7081f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb705d9b640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708156b30 0x7fb70814f250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00e870 tx=0x7fb6fc00ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb6f6ffd640 1 -- 172.21.15.1:0/3729409050 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc017070 con 0x7fb708156b30 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb6f6ffd640 1 -- 172.21.15.1:0/3729409050 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb6fc00c940 con 0x7fb708156b30 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb6f6ffd640 1 -- 172.21.15.1:0/3729409050 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6fc00cd10 con 0x7fb708156b30 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb7081f66f0 con 0x7fb708156b30 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.605+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7081e6db0 con 0x7fb708156b30 2024-05-09T05:23:29.604 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.606+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb708050870 con 0x7fb708156b30 2024-05-09T05:23:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.607+0000 7fb6f6ffd640 1 -- 172.21.15.1:0/3729409050 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb6fc01a070 con 0x7fb708156b30 2024-05-09T05:23:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.607+0000 7fb6f6ffd640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6e8076300 0x7fb6e8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.608+0000 7fb6f6ffd640 1 -- 172.21.15.1:0/3729409050 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb6fc097910 con 0x7fb708156b30 2024-05-09T05:23:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.608+0000 7fb70659c640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6e8076300 0x7fb6e8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:29.605 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.608+0000 7fb70659c640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6e8076300 0x7fb6e8078810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb6f80010e0 tx=0x7fb6f8009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:29.608 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.612+0000 7fb6f6ffd640 1 -- 172.21.15.1:0/3729409050 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb6fc061340 con 0x7fb708156b30 2024-05-09T05:23:29.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.879+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "mon scrub"} v 0) -- 0x7fb7080b7ac0 con 0x7fb708156b30 2024-05-09T05:23:29.875 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.880+0000 7fb6f6ffd640 1 -- 172.21.15.1:0/3729409050 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "mon scrub"}]=0 v0) ==== 57+0+0 (secure 0 0 0) 0x7fb6fc069330 con 0x7fb708156b30 2024-05-09T05:23:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6e8076300 msgr2=0x7fb6e8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6e8076300 0x7fb6e8078810 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fb6f80010e0 tx=0x7fb6f8009040 comp rx=0 tx=0).stop 2024-05-09T05:23:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708156b30 msgr2=0x7fb70814f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708156b30 0x7fb70814f250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fb6fc00e870 tx=0x7fb6fc00ed30 comp rx=0 tx=0).stop 2024-05-09T05:23:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 shutdown_connections 2024-05-09T05:23:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb6e8076300 0x7fb6e8078810 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb708154f50 0x7fb7081f58e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.878 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb708156b30 0x7fb70814f250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 --2- 172.21.15.1:0/3729409050 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb70806f440 0x7fb7081f3e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 >> 172.21.15.1:0/3729409050 conn(0x7fb7080b3670 msgr2=0x7fb7080d0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 shutdown_connections 2024-05-09T05:23:29.879 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:29.882+0000 7fb70ca53640 1 -- 172.21.15.1:0/3729409050 wait complete. 2024-05-09T05:23:29.890 INFO:tasks.mon_thrash.mon_thrasher:waiting for 10.0 secs before continuing thrashing 2024-05-09T05:23:36.786 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:23:36.790 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:23:39.891 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:23:40.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.024+0000 7fddad95e640 1 Processor -- start 2024-05-09T05:23:40.021 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.025+0000 7fddad95e640 1 -- start start 2024-05-09T05:23:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fddad95e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 0x7fdda8156b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fddad95e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdda80d1530 0x7fdda815ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.022 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fddad95e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fdda815ff70 con 0x7fdda80ce4b0 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fddad95e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fdda8157070 con 0x7fdda80d1530 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fddad95e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdda80d1940 con 0x7fdda806f440 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fdda77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdda80d1530 0x7fdda815ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fdda77fe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fdda80d1530 0x7fdda815ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60198/0 (socket says 172.21.15.1:60198) 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fdda77fe640 1 -- 172.21.15.1:0/3163607196 learned_addr learned my addr 172.21.15.1:0/3163607196 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fdda6ffd640 1 --2- 172.21.15.1:0/3163607196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 0x7fdda8156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fdda77fe640 1 -- 172.21.15.1:0/3163607196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 msgr2=0x7fdda8156b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fdda77fe640 1 --2- 172.21.15.1:0/3163607196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 0x7fdda8156b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fdda77fe640 1 -- 172.21.15.1:0/3163607196 >> v1:172.21.15.1:6789/0 conn(0x7fdda80ce4b0 legacy=0x7fdda80ce8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.023 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.026+0000 7fdda77fe640 1 -- 172.21.15.1:0/3163607196 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda8160250 con 0x7fdda80d1530 2024-05-09T05:23:40.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.027+0000 7fdda77fe640 1 --2- 172.21.15.1:0/3163607196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80d1530 0x7fdda815ca30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009e50 tx=0x7fdda002a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59c8d76cba4c6f34 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.024 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.027+0000 7fdda5ffb640 1 -- 172.21.15.1:0/3163607196 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda0036070 con 0x7fdda80d1530 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.027+0000 7fdda5ffb640 1 -- 172.21.15.1:0/3163607196 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdda0031600 con 0x7fdda80d1530 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.027+0000 7fdda5ffb640 1 -- 172.21.15.1:0/3163607196 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdda00319d0 con 0x7fdda80d1530 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.028+0000 7fddad95e640 1 -- 172.21.15.1:0/3163607196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80d1530 msgr2=0x7fdda815ca30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.028+0000 7fddad95e640 1 --2- 172.21.15.1:0/3163607196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80d1530 0x7fdda815ca30 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fdda0009e50 tx=0x7fdda002a710 comp rx=0 tx=0).stop 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.028+0000 7fddad95e640 1 -- 172.21.15.1:0/3163607196 shutdown_connections 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.028+0000 7fddad95e640 1 --2- 172.21.15.1:0/3163607196 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80d1530 0x7fdda815ca30 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.028+0000 7fddad95e640 1 --2- 172.21.15.1:0/3163607196 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 0x7fdda8156b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.028+0000 7fddad95e640 1 -- 172.21.15.1:0/3163607196 >> 172.21.15.1:0/3163607196 conn(0x7fdda80b3670 msgr2=0x7fdda80b3a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.028+0000 7fddad95e640 1 -- 172.21.15.1:0/3163607196 shutdown_connections 2024-05-09T05:23:40.025 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.028+0000 7fddad95e640 1 -- 172.21.15.1:0/3163607196 wait complete. 2024-05-09T05:23:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.030+0000 7fddad95e640 1 Processor -- start 2024-05-09T05:23:40.026 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.030+0000 7fddad95e640 1 -- start start 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.031+0000 7fddad95e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 0x7fdda81578b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.031+0000 7fddad95e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ce4b0 0x7fdda8158e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.031+0000 7fddad95e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdda80d1530 0x7fdda8155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.031+0000 7fddad95e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fdda8160810 con 0x7fdda80ce4b0 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.031+0000 7fddad95e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fdda80cd180 con 0x7fdda80d1530 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.031+0000 7fddad95e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fdda80cf940 con 0x7fdda806f440 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ce4b0 0x7fdda8158e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ce4b0 0x7fdda8158e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60210/0 (socket says 172.21.15.1:60210) 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 -- 172.21.15.1:0/3882738628 learned_addr learned my addr 172.21.15.1:0/3882738628 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda77fe640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdda80d1530 0x7fdda8155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda6ffd640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 0x7fdda81578b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.029 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 -- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 msgr2=0x7fdda81578b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 0x7fdda81578b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 -- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdda80d1530 msgr2=0x7fdda8155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdda80d1530 0x7fdda8155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 -- 172.21.15.1:0/3882738628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdda8157df0 con 0x7fdda80ce4b0 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdda67fc640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ce4b0 0x7fdda8158e40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00c390 tx=0x7fdd9c00c850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdd8ffff640 1 -- 172.21.15.1:0/3882738628 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c017070 con 0x7fdda80ce4b0 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdd8ffff640 1 -- 172.21.15.1:0/3882738628 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fdd9c0035f0 con 0x7fdda80ce4b0 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdda8159380 con 0x7fdda80ce4b0 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fdd8ffff640 1 -- 172.21.15.1:0/3882738628 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd9c0039c0 con 0x7fdda80ce4b0 2024-05-09T05:23:40.030 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.032+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdda8159b50 con 0x7fdda80ce4b0 2024-05-09T05:23:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.033+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdda81571f0 con 0x7fdda80ce4b0 2024-05-09T05:23:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.034+0000 7fdd8ffff640 1 -- 172.21.15.1:0/3882738628 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fdd9c01a070 con 0x7fdda80ce4b0 2024-05-09T05:23:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.035+0000 7fdd8ffff640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd94076300 0x7fdd94078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.035+0000 7fdd8ffff640 1 -- 172.21.15.1:0/3882738628 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fdd9c0968d0 con 0x7fdda80ce4b0 2024-05-09T05:23:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.035+0000 7fdda6ffd640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd94076300 0x7fdd94078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.031 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.036+0000 7fdda6ffd640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd94076300 0x7fdd94078810 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdd98005f80 tx=0x7fdd98008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.035 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.040+0000 7fdd8ffff640 1 -- 172.21.15.1:0/3882738628 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fdd9c05fb60 con 0x7fdda80ce4b0 2024-05-09T05:23:40.265 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.269+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdda80b7ac0 con 0x7fdda80ce4b0 2024-05-09T05:23:40.266 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.270+0000 7fdd8ffff640 1 -- 172.21.15.1:0/3882738628 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6915 (secure 0 0 0) 0x7fdd9c063e60 con 0x7fdda80ce4b0 2024-05-09T05:23:40.266 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:23:40.266 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:33.214091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34160,"mds_1":34157,"mds_2":4311,"mds_3":4287,"mds_4":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4311":{"gid":4311,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[4273,34160,4287,4311,34157]},"id":1}]} 2024-05-09T05:23:40.266 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd94076300 msgr2=0x7fdd94078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd94076300 0x7fdd94078810 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fdd98005f80 tx=0x7fdd98008040 comp rx=0 tx=0).stop 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ce4b0 msgr2=0x7fdda8158e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ce4b0 0x7fdda8158e40 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdd9c00c390 tx=0x7fdd9c00c850 comp rx=0 tx=0).stop 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 shutdown_connections 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fdd94076300 0x7fdd94078810 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fdda80d1530 0x7fdda8155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fdda80ce4b0 0x7fdda8158e40 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.268 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 --2- 172.21.15.1:0/3882738628 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fdda806f440 0x7fdda81578b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 >> 172.21.15.1:0/3882738628 conn(0x7fdda80b3670 msgr2=0x7fdda806d800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:40.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 shutdown_connections 2024-05-09T05:23:40.269 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.272+0000 7fddad95e640 1 -- 172.21.15.1:0/3882738628 wait complete. 2024-05-09T05:23:40.280 INFO:tasks.mon_thrash.ceph_manager:waiting for quorum size 3 2024-05-09T05:23:40.280 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph quorum_status 2024-05-09T05:23:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.398+0000 7fad46489640 1 Processor -- start 2024-05-09T05:23:40.394 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.398+0000 7fad46489640 1 -- start start 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.399+0000 7fad46489640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad40070bd0 0x7fad40070fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.399+0000 7fad46489640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad400c6bc0 con 0x7fad40156c90 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.399+0000 7fad46489640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fad4015c8e0 con 0x7fad400c64b0 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.400+0000 7fad46489640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fad401570a0 con 0x7fad40070bd0 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.400+0000 7fad44a26640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad40070bd0 0x7fad40070fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.400+0000 7fad44a26640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fad40070bd0 0x7fad40070fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60220/0 (socket says 172.21.15.1:60220) 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.400+0000 7fad44a26640 1 -- 172.21.15.1:0/1876561558 learned_addr learned my addr 172.21.15.1:0/1876561558 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.400+0000 7fad44a26640 1 -- 172.21.15.1:0/1876561558 >> v1:172.21.15.196:6789/0 conn(0x7fad400c64b0 legacy=0x7fad4006e1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.400+0000 7fad44a26640 1 -- 172.21.15.1:0/1876561558 >> v1:172.21.15.1:6789/0 conn(0x7fad40156c90 legacy=0x7fad40068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.396 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.400+0000 7fad44a26640 1 -- 172.21.15.1:0/1876561558 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad4006eaf0 con 0x7fad40070bd0 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.401+0000 7fad44a26640 1 --2- 172.21.15.1:0/1876561558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad40070fe0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fad38009e50 tx=0x7fad3802a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65861b7d78d6db29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.401+0000 7fad3effd640 1 -- 172.21.15.1:0/1876561558 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad38036070 con 0x7fad40070bd0 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.401+0000 7fad3effd640 1 -- 172.21.15.1:0/1876561558 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad38031600 con 0x7fad40070bd0 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.401+0000 7fad3effd640 1 -- 172.21.15.1:0/1876561558 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad380319f0 con 0x7fad40070bd0 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.402+0000 7fad46489640 1 -- 172.21.15.1:0/1876561558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 msgr2=0x7fad40070fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.402+0000 7fad46489640 1 --2- 172.21.15.1:0/1876561558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad40070fe0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fad38009e50 tx=0x7fad3802a7c0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.402+0000 7fad46489640 1 -- 172.21.15.1:0/1876561558 shutdown_connections 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.402+0000 7fad46489640 1 --2- 172.21.15.1:0/1876561558 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad40070fe0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.402+0000 7fad46489640 1 -- 172.21.15.1:0/1876561558 >> 172.21.15.1:0/1876561558 conn(0x7fad400cc6a0 msgr2=0x7fad400ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.402+0000 7fad46489640 1 -- 172.21.15.1:0/1876561558 shutdown_connections 2024-05-09T05:23:40.398 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.402+0000 7fad46489640 1 -- 172.21.15.1:0/1876561558 wait complete. 2024-05-09T05:23:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.403+0000 7fad46489640 1 Processor -- start 2024-05-09T05:23:40.399 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.404+0000 7fad46489640 1 -- start start 2024-05-09T05:23:40.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad46489640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad4014fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad46489640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad400c64b0 0x7fad401503e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad46489640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad40156c90 0x7fad4015af30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad46489640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fad4015d160 con 0x7fad40070bd0 2024-05-09T05:23:40.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad46489640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fad4015ce80 con 0x7fad400c64b0 2024-05-09T05:23:40.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad46489640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fad4015d440 con 0x7fad40156c90 2024-05-09T05:23:40.402 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad4014fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3ffff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad4014fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60236/0 (socket says 172.21.15.1:60236) 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3ffff640 1 -- 172.21.15.1:0/4071195667 learned_addr learned my addr 172.21.15.1:0/4071195667 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3f7fe640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad400c64b0 0x7fad401503e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad44a26640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad40156c90 0x7fad4015af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3ffff640 1 -- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad40156c90 msgr2=0x7fad4015af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3ffff640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad40156c90 0x7fad4015af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3ffff640 1 -- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad400c64b0 msgr2=0x7fad401503e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3ffff640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad400c64b0 0x7fad401503e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad3ffff640 1 -- 172.21.15.1:0/4071195667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad401f7150 con 0x7fad40070bd0 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.405+0000 7fad44a26640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad40156c90 0x7fad4015af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.406+0000 7fad3ffff640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad4014fc10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fad3000c430 tx=0x7fad3000c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.406+0000 7fad3cff9640 1 -- 172.21.15.1:0/4071195667 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad30017070 con 0x7fad40070bd0 2024-05-09T05:23:40.403 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.406+0000 7fad3cff9640 1 -- 172.21.15.1:0/4071195667 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fad300052d0 con 0x7fad40070bd0 2024-05-09T05:23:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.406+0000 7fad3cff9640 1 -- 172.21.15.1:0/4071195667 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad30003830 con 0x7fad40070bd0 2024-05-09T05:23:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.406+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad40156500 con 0x7fad40070bd0 2024-05-09T05:23:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.406+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad4015abb0 con 0x7fad40070bd0 2024-05-09T05:23:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.407+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad4006eec0 con 0x7fad40070bd0 2024-05-09T05:23:40.404 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.408+0000 7fad3cff9640 1 -- 172.21.15.1:0/4071195667 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fad3001a070 con 0x7fad40070bd0 2024-05-09T05:23:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.408+0000 7fad3cff9640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad2c0762d0 0x7fad2c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.408+0000 7fad3cff9640 1 -- 172.21.15.1:0/4071195667 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fad30096a00 con 0x7fad40070bd0 2024-05-09T05:23:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.408+0000 7fad3f7fe640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad2c0762d0 0x7fad2c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.405 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.409+0000 7fad3f7fe640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad2c0762d0 0x7fad2c0787e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fad4015c8b0 tx=0x7fad34009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.409 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.413+0000 7fad3cff9640 1 -- 172.21.15.1:0/4071195667 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fad3005fc00 con 0x7fad40070bd0 2024-05-09T05:23:40.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.677+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "quorum_status"} v 0) -- 0x7fad400caf00 con 0x7fad40070bd0 2024-05-09T05:23:40.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.678+0000 7fad3cff9640 1 -- 172.21.15.1:0/4071195667 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "quorum_status"}]=0 v0) ==== 61+0+1562 (secure 0 0 0) 0x7fad300682d0 con 0x7fad40070bd0 2024-05-09T05:23:40.674 INFO:teuthology.orchestra.run.smithi001.stdout:{"election_epoch":138,"quorum":[0,1,2],"quorum_names":["a","b","c"],"quorum_leader_name":"a","quorum_age":12,"features":{"quorum_con":"4540701547738038271","quorum_mon":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"]},"monmap":{"epoch":1,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","modified":"2024-05-09T05:05:03.309340Z","created":"2024-05-09T05:05:03.309340Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:3300","nonce":0},{"type":"v1","addr":"172.21.15.1:6789","nonce":0}]},"addr":"172.21.15.1:6789/0","public_addr":"172.21.15.1:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3300","nonce":0},{"type":"v1","addr":"172.21.15.196:6789","nonce":0}]},"addr":"172.21.15.196:6789/0","public_addr":"172.21.15.196:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:3301","nonce":0},{"type":"v1","addr":"172.21.15.196:6790","nonce":0}]},"addr":"172.21.15.196:6790/0","public_addr":"172.21.15.196:6790/0","priority":0,"weight":0,"crush_location":"{}"}]}} 2024-05-09T05:23:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad2c0762d0 msgr2=0x7fad2c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.676 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad2c0762d0 0x7fad2c0787e0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fad4015c8b0 tx=0x7fad34009040 comp rx=0 tx=0).stop 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 msgr2=0x7fad4014fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad4014fc10 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7fad3000c430 tx=0x7fad3000c8f0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 shutdown_connections 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fad2c0762d0 0x7fad2c0787e0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fad400c64b0 0x7fad401503e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fad40156c90 0x7fad4015af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 --2- 172.21.15.1:0/4071195667 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fad40070bd0 0x7fad4014fc10 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 >> 172.21.15.1:0/4071195667 conn(0x7fad400cc6a0 msgr2=0x7fad40070fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 shutdown_connections 2024-05-09T05:23:40.677 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.680+0000 7fad46489640 1 -- 172.21.15.1:0/4071195667 wait complete. 2024-05-09T05:23:40.688 INFO:tasks.mon_thrash.ceph_manager:quorum is size 3 2024-05-09T05:23:40.688 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2024-05-09T05:23:40.699 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2024-05-09T05:23:40.699 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:23:40.699 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:23:40.724 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:23:40.725 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:23:40.782 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:23:40.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.942+0000 7f34c7861640 1 Processor -- start 2024-05-09T05:23:40.938 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.942+0000 7f34c7861640 1 -- start start 2024-05-09T05:23:40.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c7861640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c00c5a60 0x7f34c0159f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c7861640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34c00c3b40 0x7f34c00c3f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c7861640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f34c00707a0 con 0x7f34c01426a0 2024-05-09T05:23:40.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c7861640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f34c0153c00 con 0x7f34c00c3b40 2024-05-09T05:23:40.940 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c7861640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f34c0071be0 con 0x7f34c00c5a60 2024-05-09T05:23:40.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c5dfe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34c00c3b40 0x7f34c00c3f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c5dfe640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f34c00c3b40 0x7f34c00c3f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60252/0 (socket says 172.21.15.1:60252) 2024-05-09T05:23:40.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c5dfe640 1 -- 172.21.15.1:0/4170235316 learned_addr learned my addr 172.21.15.1:0/4170235316 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:40.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c55fd640 1 --2- 172.21.15.1:0/4170235316 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c00c5a60 0x7f34c0159f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c5dfe640 1 -- 172.21.15.1:0/4170235316 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c00c5a60 msgr2=0x7f34c0159f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:40.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c5dfe640 1 --2- 172.21.15.1:0/4170235316 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c00c5a60 0x7f34c0159f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c5dfe640 1 -- 172.21.15.1:0/4170235316 >> v1:172.21.15.1:6789/0 conn(0x7f34c01426a0 legacy=0x7f34c0072070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.941 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.944+0000 7f34c5dfe640 1 -- 172.21.15.1:0/4170235316 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c0070a80 con 0x7f34c00c3b40 2024-05-09T05:23:40.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.945+0000 7f34c5dfe640 1 --2- 172.21.15.1:0/4170235316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c00c3f50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009870 tx=0x7f34b802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6826ca75f52677 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.945+0000 7f34bffff640 1 -- 172.21.15.1:0/4170235316 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b8037070 con 0x7f34c00c3b40 2024-05-09T05:23:40.942 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.945+0000 7f34bffff640 1 -- 172.21.15.1:0/4170235316 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b8032440 con 0x7f34c00c3b40 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.945+0000 7f34bffff640 1 -- 172.21.15.1:0/4170235316 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b8032810 con 0x7f34c00c3b40 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.946+0000 7f34c7861640 1 -- 172.21.15.1:0/4170235316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 msgr2=0x7f34c00c3f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.946+0000 7f34c7861640 1 --2- 172.21.15.1:0/4170235316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c00c3f50 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f34b8009870 tx=0x7f34b802a270 comp rx=0 tx=0).stop 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.946+0000 7f34c7861640 1 -- 172.21.15.1:0/4170235316 shutdown_connections 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.946+0000 7f34c7861640 1 --2- 172.21.15.1:0/4170235316 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c00c3f50 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.946+0000 7f34c7861640 1 --2- 172.21.15.1:0/4170235316 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c00c5a60 0x7f34c0159f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.946+0000 7f34c7861640 1 -- 172.21.15.1:0/4170235316 >> 172.21.15.1:0/4170235316 conn(0x7f34c00ced00 msgr2=0x7f34c00cf110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.946+0000 7f34c7861640 1 -- 172.21.15.1:0/4170235316 shutdown_connections 2024-05-09T05:23:40.943 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.946+0000 7f34c7861640 1 -- 172.21.15.1:0/4170235316 wait complete. 2024-05-09T05:23:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.948+0000 7f34c7861640 1 Processor -- start 2024-05-09T05:23:40.944 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.948+0000 7f34c7861640 1 -- start start 2024-05-09T05:23:40.946 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c7861640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c0151ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c7861640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34c00c5a60 0x7f34c0153050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c7861640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c01426a0 0x7f34c0140960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c7861640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0071870 con 0x7f34c00c3b40 2024-05-09T05:23:40.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c7861640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f34c0071040 con 0x7f34c00c5a60 2024-05-09T05:23:40.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c7861640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f34c00ca260 con 0x7f34c01426a0 2024-05-09T05:23:40.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c55fd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c0151ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c55fd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c0151ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60262/0 (socket says 172.21.15.1:60262) 2024-05-09T05:23:40.947 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c55fd640 1 -- 172.21.15.1:0/662434796 learned_addr learned my addr 172.21.15.1:0/662434796 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:40.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c5dfe640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c01426a0 0x7f34c0140960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c4dfc640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34c00c5a60 0x7f34c0153050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c55fd640 1 -- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c01426a0 msgr2=0x7f34c0140960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c55fd640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c01426a0 0x7f34c0140960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c55fd640 1 -- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34c00c5a60 msgr2=0x7f34c0153050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:40.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c55fd640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34c00c5a60 0x7f34c0153050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:40.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.949+0000 7f34c55fd640 1 -- 172.21.15.1:0/662434796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34c0155030 con 0x7f34c00c3b40 2024-05-09T05:23:40.948 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.950+0000 7f34c55fd640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c0151ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009d10 tx=0x7f34b000c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.950+0000 7f34bdffb640 1 -- 172.21.15.1:0/662434796 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0018020 con 0x7f34c00c3b40 2024-05-09T05:23:40.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.950+0000 7f34bdffb640 1 -- 172.21.15.1:0/662434796 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f34b0014480 con 0x7f34c00c3b40 2024-05-09T05:23:40.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.950+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34c0152890 con 0x7f34c00c3b40 2024-05-09T05:23:40.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.950+0000 7f34bdffb640 1 -- 172.21.15.1:0/662434796 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0014830 con 0x7f34c00c3b40 2024-05-09T05:23:40.949 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.950+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34c0154d50 con 0x7f34c00c3b40 2024-05-09T05:23:40.950 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.951+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34c00696d0 con 0x7f34c00c3b40 2024-05-09T05:23:40.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.952+0000 7f34bdffb640 1 -- 172.21.15.1:0/662434796 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f34b0005d10 con 0x7f34c00c3b40 2024-05-09T05:23:40.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.952+0000 7f34bdffb640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34ac076300 0x7f34ac078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:40.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.953+0000 7f34bdffb640 1 -- 172.21.15.1:0/662434796 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f34b00980a0 con 0x7f34c00c3b40 2024-05-09T05:23:40.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.953+0000 7f34c4dfc640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34ac076300 0x7f34ac078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:40.951 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.953+0000 7f34c4dfc640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34ac076300 0x7f34ac078810 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f34b40015e0 tx=0x7f34b4009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:40.953 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:40.957+0000 7f34bdffb640 1 -- 172.21.15.1:0/662434796 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f34b0022070 con 0x7f34c00c3b40 2024-05-09T05:23:41.160 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.164+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f34c00ccfd0 con 0x7f34c00c3b40 2024-05-09T05:23:41.161 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.164+0000 7f34bdffb640 1 -- 172.21.15.1:0/662434796 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7f34b0010040 con 0x7f34c00c3b40 2024-05-09T05:23:41.161 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-05-09T05:23:41.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34ac076300 msgr2=0x7f34ac078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:41.163 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34ac076300 0x7f34ac078810 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f34b40015e0 tx=0x7f34b4009040 comp rx=0 tx=0).stop 2024-05-09T05:23:41.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 msgr2=0x7f34c0151ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:41.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c0151ba0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009d10 tx=0x7f34b000c490 comp rx=0 tx=0).stop 2024-05-09T05:23:41.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 shutdown_connections 2024-05-09T05:23:41.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f34ac076300 0x7f34ac078810 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f34c00c5a60 0x7f34c0153050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.164 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f34c01426a0 0x7f34c0140960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 --2- 172.21.15.1:0/662434796 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f34c00c3b40 0x7f34c0151ba0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.166+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 >> 172.21.15.1:0/662434796 conn(0x7f34c00ced00 msgr2=0x7f34c0141890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:41.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.167+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 shutdown_connections 2024-05-09T05:23:41.165 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.167+0000 7f34c7861640 1 -- 172.21.15.1:0/662434796 wait complete. 2024-05-09T05:23:41.175 DEBUG:teuthology.orchestra.run.smithi001:> set -ex 2024-05-09T05:23:41.175 DEBUG:teuthology.orchestra.run.smithi001:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T05:23:41.193 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T05:23:41.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.353+0000 7fbd542b8640 1 Processor -- start 2024-05-09T05:23:41.349 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.353+0000 7fbd542b8640 1 -- start start 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.354+0000 7fbd542b8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd4c154530 0x7fbd4c0cae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.354+0000 7fbd542b8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fbd4c0cb350 0x7fbd4c164ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.354+0000 7fbd542b8640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbd4c0cbf90 con 0x7fbd4c072400 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.354+0000 7fbd542b8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbd4c0cb760 con 0x7fbd4c154530 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.354+0000 7fbd542b8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fbd4c0ca3c0 con 0x7fbd4c0cb350 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.354+0000 7fbd51853640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd4c154530 0x7fbd4c0cae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.354+0000 7fbd51853640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbd4c154530 0x7fbd4c0cae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60266/0 (socket says 172.21.15.1:60266) 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.354+0000 7fbd51853640 1 -- 172.21.15.1:0/3181198927 learned_addr learned my addr 172.21.15.1:0/3181198927 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.355+0000 7fbd52855640 1 --2- 172.21.15.1:0/3181198927 >> v2:172.21.15.196:3300/0 conn(0x7fbd4c0cb350 0x7fbd4c164ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.355+0000 7fbd51853640 1 -- 172.21.15.1:0/3181198927 >> v2:172.21.15.196:3300/0 conn(0x7fbd4c0cb350 msgr2=0x7fbd4c164ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:41.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.355+0000 7fbd51853640 1 --2- 172.21.15.1:0/3181198927 >> v2:172.21.15.196:3300/0 conn(0x7fbd4c0cb350 0x7fbd4c164ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.355+0000 7fbd51853640 1 -- 172.21.15.1:0/3181198927 >> v1:172.21.15.196:6789/0 conn(0x7fbd4c072400 legacy=0x7fbd4c070f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.355+0000 7fbd51853640 1 -- 172.21.15.1:0/3181198927 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4c15a780 con 0x7fbd4c154530 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.355+0000 7fbd52855640 1 --2- 172.21.15.1:0/3181198927 >> v2:172.21.15.196:3300/0 conn(0x7fbd4c0cb350 0x7fbd4c164ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.355+0000 7fbd51853640 1 --2- 172.21.15.1:0/3181198927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c154530 0x7fbd4c0cae10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c072810 tx=0x7fbd3c02a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ba72bef09d06b4a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd51052640 1 -- 172.21.15.1:0/3181198927 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c037070 con 0x7fbd4c154530 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd51052640 1 -- 172.21.15.1:0/3181198927 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd3c032440 con 0x7fbd4c154530 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd51052640 1 -- 172.21.15.1:0/3181198927 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd3c032810 con 0x7fbd4c154530 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd542b8640 1 -- 172.21.15.1:0/3181198927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c154530 msgr2=0x7fbd4c0cae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd542b8640 1 --2- 172.21.15.1:0/3181198927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c154530 0x7fbd4c0cae10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fbd4c072810 tx=0x7fbd3c02a270 comp rx=0 tx=0).stop 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd542b8640 1 -- 172.21.15.1:0/3181198927 shutdown_connections 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd542b8640 1 --2- 172.21.15.1:0/3181198927 >> v2:172.21.15.196:3300/0 conn(0x7fbd4c0cb350 0x7fbd4c164ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd542b8640 1 --2- 172.21.15.1:0/3181198927 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c154530 0x7fbd4c0cae10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd542b8640 1 -- 172.21.15.1:0/3181198927 >> 172.21.15.1:0/3181198927 conn(0x7fbd4c0d0470 msgr2=0x7fbd4c0cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:41.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.356+0000 7fbd542b8640 1 -- 172.21.15.1:0/3181198927 shutdown_connections 2024-05-09T05:23:41.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.357+0000 7fbd542b8640 1 -- 172.21.15.1:0/3181198927 wait complete. 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.358+0000 7fbd542b8640 1 Processor -- start 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.358+0000 7fbd542b8640 1 -- start start 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.359+0000 7fbd542b8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c072400 0x7fbd4c1f1610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.359+0000 7fbd542b8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd4c0cb350 0x7fbd4c15dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.359+0000 7fbd542b8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd4c154530 0x7fbd4c0b4400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.359+0000 7fbd542b8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c154940 con 0x7fbd4c072400 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.359+0000 7fbd542b8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbd4c0ca8f0 con 0x7fbd4c154530 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.359+0000 7fbd542b8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbd4c0b2630 con 0x7fbd4c0cb350 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c072400 0x7fbd4c1f1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c072400 0x7fbd4c1f1610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60276/0 (socket says 172.21.15.1:60276) 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 -- 172.21.15.1:0/2533832501 learned_addr learned my addr 172.21.15.1:0/2533832501 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52855640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd4c154530 0x7fbd4c0b4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:41.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd51853640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd4c0cb350 0x7fbd4c15dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 -- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd4c0cb350 msgr2=0x7fbd4c15dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd4c0cb350 0x7fbd4c15dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 -- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd4c154530 msgr2=0x7fbd4c0b4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd4c154530 0x7fbd4c0b4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 -- 172.21.15.1:0/2533832501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd4c14e940 con 0x7fbd4c072400 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52855640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd4c154530 0x7fbd4c0b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd51853640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd4c0cb350 0x7fbd4c15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd52054640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c072400 0x7fbd4c1f1610 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800c3d0 tx=0x7fbd4800c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd42ffd640 1 -- 172.21.15.1:0/2533832501 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48017070 con 0x7fbd4c072400 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd42ffd640 1 -- 172.21.15.1:0/2533832501 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbd480052d0 con 0x7fbd4c072400 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd4c1e6620 con 0x7fbd4c072400 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd42ffd640 1 -- 172.21.15.1:0/2533832501 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd48003830 con 0x7fbd4c072400 2024-05-09T05:23:41.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.360+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd4c1f0e10 con 0x7fbd4c072400 2024-05-09T05:23:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.361+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbd4c050870 con 0x7fbd4c072400 2024-05-09T05:23:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.363+0000 7fbd42ffd640 1 -- 172.21.15.1:0/2533832501 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbd4801a070 con 0x7fbd4c072400 2024-05-09T05:23:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.363+0000 7fbd42ffd640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd30076300 0x7fbd30078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.363+0000 7fbd42ffd640 1 -- 172.21.15.1:0/2533832501 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbd48096b70 con 0x7fbd4c072400 2024-05-09T05:23:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.364+0000 7fbd51853640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd30076300 0x7fbd30078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:41.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.364+0000 7fbd51853640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd30076300 0x7fbd30078810 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c034000 tx=0x7fbd3c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:41.363 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.367+0000 7fbd42ffd640 1 -- 172.21.15.1:0/2533832501 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbd48021080 con 0x7fbd4c072400 2024-05-09T05:23:41.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.573+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7fbd4c0b3460 con 0x7fbd4c072400 2024-05-09T05:23:41.569 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.573+0000 7fbd42ffd640 1 -- 172.21.15.1:0/2533832501 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 0 entries v23) ==== 80+0+0 (secure 0 0 0) 0x7fbd48060530 con 0x7fbd4c072400 2024-05-09T05:23:41.569 INFO:teuthology.orchestra.run.smithi001.stderr:listed 0 entries 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.575+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd30076300 msgr2=0x7fbd30078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.575+0000 7fbd542b8640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd30076300 0x7fbd30078810 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fbd3c034000 tx=0x7fbd3c02e000 comp rx=0 tx=0).stop 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.575+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c072400 msgr2=0x7fbd4c1f1610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.575+0000 7fbd542b8640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c072400 0x7fbd4c1f1610 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fbd4800c3d0 tx=0x7fbd4800c890 comp rx=0 tx=0).stop 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.576+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 shutdown_connections 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.576+0000 7fbd542b8640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbd30076300 0x7fbd30078810 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.576+0000 7fbd542b8640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbd4c154530 0x7fbd4c0b4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.576+0000 7fbd542b8640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbd4c0cb350 0x7fbd4c15dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.576+0000 7fbd542b8640 1 --2- 172.21.15.1:0/2533832501 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbd4c072400 0x7fbd4c1f1610 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:41.572 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.576+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 >> 172.21.15.1:0/2533832501 conn(0x7fbd4c0d0470 msgr2=0x7fbd4c0c59e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.576+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 shutdown_connections 2024-05-09T05:23:41.573 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:41.576+0000 7fbd542b8640 1 -- 172.21.15.1:0/2533832501 wait complete. 2024-05-09T05:23:41.583 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi001.front.sepia.ceph.com... 2024-05-09T05:23:41.583 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:23:41.583 DEBUG:teuthology.orchestra.run.smithi001:> sudo fusermount -u /home/ubuntu/cephtest/mnt.0 2024-05-09T05:23:41.624 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:23:41.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi001.stderr:ceph-fuse[43333]: fuse finished with error 0 and tester_r 0 2024-05-09T05:23:47.727 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:23:47.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:23:47.727 DEBUG:teuthology.orchestra.run.smithi001:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T05:23:47.748 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:23:47.748 DEBUG:teuthology.orchestra.run.smithi001:> ip netns list 2024-05-09T05:23:47.831 INFO:teuthology.orchestra.run.smithi001.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T05:23:47.831 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:23:47.831 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T05:23:47.903 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T05:23:47.903 DEBUG:teuthology.orchestra.run.smithi001:> sudo ip link delete ceph-brx 2024-05-09T05:23:48.011 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-05-09T05:23:48.020 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2024-05-09T05:23:48.020 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:23:48.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.146+0000 7f0f22d8d640 1 Processor -- start 2024-05-09T05:23:48.144 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.146+0000 7f0f22d8d640 1 -- start start 2024-05-09T05:23:48.145 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f22d8d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0c5150 0x7f0f1c0c61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f22d8d640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0f1c15ff10 0x7f0f1c156b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f22d8d640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f0f1c160320 con 0x7f0f1c0cfcd0 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f22d8d640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f0f1c157070 con 0x7f0f1c15ff10 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f22d8d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0f1c06f8d0 con 0x7f0f1c0c5150 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f1bfff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0f1c15ff10 0x7f0f1c156b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f20b29640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0c5150 0x7f0f1c0c61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f20b29640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0c5150 0x7f0f1c0c61d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:53520/0 (socket says 172.21.15.1:53520) 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f1bfff640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f0f1c15ff10 0x7f0f1c156b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:38240/0 (socket says 172.21.15.1:38240) 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f20b29640 1 -- 172.21.15.1:0/3279963862 learned_addr learned my addr 172.21.15.1:0/3279963862 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.148+0000 7f0f2132a640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7f0f1c0cfcd0 0x7f0f1c0cc4a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:58620/0 (socket says 172.21.15.1:58620) 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.149+0000 7f0f1bfff640 1 -- 172.21.15.1:0/3279963862 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0c5150 msgr2=0x7f0f1c0c61d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.149+0000 7f0f1bfff640 1 --2- 172.21.15.1:0/3279963862 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0c5150 0x7f0f1c0c61d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.146 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.149+0000 7f0f1bfff640 1 -- 172.21.15.1:0/3279963862 >> v1:172.21.15.196:6789/0 conn(0x7f0f1c0cfcd0 legacy=0x7f0f1c0cc4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.149+0000 7f0f1bfff640 1 -- 172.21.15.1:0/3279963862 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1c0caf50 con 0x7f0f1c15ff10 2024-05-09T05:23:48.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.149+0000 7f0f20b29640 1 --2- 172.21.15.1:0/3279963862 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0c5150 0x7f0f1c0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.149+0000 7f0f1bfff640 1 --2- 172.21.15.1:0/3279963862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c156b30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0f100097a0 tx=0x7f0f1002a210 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbd94faf009b6a0e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f1b7fe640 1 -- 172.21.15.1:0/3279963862 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f10037070 con 0x7f0f1c15ff10 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f1b7fe640 1 -- 172.21.15.1:0/3279963862 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f10032440 con 0x7f0f1c15ff10 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f1b7fe640 1 -- 172.21.15.1:0/3279963862 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f10032810 con 0x7f0f1c15ff10 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3279963862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c15ff10 msgr2=0x7f0f1c156b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3279963862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c156b30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f0f100097a0 tx=0x7f0f1002a210 comp rx=0 tx=0).stop 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3279963862 shutdown_connections 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3279963862 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c156b30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3279963862 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0c5150 0x7f0f1c0c61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3279963862 >> 172.21.15.1:0/3279963862 conn(0x7f0f1c068830 msgr2=0x7f0f1c068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:48.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3279963862 shutdown_connections 2024-05-09T05:23:48.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.150+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3279963862 wait complete. 2024-05-09T05:23:48.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.152+0000 7f0f22d8d640 1 Processor -- start 2024-05-09T05:23:48.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.152+0000 7f0f22d8d640 1 -- start start 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f22d8d640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c0c5150 0x7f0f1c0ce350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f22d8d640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0cfcd0 0x7f0f1c1f2f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f22d8d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c15f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f22d8d640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c0c5cb0 con 0x7f0f1c15ff10 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f22d8d640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f0f1c0c59d0 con 0x7f0f1c0c5150 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f22d8d640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f0f1c0cb750 con 0x7f0f1c0cfcd0 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f2132a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c15f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f20b29640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c0c5150 0x7f0f1c0ce350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f1bfff640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0cfcd0 0x7f0f1c1f2f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.152 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f2132a640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c15f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60290/0 (socket says 172.21.15.1:60290) 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f2132a640 1 -- 172.21.15.1:0/3007541556 learned_addr learned my addr 172.21.15.1:0/3007541556 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f2132a640 1 -- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0cfcd0 msgr2=0x7f0f1c1f2f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f2132a640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0cfcd0 0x7f0f1c1f2f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f2132a640 1 -- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c0c5150 msgr2=0x7f0f1c0ce350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f2132a640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c0c5150 0x7f0f1c0ce350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.154+0000 7f0f2132a640 1 -- 172.21.15.1:0/3007541556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f1c1f34d0 con 0x7f0f1c15ff10 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.155+0000 7f0f2132a640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c15f050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400c730 tx=0x7f0f1400cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.155+0000 7f0f197fa640 1 -- 172.21.15.1:0/3007541556 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f14017070 con 0x7f0f1c15ff10 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.155+0000 7f0f197fa640 1 -- 172.21.15.1:0/3007541556 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f0f1400ce40 con 0x7f0f1c15ff10 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.155+0000 7f0f197fa640 1 -- 172.21.15.1:0/3007541556 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1400a7e0 con 0x7f0f1c15ff10 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.155+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f1c14ebd0 con 0x7f0f1c15ff10 2024-05-09T05:23:48.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.155+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f1c14f190 con 0x7f0f1c15ff10 2024-05-09T05:23:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.156+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f1c1f36a0 con 0x7f0f1c15ff10 2024-05-09T05:23:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.157+0000 7f0f197fa640 1 -- 172.21.15.1:0/3007541556 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f0f1401a070 con 0x7f0f1c15ff10 2024-05-09T05:23:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.157+0000 7f0f197fa640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f08076300 0x7f0f08078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.157+0000 7f0f197fa640 1 -- 172.21.15.1:0/3007541556 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f0f14096ad0 con 0x7f0f1c15ff10 2024-05-09T05:23:48.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.158+0000 7f0f20b29640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f08076300 0x7f0f08078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.158+0000 7f0f20b29640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f08076300 0x7f0f08078810 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c006ca0 tx=0x7f0f0c009240 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.159 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.162+0000 7f0f197fa640 1 -- 172.21.15.1:0/3007541556 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f0f1c160320 con 0x7f0f1c15ff10 2024-05-09T05:23:48.345 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.348+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0f1c0cb9f0 con 0x7f0f08076300 2024-05-09T05:23:48.352 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.355+0000 7f0f197fa640 1 -- 172.21.15.1:0/3007541556 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253608 (secure 0 0 0) 0x7f0f1c0cb9f0 con 0x7f0f08076300 2024-05-09T05:23:48.355 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:23:48.357 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:23:48.358 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f08076300 msgr2=0x7f0f08078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f08076300 0x7f0f08078810 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f0f0c006ca0 tx=0x7f0f0c009240 comp rx=0 tx=0).stop 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c15ff10 msgr2=0x7f0f1c15f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c15f050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0f1400c730 tx=0x7f0f1400cbf0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 shutdown_connections 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0f08076300 0x7f0f08078810 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f0f1c15ff10 0x7f0f1c15f050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f0f1c0cfcd0 0x7f0f1c1f2f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 --2- 172.21.15.1:0/3007541556 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f0f1c0c5150 0x7f0f1c0ce350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 >> 172.21.15.1:0/3007541556 conn(0x7f0f1c068830 msgr2=0x7f0f1c0ca0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 shutdown_connections 2024-05-09T05:23:48.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.359+0000 7f0f22d8d640 1 -- 172.21.15.1:0/3007541556 wait complete. 2024-05-09T05:23:48.367 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-05-09T05:23:47.833121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19180393,"num_objects":105,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":102,"num_read_kb":167,"num_write":1813207,"num_write_kb":20380719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366735,"ondisk_log_size":366735,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4716776,"kb_used_data":39528,"kb_used_omap":12,"kb_used_meta":4677171,"kb_avail":750257944,"statfs":{"total":773094113280,"available":768264134656,"internally_reserved":0,"allocated":40476672,"data_stored":39046538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4789423665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003605"},"pg_stats":[{"pgid":"3.3f","version":"23'5187","reported_seq":5211,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.889349+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:23:41.889349+0000","last_peered":"2024-05-09T05:23:41.889349+0000","last_clean":"2024-05-09T05:23:41.889349+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:23:41.889349+0000","last_undegraded":"2024-05-09T05:23:41.889349+0000","last_fullsized":"2024-05-09T05:23:41.889349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26135,"num_write_kb":280897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'6912","reported_seq":6936,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.786450+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:23:41.786450+0000","last_peered":"2024-05-09T05:23:41.786450+0000","last_clean":"2024-05-09T05:23:41.786450+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:23:41.786450+0000","last_undegraded":"2024-05-09T05:23:41.786450+0000","last_fullsized":"2024-05-09T05:23:41.786450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35479,"num_write_kb":400336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'5571","reported_seq":5595,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.785914+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:23:41.785914+0000","last_peered":"2024-05-09T05:23:41.785914+0000","last_clean":"2024-05-09T05:23:41.785914+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:23:41.785914+0000","last_undegraded":"2024-05-09T05:23:41.785914+0000","last_fullsized":"2024-05-09T05:23:41.785914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:48.770365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27199,"num_write_kb":317974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'5494","reported_seq":5518,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.804206+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:23:41.804206+0000","last_peered":"2024-05-09T05:23:41.804206+0000","last_clean":"2024-05-09T05:23:41.804206+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:23:41.804206+0000","last_undegraded":"2024-05-09T05:23:41.804206+0000","last_fullsized":"2024-05-09T05:23:41.804206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28054,"num_write_kb":296869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'7058","reported_seq":7081,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790501+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:23:41.790501+0000","last_peered":"2024-05-09T05:23:41.790501+0000","last_clean":"2024-05-09T05:23:41.790501+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:23:41.790501+0000","last_undegraded":"2024-05-09T05:23:41.790501+0000","last_fullsized":"2024-05-09T05:23:41.790501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34281,"num_write_kb":379027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'4948","reported_seq":4971,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.889531+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:23:41.889531+0000","last_peered":"2024-05-09T05:23:41.889531+0000","last_clean":"2024-05-09T05:23:41.889531+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:23:41.889531+0000","last_undegraded":"2024-05-09T05:23:41.889531+0000","last_fullsized":"2024-05-09T05:23:41.889531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24780,"num_write_kb":275405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'5454","reported_seq":5477,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.811426+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:23:41.811426+0000","last_peered":"2024-05-09T05:23:41.811426+0000","last_clean":"2024-05-09T05:23:41.811426+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:23:41.811426+0000","last_undegraded":"2024-05-09T05:23:41.811426+0000","last_fullsized":"2024-05-09T05:23:41.811426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27432,"num_write_kb":306787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'6053","reported_seq":6076,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.840232+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:23:41.840232+0000","last_peered":"2024-05-09T05:23:41.840232+0000","last_clean":"2024-05-09T05:23:41.840232+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:23:41.840232+0000","last_undegraded":"2024-05-09T05:23:41.840232+0000","last_fullsized":"2024-05-09T05:23:41.840232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6053,"log_dups_size":0,"ondisk_log_size":6053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29445,"num_write_kb":339212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"23'7129","reported_seq":7152,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.913522+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:23:41.913522+0000","last_peered":"2024-05-09T05:23:41.913522+0000","last_clean":"2024-05-09T05:23:41.913522+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:23:41.913522+0000","last_undegraded":"2024-05-09T05:23:41.913522+0000","last_fullsized":"2024-05-09T05:23:41.913522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36611,"num_write_kb":392539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'6358","reported_seq":6381,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.896558+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:23:41.896558+0000","last_peered":"2024-05-09T05:23:41.896558+0000","last_clean":"2024-05-09T05:23:41.896558+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:23:41.896558+0000","last_undegraded":"2024-05-09T05:23:41.896558+0000","last_fullsized":"2024-05-09T05:23:41.896558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32647,"num_write_kb":360281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'4747","reported_seq":4770,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.882769+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:23:41.882769+0000","last_peered":"2024-05-09T05:23:41.882769+0000","last_clean":"2024-05-09T05:23:41.882769+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:23:41.882769+0000","last_undegraded":"2024-05-09T05:23:41.882769+0000","last_fullsized":"2024-05-09T05:23:41.882769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22850,"num_write_kb":288147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'4814","reported_seq":4837,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.916891+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:23:41.916891+0000","last_peered":"2024-05-09T05:23:41.916891+0000","last_clean":"2024-05-09T05:23:41.916891+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:23:41.916891+0000","last_undegraded":"2024-05-09T05:23:41.916891+0000","last_fullsized":"2024-05-09T05:23:41.916891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":267591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'5105","reported_seq":5128,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.762396+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:23:41.762396+0000","last_peered":"2024-05-09T05:23:41.762396+0000","last_clean":"2024-05-09T05:23:41.762396+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:23:41.762396+0000","last_undegraded":"2024-05-09T05:23:41.762396+0000","last_fullsized":"2024-05-09T05:23:41.762396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25963,"num_write_kb":288363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'5957","reported_seq":5980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.819716+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:23:41.819716+0000","last_peered":"2024-05-09T05:23:41.819716+0000","last_clean":"2024-05-09T05:23:41.819716+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:23:41.819716+0000","last_undegraded":"2024-05-09T05:23:41.819716+0000","last_fullsized":"2024-05-09T05:23:41.819716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:37.258923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29976,"num_write_kb":329157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'5739","reported_seq":5762,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.843331+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:23:41.843331+0000","last_peered":"2024-05-09T05:23:41.843331+0000","last_clean":"2024-05-09T05:23:41.843331+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:23:41.843331+0000","last_undegraded":"2024-05-09T05:23:41.843331+0000","last_fullsized":"2024-05-09T05:23:41.843331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28226,"num_write_kb":322058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'5341","reported_seq":5364,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.852751+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:23:41.852751+0000","last_peered":"2024-05-09T05:23:41.852751+0000","last_clean":"2024-05-09T05:23:41.852751+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:23:41.852751+0000","last_undegraded":"2024-05-09T05:23:41.852751+0000","last_fullsized":"2024-05-09T05:23:41.852751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27307,"num_write_kb":289793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"23'6965","reported_seq":6982,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803206+0000","last_change":"2024-05-09T05:07:01.750820+0000","last_active":"2024-05-09T05:23:41.803206+0000","last_peered":"2024-05-09T05:23:41.803206+0000","last_clean":"2024-05-09T05:23:41.803206+0000","last_became_active":"2024-05-09T05:07:01.750492+0000","last_became_peered":"2024-05-09T05:07:01.750492+0000","last_unstale":"2024-05-09T05:23:41.803206+0000","last_undegraded":"2024-05-09T05:23:41.803206+0000","last_fullsized":"2024-05-09T05:23:41.803206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6965,"log_dups_size":0,"ondisk_log_size":6965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:23.141520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34877,"num_write_kb":385595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"23'4549","reported_seq":4572,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.791997+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:23:41.791997+0000","last_peered":"2024-05-09T05:23:41.791997+0000","last_clean":"2024-05-09T05:23:41.791997+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:23:41.791997+0000","last_undegraded":"2024-05-09T05:23:41.791997+0000","last_fullsized":"2024-05-09T05:23:41.791997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22029,"num_write_kb":252128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'5335","reported_seq":5358,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.781003+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:23:41.781003+0000","last_peered":"2024-05-09T05:23:41.781003+0000","last_clean":"2024-05-09T05:23:41.781003+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:23:41.781003+0000","last_undegraded":"2024-05-09T05:23:41.781003+0000","last_fullsized":"2024-05-09T05:23:41.781003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26106,"num_write_kb":309468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'5370","reported_seq":5393,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.861348+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:23:41.861348+0000","last_peered":"2024-05-09T05:23:41.861348+0000","last_clean":"2024-05-09T05:23:41.861348+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:23:41.861348+0000","last_undegraded":"2024-05-09T05:23:41.861348+0000","last_fullsized":"2024-05-09T05:23:41.861348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26990,"num_write_kb":308937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'5403","reported_seq":5426,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.859039+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:23:41.859039+0000","last_peered":"2024-05-09T05:23:41.859039+0000","last_clean":"2024-05-09T05:23:41.859039+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:23:41.859039+0000","last_undegraded":"2024-05-09T05:23:41.859039+0000","last_fullsized":"2024-05-09T05:23:41.859039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:02.665344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26698,"num_write_kb":315887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'6082","reported_seq":6105,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.904062+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:23:41.904062+0000","last_peered":"2024-05-09T05:23:41.904062+0000","last_clean":"2024-05-09T05:23:41.904062+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:23:41.904062+0000","last_undegraded":"2024-05-09T05:23:41.904062+0000","last_fullsized":"2024-05-09T05:23:41.904062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30462,"num_write_kb":343332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"23'5512","reported_seq":5535,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.774669+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:23:41.774669+0000","last_peered":"2024-05-09T05:23:41.774669+0000","last_clean":"2024-05-09T05:23:41.774669+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:23:41.774669+0000","last_undegraded":"2024-05-09T05:23:41.774669+0000","last_fullsized":"2024-05-09T05:23:41.774669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27471,"num_write_kb":310767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'5065","reported_seq":5089,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.839542+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:23:41.839542+0000","last_peered":"2024-05-09T05:23:41.839542+0000","last_clean":"2024-05-09T05:23:41.839542+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:23:41.839542+0000","last_undegraded":"2024-05-09T05:23:41.839542+0000","last_fullsized":"2024-05-09T05:23:41.839542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24983,"num_write_kb":280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'6632","reported_seq":6656,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.735021+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:23:41.735021+0000","last_peered":"2024-05-09T05:23:41.735021+0000","last_clean":"2024-05-09T05:23:41.735021+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:23:41.735021+0000","last_undegraded":"2024-05-09T05:23:41.735021+0000","last_fullsized":"2024-05-09T05:23:41.735021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:25:28.166958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33114,"num_write_kb":370371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'5121","reported_seq":5150,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.872969+0000","last_change":"2024-05-09T05:07:01.662425+0000","last_active":"2024-05-09T05:23:41.872969+0000","last_peered":"2024-05-09T05:23:41.872969+0000","last_clean":"2024-05-09T05:23:41.872969+0000","last_became_active":"2024-05-09T05:07:01.662096+0000","last_became_peered":"2024-05-09T05:07:01.662096+0000","last_unstale":"2024-05-09T05:23:41.872969+0000","last_undegraded":"2024-05-09T05:23:41.872969+0000","last_fullsized":"2024-05-09T05:23:41.872969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:50.949207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25567,"num_write_kb":287181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'6014","reported_seq":6038,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.767615+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:23:41.767615+0000","last_peered":"2024-05-09T05:23:41.767615+0000","last_clean":"2024-05-09T05:23:41.767615+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:23:41.767615+0000","last_undegraded":"2024-05-09T05:23:41.767615+0000","last_fullsized":"2024-05-09T05:23:41.767615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30657,"num_write_kb":340145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'5132","reported_seq":5149,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.864977+0000","last_change":"2024-05-09T05:07:01.723062+0000","last_active":"2024-05-09T05:23:41.864977+0000","last_peered":"2024-05-09T05:23:41.864977+0000","last_clean":"2024-05-09T05:23:41.864977+0000","last_became_active":"2024-05-09T05:07:01.722736+0000","last_became_peered":"2024-05-09T05:07:01.722736+0000","last_unstale":"2024-05-09T05:23:41.864977+0000","last_undegraded":"2024-05-09T05:23:41.864977+0000","last_fullsized":"2024-05-09T05:23:41.864977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:50.949207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25502,"num_write_kb":286500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"23'4664","reported_seq":4688,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.846625+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:23:41.846625+0000","last_peered":"2024-05-09T05:23:41.846625+0000","last_clean":"2024-05-09T05:23:41.846625+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:23:41.846625+0000","last_undegraded":"2024-05-09T05:23:41.846625+0000","last_fullsized":"2024-05-09T05:23:41.846625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23461,"num_write_kb":270138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'6545","reported_seq":6569,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.857347+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:23:41.857347+0000","last_peered":"2024-05-09T05:23:41.857347+0000","last_clean":"2024-05-09T05:23:41.857347+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:23:41.857347+0000","last_undegraded":"2024-05-09T05:23:41.857347+0000","last_fullsized":"2024-05-09T05:23:41.857347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6545,"log_dups_size":0,"ondisk_log_size":6545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32981,"num_write_kb":377918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'6387","reported_seq":6411,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.809273+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:23:41.809273+0000","last_peered":"2024-05-09T05:23:41.809273+0000","last_clean":"2024-05-09T05:23:41.809273+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:23:41.809273+0000","last_undegraded":"2024-05-09T05:23:41.809273+0000","last_fullsized":"2024-05-09T05:23:41.809273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32133,"num_write_kb":346002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'7361","reported_seq":7385,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.851736+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:23:41.851736+0000","last_peered":"2024-05-09T05:23:41.851736+0000","last_clean":"2024-05-09T05:23:41.851736+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:23:41.851736+0000","last_undegraded":"2024-05-09T05:23:41.851736+0000","last_fullsized":"2024-05-09T05:23:41.851736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36142,"num_write_kb":398870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'5228","reported_seq":5251,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803020+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:23:41.803020+0000","last_peered":"2024-05-09T05:23:41.803020+0000","last_clean":"2024-05-09T05:23:41.803020+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:23:41.803020+0000","last_undegraded":"2024-05-09T05:23:41.803020+0000","last_fullsized":"2024-05-09T05:23:41.803020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25537,"num_write_kb":278419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'577","reported_seq":602,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:36.679365+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:23:36.679365+0000","last_peered":"2024-05-09T05:23:36.679365+0000","last_clean":"2024-05-09T05:23:36.679365+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:23:36.679365+0000","last_undegraded":"2024-05-09T05:23:36.679365+0000","last_fullsized":"2024-05-09T05:23:36.679365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":573,"num_write_kb":4218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'4521","reported_seq":4544,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.926128+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:23:41.926128+0000","last_peered":"2024-05-09T05:23:41.926128+0000","last_clean":"2024-05-09T05:23:41.926128+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:23:41.926128+0000","last_undegraded":"2024-05-09T05:23:41.926128+0000","last_fullsized":"2024-05-09T05:23:41.926128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22336,"num_write_kb":249223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'111","reported_seq":136,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.634306+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:23:41.634306+0000","last_peered":"2024-05-09T05:23:41.634306+0000","last_clean":"2024-05-09T05:23:41.634306+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:23:41.634306+0000","last_undegraded":"2024-05-09T05:23:41.634306+0000","last_fullsized":"2024-05-09T05:23:41.634306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1715788,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":1741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.829290+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:22:54.829290+0000","last_peered":"2024-05-09T05:22:54.829290+0000","last_clean":"2024-05-09T05:22:54.829290+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:22:54.829290+0000","last_undegraded":"2024-05-09T05:22:54.829290+0000","last_fullsized":"2024-05-09T05:22:54.829290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'5503","reported_seq":5526,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.842314+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:23:41.842314+0000","last_peered":"2024-05-09T05:23:41.842314+0000","last_clean":"2024-05-09T05:23:41.842314+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:23:41.842314+0000","last_undegraded":"2024-05-09T05:23:41.842314+0000","last_fullsized":"2024-05-09T05:23:41.842314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27748,"num_write_kb":311965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.156887+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:22:59.156887+0000","last_peered":"2024-05-09T05:22:59.156887+0000","last_clean":"2024-05-09T05:22:59.156887+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:22:59.156887+0000","last_undegraded":"2024-05-09T05:22:59.156887+0000","last_fullsized":"2024-05-09T05:22:59.156887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'6912","reported_seq":6935,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.823969+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:23:41.823969+0000","last_peered":"2024-05-09T05:23:41.823969+0000","last_clean":"2024-05-09T05:23:41.823969+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:23:41.823969+0000","last_undegraded":"2024-05-09T05:23:41.823969+0000","last_fullsized":"2024-05-09T05:23:41.823969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34348,"num_write_kb":375438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'9","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.705733+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:23:41.705733+0000","last_peered":"2024-05-09T05:23:41.705733+0000","last_clean":"2024-05-09T05:23:41.705733+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:23:41.705733+0000","last_undegraded":"2024-05-09T05:23:41.705733+0000","last_fullsized":"2024-05-09T05:23:41.705733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'5546","reported_seq":5569,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.742360+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:23:41.742360+0000","last_peered":"2024-05-09T05:23:41.742360+0000","last_clean":"2024-05-09T05:23:41.742360+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:23:41.742360+0000","last_undegraded":"2024-05-09T05:23:41.742360+0000","last_fullsized":"2024-05-09T05:23:41.742360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5546,"log_dups_size":0,"ondisk_log_size":5546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27252,"num_write_kb":314086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.148513+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:22:59.148513+0000","last_peered":"2024-05-09T05:22:59.148513+0000","last_clean":"2024-05-09T05:22:59.148513+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:22:59.148513+0000","last_undegraded":"2024-05-09T05:22:59.148513+0000","last_fullsized":"2024-05-09T05:22:59.148513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'5049","reported_seq":5072,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.871865+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:23:41.871865+0000","last_peered":"2024-05-09T05:23:41.871865+0000","last_clean":"2024-05-09T05:23:41.871865+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:23:41.871865+0000","last_undegraded":"2024-05-09T05:23:41.871865+0000","last_fullsized":"2024-05-09T05:23:41.871865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:26.607004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25089,"num_write_kb":276787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'25","reported_seq":50,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.204360+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:22:59.204360+0000","last_peered":"2024-05-09T05:22:59.204360+0000","last_clean":"2024-05-09T05:22:59.204360+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:22:59.204360+0000","last_undegraded":"2024-05-09T05:22:59.204360+0000","last_fullsized":"2024-05-09T05:22:59.204360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'7381","reported_seq":7404,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.917166+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:23:41.917166+0000","last_peered":"2024-05-09T05:23:41.917166+0000","last_clean":"2024-05-09T05:23:41.917166+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:23:41.917166+0000","last_undegraded":"2024-05-09T05:23:41.917166+0000","last_fullsized":"2024-05-09T05:23:41.917166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7381,"log_dups_size":0,"ondisk_log_size":7381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36744,"num_write_kb":418024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'333","reported_seq":358,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.155427+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:22:59.155427+0000","last_peered":"2024-05-09T05:22:59.155427+0000","last_clean":"2024-05-09T05:22:59.155427+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:22:59.155427+0000","last_undegraded":"2024-05-09T05:22:59.155427+0000","last_fullsized":"2024-05-09T05:22:59.155427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":331,"num_write_kb":4250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'5538","reported_seq":5561,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.787753+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:23:41.787753+0000","last_peered":"2024-05-09T05:23:41.787753+0000","last_clean":"2024-05-09T05:23:41.787753+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:23:41.787753+0000","last_undegraded":"2024-05-09T05:23:41.787753+0000","last_fullsized":"2024-05-09T05:23:41.787753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27557,"num_write_kb":302974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"23'15","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.113005+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:22:59.113005+0000","last_peered":"2024-05-09T05:22:59.113005+0000","last_clean":"2024-05-09T05:22:59.113005+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:22:59.113005+0000","last_undegraded":"2024-05-09T05:22:59.113005+0000","last_fullsized":"2024-05-09T05:22:59.113005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'5579","reported_seq":5602,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.833702+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:23:41.833702+0000","last_peered":"2024-05-09T05:23:41.833702+0000","last_clean":"2024-05-09T05:23:41.833702+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:23:41.833702+0000","last_undegraded":"2024-05-09T05:23:41.833702+0000","last_fullsized":"2024-05-09T05:23:41.833702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:41.283927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27823,"num_write_kb":303973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'10","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.117627+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:22:59.117627+0000","last_peered":"2024-05-09T05:22:59.117627+0000","last_clean":"2024-05-09T05:22:59.117627+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:22:59.117627+0000","last_undegraded":"2024-05-09T05:22:59.117627+0000","last_fullsized":"2024-05-09T05:22:59.117627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'6303","reported_seq":6326,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.912348+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:23:41.912348+0000","last_peered":"2024-05-09T05:23:41.912348+0000","last_clean":"2024-05-09T05:23:41.912348+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:23:41.912348+0000","last_undegraded":"2024-05-09T05:23:41.912348+0000","last_fullsized":"2024-05-09T05:23:41.912348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30554,"num_write_kb":351862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'13","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.160623+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:22:59.160623+0000","last_peered":"2024-05-09T05:22:59.160623+0000","last_clean":"2024-05-09T05:22:59.160623+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:22:59.160623+0000","last_undegraded":"2024-05-09T05:22:59.160623+0000","last_fullsized":"2024-05-09T05:22:59.160623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'5877","reported_seq":5900,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.800435+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:23:41.800435+0000","last_peered":"2024-05-09T05:23:41.800435+0000","last_clean":"2024-05-09T05:23:41.800435+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:23:41.800435+0000","last_undegraded":"2024-05-09T05:23:41.800435+0000","last_fullsized":"2024-05-09T05:23:41.800435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29217,"num_write_kb":334206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"23'21","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.141090+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:22:59.141090+0000","last_peered":"2024-05-09T05:22:59.141090+0000","last_clean":"2024-05-09T05:22:59.141090+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:22:59.141090+0000","last_undegraded":"2024-05-09T05:22:59.141090+0000","last_fullsized":"2024-05-09T05:22:59.141090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'5443","reported_seq":5466,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.843005+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:23:41.843005+0000","last_peered":"2024-05-09T05:23:41.843005+0000","last_clean":"2024-05-09T05:23:41.843005+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:23:41.843005+0000","last_undegraded":"2024-05-09T05:23:41.843005+0000","last_fullsized":"2024-05-09T05:23:41.843005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27739,"num_write_kb":311936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.151151+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:22:59.151151+0000","last_peered":"2024-05-09T05:22:59.151151+0000","last_clean":"2024-05-09T05:22:59.151151+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:22:59.151151+0000","last_undegraded":"2024-05-09T05:22:59.151151+0000","last_fullsized":"2024-05-09T05:22:59.151151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'5295","reported_seq":5318,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.754503+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:23:41.754503+0000","last_peered":"2024-05-09T05:23:41.754503+0000","last_clean":"2024-05-09T05:23:41.754503+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:23:41.754503+0000","last_undegraded":"2024-05-09T05:23:41.754503+0000","last_fullsized":"2024-05-09T05:23:41.754503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25732,"num_write_kb":299773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'9","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.132085+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:22:59.132085+0000","last_peered":"2024-05-09T05:22:59.132085+0000","last_clean":"2024-05-09T05:22:59.132085+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:22:59.132085+0000","last_undegraded":"2024-05-09T05:22:59.132085+0000","last_fullsized":"2024-05-09T05:22:59.132085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'6402","reported_seq":6425,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.913813+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:23:41.913813+0000","last_peered":"2024-05-09T05:23:41.913813+0000","last_clean":"2024-05-09T05:23:41.913813+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:23:41.913813+0000","last_undegraded":"2024-05-09T05:23:41.913813+0000","last_fullsized":"2024-05-09T05:23:41.913813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31156,"num_write_kb":362485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:21:51.575458+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:21:51.575458+0000","last_peered":"2024-05-09T05:21:51.575458+0000","last_clean":"2024-05-09T05:21:51.575458+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:21:51.575458+0000","last_undegraded":"2024-05-09T05:21:51.575458+0000","last_fullsized":"2024-05-09T05:21:51.575458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'5559","reported_seq":5582,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.794865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:23:41.794865+0000","last_peered":"2024-05-09T05:23:41.794865+0000","last_clean":"2024-05-09T05:23:41.794865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:23:41.794865+0000","last_undegraded":"2024-05-09T05:23:41.794865+0000","last_fullsized":"2024-05-09T05:23:41.794865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28130,"num_write_kb":325652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'10","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.117825+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:22:59.117825+0000","last_peered":"2024-05-09T05:22:59.117825+0000","last_clean":"2024-05-09T05:22:59.117825+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:22:59.117825+0000","last_undegraded":"2024-05-09T05:22:59.117825+0000","last_fullsized":"2024-05-09T05:22:59.117825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'5794","reported_seq":5817,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.896072+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:23:41.896072+0000","last_peered":"2024-05-09T05:23:41.896072+0000","last_clean":"2024-05-09T05:23:41.896072+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:23:41.896072+0000","last_undegraded":"2024-05-09T05:23:41.896072+0000","last_fullsized":"2024-05-09T05:23:41.896072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28847,"num_write_kb":320571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":122,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:19:52.665112+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:19:52.665112+0000","last_peered":"2024-05-09T05:19:52.665112+0000","last_clean":"2024-05-09T05:19:52.665112+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:19:52.665112+0000","last_undegraded":"2024-05-09T05:19:52.665112+0000","last_fullsized":"2024-05-09T05:19:52.665112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'7","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:20:42.700037+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:20:42.700037+0000","last_peered":"2024-05-09T05:20:42.700037+0000","last_clean":"2024-05-09T05:20:42.700037+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:20:42.700037+0000","last_undegraded":"2024-05-09T05:20:42.700037+0000","last_fullsized":"2024-05-09T05:20:42.700037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'5510","reported_seq":5534,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.818585+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:23:41.818585+0000","last_peered":"2024-05-09T05:23:41.818585+0000","last_clean":"2024-05-09T05:23:41.818585+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:23:41.818585+0000","last_undegraded":"2024-05-09T05:23:41.818585+0000","last_fullsized":"2024-05-09T05:23:41.818585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5510,"log_dups_size":0,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27376,"num_write_kb":322728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'9","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.151152+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:22:59.151152+0000","last_peered":"2024-05-09T05:22:59.151152+0000","last_clean":"2024-05-09T05:22:59.151152+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:22:59.151152+0000","last_undegraded":"2024-05-09T05:22:59.151152+0000","last_fullsized":"2024-05-09T05:22:59.151152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'5383","reported_seq":5407,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.820369+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:23:41.820369+0000","last_peered":"2024-05-09T05:23:41.820369+0000","last_clean":"2024-05-09T05:23:41.820369+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:23:41.820369+0000","last_undegraded":"2024-05-09T05:23:41.820369+0000","last_fullsized":"2024-05-09T05:23:41.820369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:23.362012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27075,"num_write_kb":305200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'45","reported_seq":107,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.161683+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:22:59.161683+0000","last_peered":"2024-05-09T05:22:59.161683+0000","last_clean":"2024-05-09T05:22:59.161683+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:22:59.161683+0000","last_undegraded":"2024-05-09T05:22:59.161683+0000","last_fullsized":"2024-05-09T05:22:59.161683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88293,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":43,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'5441","reported_seq":5465,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.775191+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:23:41.775191+0000","last_peered":"2024-05-09T05:23:41.775191+0000","last_clean":"2024-05-09T05:23:41.775191+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:23:41.775191+0000","last_undegraded":"2024-05-09T05:23:41.775191+0000","last_fullsized":"2024-05-09T05:23:41.775191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27385,"num_write_kb":302253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.133458+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:22:59.133458+0000","last_peered":"2024-05-09T05:22:59.133458+0000","last_clean":"2024-05-09T05:22:59.133458+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:22:59.133458+0000","last_undegraded":"2024-05-09T05:22:59.133458+0000","last_fullsized":"2024-05-09T05:22:59.133458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'5472","reported_seq":5496,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.928867+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:23:41.928867+0000","last_peered":"2024-05-09T05:23:41.928867+0000","last_clean":"2024-05-09T05:23:41.928867+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:23:41.928867+0000","last_undegraded":"2024-05-09T05:23:41.928867+0000","last_fullsized":"2024-05-09T05:23:41.928867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28049,"num_write_kb":304974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'6","reported_seq":32,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:55.531860+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:22:55.531860+0000","last_peered":"2024-05-09T05:22:55.531860+0000","last_clean":"2024-05-09T05:22:55.531860+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:22:55.531860+0000","last_undegraded":"2024-05-09T05:22:55.531860+0000","last_fullsized":"2024-05-09T05:22:55.531860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'4678","reported_seq":4702,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.919111+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:23:41.919111+0000","last_peered":"2024-05-09T05:23:41.919111+0000","last_clean":"2024-05-09T05:23:41.919111+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:23:41.919111+0000","last_undegraded":"2024-05-09T05:23:41.919111+0000","last_fullsized":"2024-05-09T05:23:41.919111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23621,"num_write_kb":274908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"23'8","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.841082+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:22:54.841082+0000","last_peered":"2024-05-09T05:22:54.841082+0000","last_clean":"2024-05-09T05:22:54.841082+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:22:54.841082+0000","last_undegraded":"2024-05-09T05:22:54.841082+0000","last_fullsized":"2024-05-09T05:22:54.841082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'5941","reported_seq":5965,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.780983+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:23:41.780983+0000","last_peered":"2024-05-09T05:23:41.780983+0000","last_clean":"2024-05-09T05:23:41.780983+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:23:41.780983+0000","last_undegraded":"2024-05-09T05:23:41.780983+0000","last_fullsized":"2024-05-09T05:23:41.780983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":328170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'609","reported_seq":635,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:00.281854+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:23:00.281854+0000","last_peered":"2024-05-09T05:23:00.281854+0000","last_clean":"2024-05-09T05:23:00.281854+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:23:00.281854+0000","last_undegraded":"2024-05-09T05:23:00.281854+0000","last_fullsized":"2024-05-09T05:23:00.281854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'5503","reported_seq":5527,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.773405+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:23:41.773405+0000","last_peered":"2024-05-09T05:23:41.773405+0000","last_clean":"2024-05-09T05:23:41.773405+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:23:41.773405+0000","last_undegraded":"2024-05-09T05:23:41.773405+0000","last_fullsized":"2024-05-09T05:23:41.773405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:33.840344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26876,"num_write_kb":308483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.130335+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:22:59.130335+0000","last_peered":"2024-05-09T05:22:59.130335+0000","last_clean":"2024-05-09T05:22:59.130335+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:22:59.130335+0000","last_undegraded":"2024-05-09T05:22:59.130335+0000","last_fullsized":"2024-05-09T05:22:59.130335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'4642","reported_seq":4666,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.787942+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:23:41.787942+0000","last_peered":"2024-05-09T05:23:41.787942+0000","last_clean":"2024-05-09T05:23:41.787942+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:23:41.787942+0000","last_undegraded":"2024-05-09T05:23:41.787942+0000","last_fullsized":"2024-05-09T05:23:41.787942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:50.236344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22705,"num_write_kb":262889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'5","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.155406+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:22:59.155406+0000","last_peered":"2024-05-09T05:22:59.155406+0000","last_clean":"2024-05-09T05:22:59.155406+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:22:59.155406+0000","last_undegraded":"2024-05-09T05:22:59.155406+0000","last_fullsized":"2024-05-09T05:22:59.155406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'5215","reported_seq":5239,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790504+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:23:41.790504+0000","last_peered":"2024-05-09T05:23:41.790504+0000","last_clean":"2024-05-09T05:23:41.790504+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:23:41.790504+0000","last_undegraded":"2024-05-09T05:23:41.790504+0000","last_fullsized":"2024-05-09T05:23:41.790504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25185,"num_write_kb":293431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.138117+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:22:59.138117+0000","last_peered":"2024-05-09T05:22:59.138117+0000","last_clean":"2024-05-09T05:22:59.138117+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:22:59.138117+0000","last_undegraded":"2024-05-09T05:22:59.138117+0000","last_fullsized":"2024-05-09T05:22:59.138117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'5538","reported_seq":5562,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803051+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:23:41.803051+0000","last_peered":"2024-05-09T05:23:41.803051+0000","last_clean":"2024-05-09T05:23:41.803051+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:23:41.803051+0000","last_undegraded":"2024-05-09T05:23:41.803051+0000","last_fullsized":"2024-05-09T05:23:41.803051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27899,"num_write_kb":316727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"23'6","reported_seq":32,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.135039+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:22:59.135039+0000","last_peered":"2024-05-09T05:22:59.135039+0000","last_clean":"2024-05-09T05:22:59.135039+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:22:59.135039+0000","last_undegraded":"2024-05-09T05:22:59.135039+0000","last_fullsized":"2024-05-09T05:22:59.135039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'5078","reported_seq":5101,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.813122+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:23:41.813122+0000","last_peered":"2024-05-09T05:23:41.813122+0000","last_clean":"2024-05-09T05:23:41.813122+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:23:41.813122+0000","last_undegraded":"2024-05-09T05:23:41.813122+0000","last_fullsized":"2024-05-09T05:23:41.813122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:58:35.651549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24494,"num_write_kb":270229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1097","reported_seq":1122,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.133596+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:22:59.133596+0000","last_peered":"2024-05-09T05:22:59.133596+0000","last_clean":"2024-05-09T05:22:59.133596+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:22:59.133596+0000","last_undegraded":"2024-05-09T05:22:59.133596+0000","last_fullsized":"2024-05-09T05:22:59.133596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1089,"num_write_kb":4419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'4956","reported_seq":4980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.764163+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:23:41.764163+0000","last_peered":"2024-05-09T05:23:41.764163+0000","last_clean":"2024-05-09T05:23:41.764163+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:23:41.764163+0000","last_undegraded":"2024-05-09T05:23:41.764163+0000","last_fullsized":"2024-05-09T05:23:41.764163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24725,"num_write_kb":283470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'6745","reported_seq":6768,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.926010+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:23:41.926010+0000","last_peered":"2024-05-09T05:23:41.926010+0000","last_clean":"2024-05-09T05:23:41.926010+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:23:41.926010+0000","last_undegraded":"2024-05-09T05:23:41.926010+0000","last_fullsized":"2024-05-09T05:23:41.926010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33228,"num_write_kb":379009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.163507+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:22:59.163507+0000","last_peered":"2024-05-09T05:22:59.163507+0000","last_clean":"2024-05-09T05:22:59.163507+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:22:59.163507+0000","last_undegraded":"2024-05-09T05:22:59.163507+0000","last_fullsized":"2024-05-09T05:22:59.163507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'4185","reported_seq":4201,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.772240+0000","last_change":"2024-05-09T05:07:01.723135+0000","last_active":"2024-05-09T05:23:41.772240+0000","last_peered":"2024-05-09T05:23:41.772240+0000","last_clean":"2024-05-09T05:23:41.772240+0000","last_became_active":"2024-05-09T05:07:01.722785+0000","last_became_peered":"2024-05-09T05:07:01.722785+0000","last_unstale":"2024-05-09T05:23:41.772240+0000","last_undegraded":"2024-05-09T05:23:41.772240+0000","last_fullsized":"2024-05-09T05:23:41.772240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:57:53.799866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20471,"num_write_kb":229445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'21","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.166888+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:22:59.166888+0000","last_peered":"2024-05-09T05:22:59.166888+0000","last_clean":"2024-05-09T05:22:59.166888+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:22:59.166888+0000","last_undegraded":"2024-05-09T05:22:59.166888+0000","last_fullsized":"2024-05-09T05:22:59.166888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'6872","reported_seq":6896,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.883263+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:23:41.883263+0000","last_peered":"2024-05-09T05:23:41.883263+0000","last_clean":"2024-05-09T05:23:41.883263+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:23:41.883263+0000","last_undegraded":"2024-05-09T05:23:41.883263+0000","last_fullsized":"2024-05-09T05:23:41.883263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:42.457145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34844,"num_write_kb":385600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'9","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.127515+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:22:59.127515+0000","last_peered":"2024-05-09T05:22:59.127515+0000","last_clean":"2024-05-09T05:22:59.127515+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:22:59.127515+0000","last_undegraded":"2024-05-09T05:22:59.127515+0000","last_fullsized":"2024-05-09T05:22:59.127515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'5811","reported_seq":5835,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790466+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:23:41.790466+0000","last_peered":"2024-05-09T05:23:41.790466+0000","last_clean":"2024-05-09T05:23:41.790466+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:23:41.790466+0000","last_undegraded":"2024-05-09T05:23:41.790466+0000","last_fullsized":"2024-05-09T05:23:41.790466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:32:29.331565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29692,"num_write_kb":336226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'740","reported_seq":766,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:43.840338+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:22:43.840338+0000","last_peered":"2024-05-09T05:22:43.840338+0000","last_clean":"2024-05-09T05:22:43.840338+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:22:43.840338+0000","last_undegraded":"2024-05-09T05:22:43.840338+0000","last_fullsized":"2024-05-09T05:22:43.840338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":739,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1808567,"num_write_kb":20359177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":362904,"ondisk_log_size":362904,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18590025,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":4539,"num_write_kb":19914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37421056,"data_stored":37180050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3759,"ondisk_log_size":3759,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509657,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729492,"kb_used_data":13132,"kb_used_omap":1,"kb_used_meta":716350,"kb_avail":93642348,"statfs":{"total":96636764160,"available":95889764352,"internally_reserved":0,"allocated":13447168,"data_stored":13261681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":733542858},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509658,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702748,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":702462,"kb_avail":93669092,"statfs":{"total":96636764160,"available":95917150208,"internally_reserved":0,"allocated":282624,"data_stored":90233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719321541},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542362,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384208,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":378174,"kb_avail":93987632,"statfs":{"total":96636764160,"available":96243335168,"internally_reserved":0,"allocated":6168576,"data_stored":5997855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387250630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542363,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698768,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":692734,"kb_avail":93673072,"statfs":{"total":96636764160,"available":95921225728,"internally_reserved":0,"allocated":6168576,"data_stored":5997723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":709360069},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542363,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679136,"kb_used_data":8536,"kb_used_omap":1,"kb_used_meta":670590,"kb_avail":93692704,"statfs":{"total":96636764160,"available":95941328896,"internally_reserved":0,"allocated":8740864,"data_stored":8564586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":686684616},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542364,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400676,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":400318,"kb_avail":93971164,"statfs":{"total":96636764160,"available":96226471936,"internally_reserved":0,"allocated":356352,"data_stored":175944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":409926085},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542363,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":446014,"kb_avail":93925560,"statfs":{"total":96636764160,"available":96179773440,"internally_reserved":0,"allocated":262144,"data_stored":87251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456718789},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542362,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675468,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":670526,"kb_avail":93696372,"statfs":{"total":96636764160,"available":95945084928,"internally_reserved":0,"allocated":5050368,"data_stored":4871265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":686619077},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4222976,"data_stored":4195178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":90225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8507392,"data_stored":8478867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12619776,"data_stored":12585594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:23:48.372 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:23:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.526+0000 7f18d3296640 1 Processor -- start 2024-05-09T05:23:48.523 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.526+0000 7f18d3296640 1 -- start start 2024-05-09T05:23:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d3296640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18cc0c64b0 0x7f18cc156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d3296640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f18cc06e1f0 0x7f18cc068650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.525 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d3296640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f18cc0c6bc0 con 0x7f18cc06e600 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d3296640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f18cc15c8e0 con 0x7f18cc0c64b0 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d3296640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f18cc06eaf0 con 0x7f18cc06e1f0 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d1032640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18cc0c64b0 0x7f18cc156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d1833640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f18cc06e600 0x7f18cc070bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58800/0 (socket says 172.21.15.1:58800) 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d1032640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f18cc0c64b0 0x7f18cc156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60292/0 (socket says 172.21.15.1:60292) 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d1833640 1 -- 172.21.15.1:0/1856999969 learned_addr learned my addr 172.21.15.1:0/1856999969 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d0831640 1 --2- 172.21.15.1:0/1856999969 >> v2:172.21.15.196:3300/0 conn(0x7f18cc06e1f0 0x7f18cc068650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d1032640 1 -- 172.21.15.1:0/1856999969 >> v2:172.21.15.196:3300/0 conn(0x7f18cc06e1f0 msgr2=0x7f18cc068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d1032640 1 --2- 172.21.15.1:0/1856999969 >> v2:172.21.15.196:3300/0 conn(0x7f18cc06e1f0 0x7f18cc068650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d1032640 1 -- 172.21.15.1:0/1856999969 >> v1:172.21.15.1:6789/0 conn(0x7f18cc06e600 legacy=0x7f18cc070bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.526 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.528+0000 7f18d1032640 1 -- 172.21.15.1:0/1856999969 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18cc15cba0 con 0x7f18cc0c64b0 2024-05-09T05:23:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.529+0000 7f18d1032640 1 --2- 172.21.15.1:0/1856999969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc156c90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f18cc069350 tx=0x7f18bc02a1a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d349801dafe5cc8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.529+0000 7f18cbfff640 1 -- 172.21.15.1:0/1856999969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc037070 con 0x7f18cc0c64b0 2024-05-09T05:23:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.529+0000 7f18cbfff640 1 -- 172.21.15.1:0/1856999969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18bc032440 con 0x7f18cc0c64b0 2024-05-09T05:23:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.529+0000 7f18cbfff640 1 -- 172.21.15.1:0/1856999969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18bc032810 con 0x7f18cc0c64b0 2024-05-09T05:23:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.530+0000 7f18d3296640 1 -- 172.21.15.1:0/1856999969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 msgr2=0x7f18cc156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.530+0000 7f18d3296640 1 --2- 172.21.15.1:0/1856999969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc156c90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f18cc069350 tx=0x7f18bc02a1a0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.530+0000 7f18d3296640 1 -- 172.21.15.1:0/1856999969 shutdown_connections 2024-05-09T05:23:48.528 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.530+0000 7f18d3296640 1 --2- 172.21.15.1:0/1856999969 >> v2:172.21.15.196:3300/0 conn(0x7f18cc06e1f0 0x7f18cc068650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.530+0000 7f18d3296640 1 --2- 172.21.15.1:0/1856999969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc156c90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.530+0000 7f18d3296640 1 -- 172.21.15.1:0/1856999969 >> 172.21.15.1:0/1856999969 conn(0x7f18cc0cc6a0 msgr2=0x7f18cc0ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.530+0000 7f18d3296640 1 -- 172.21.15.1:0/1856999969 shutdown_connections 2024-05-09T05:23:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.530+0000 7f18d3296640 1 -- 172.21.15.1:0/1856999969 wait complete. 2024-05-09T05:23:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.532+0000 7f18d3296640 1 Processor -- start 2024-05-09T05:23:48.529 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.532+0000 7f18d3296640 1 -- start start 2024-05-09T05:23:48.531 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d3296640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18cc06e1f0 0x7f18cc0c8bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d3296640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18cc06e600 0x7f18cc1502f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d3296640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc150830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d3296640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc15d440 con 0x7f18cc0c64b0 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d3296640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f18cc15d160 con 0x7f18cc06e600 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d3296640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f18cc0712e0 con 0x7f18cc06e1f0 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1833640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc150830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1833640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc150830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60302/0 (socket says 172.21.15.1:60302) 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1833640 1 -- 172.21.15.1:0/674175022 learned_addr learned my addr 172.21.15.1:0/674175022 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d0831640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18cc06e600 0x7f18cc1502f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1032640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18cc06e1f0 0x7f18cc0c8bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1833640 1 -- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18cc06e1f0 msgr2=0x7f18cc0c8bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1833640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18cc06e1f0 0x7f18cc0c8bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.532 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1833640 1 -- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18cc06e600 msgr2=0x7f18cc1502f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1833640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18cc06e600 0x7f18cc1502f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.533+0000 7f18d1833640 1 -- 172.21.15.1:0/674175022 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18cc0c71e0 con 0x7f18cc0c64b0 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.534+0000 7f18d1833640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc150830 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18c400cb10 tx=0x7f18c400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.534+0000 7f18c9ffb640 1 -- 172.21.15.1:0/674175022 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4017070 con 0x7f18cc0c64b0 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.534+0000 7f18c9ffb640 1 -- 172.21.15.1:0/674175022 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f18c400abb0 con 0x7f18cc0c64b0 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.534+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18cc1f2e90 con 0x7f18cc0c64b0 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.534+0000 7f18c9ffb640 1 -- 172.21.15.1:0/674175022 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18c4012400 con 0x7f18cc0c64b0 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.534+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18cc0b2610 con 0x7f18cc0c64b0 2024-05-09T05:23:48.533 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.535+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f18cc1f2c80 con 0x7f18cc0c64b0 2024-05-09T05:23:48.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.536+0000 7f18c9ffb640 1 -- 172.21.15.1:0/674175022 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f18c401a070 con 0x7f18cc0c64b0 2024-05-09T05:23:48.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.537+0000 7f18c9ffb640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18b8076300 0x7f18b8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.537+0000 7f18c9ffb640 1 -- 172.21.15.1:0/674175022 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f18c40979e0 con 0x7f18cc0c64b0 2024-05-09T05:23:48.534 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.537+0000 7f18d1032640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18b8076300 0x7f18b8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.535 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.538+0000 7f18d1032640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18b8076300 0x7f18b8078810 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f18bc034000 tx=0x7f18bc02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.538 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.541+0000 7f18c9ffb640 1 -- 172.21.15.1:0/674175022 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f18c4060cc0 con 0x7f18cc0c64b0 2024-05-09T05:23:48.725 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.728+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f18cc0c6f40 con 0x7f18b8076300 2024-05-09T05:23:48.732 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.735+0000 7f18c9ffb640 1 -- 172.21.15.1:0/674175022 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253608 (secure 0 0 0) 0x7f18cc0c6f40 con 0x7f18b8076300 2024-05-09T05:23:48.735 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:23:48.736 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:23:48.737 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18b8076300 msgr2=0x7f18b8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18b8076300 0x7f18b8078810 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f18bc034000 tx=0x7f18bc02e000 comp rx=0 tx=0).stop 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 msgr2=0x7f18cc150830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc150830 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f18c400cb10 tx=0x7f18c400a060 comp rx=0 tx=0).stop 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 shutdown_connections 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f18b8076300 0x7f18b8078810 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f18cc06e600 0x7f18cc1502f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f18cc0c64b0 0x7f18cc150830 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 --2- 172.21.15.1:0/674175022 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f18cc06e1f0 0x7f18cc0c8bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.738 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 >> 172.21.15.1:0/674175022 conn(0x7f18cc0cc6a0 msgr2=0x7f18cc151930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 shutdown_connections 2024-05-09T05:23:48.739 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.738+0000 7f18d3296640 1 -- 172.21.15.1:0/674175022 wait complete. 2024-05-09T05:23:48.747 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-05-09T05:23:47.833121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19180393,"num_objects":105,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":102,"num_read_kb":167,"num_write":1813207,"num_write_kb":20380719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366735,"ondisk_log_size":366735,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4716776,"kb_used_data":39528,"kb_used_omap":12,"kb_used_meta":4677171,"kb_avail":750257944,"statfs":{"total":773094113280,"available":768264134656,"internally_reserved":0,"allocated":40476672,"data_stored":39046538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4789423665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003605"},"pg_stats":[{"pgid":"3.3f","version":"23'5187","reported_seq":5211,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.889349+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:23:41.889349+0000","last_peered":"2024-05-09T05:23:41.889349+0000","last_clean":"2024-05-09T05:23:41.889349+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:23:41.889349+0000","last_undegraded":"2024-05-09T05:23:41.889349+0000","last_fullsized":"2024-05-09T05:23:41.889349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26135,"num_write_kb":280897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'6912","reported_seq":6936,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.786450+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:23:41.786450+0000","last_peered":"2024-05-09T05:23:41.786450+0000","last_clean":"2024-05-09T05:23:41.786450+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:23:41.786450+0000","last_undegraded":"2024-05-09T05:23:41.786450+0000","last_fullsized":"2024-05-09T05:23:41.786450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35479,"num_write_kb":400336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'5571","reported_seq":5595,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.785914+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:23:41.785914+0000","last_peered":"2024-05-09T05:23:41.785914+0000","last_clean":"2024-05-09T05:23:41.785914+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:23:41.785914+0000","last_undegraded":"2024-05-09T05:23:41.785914+0000","last_fullsized":"2024-05-09T05:23:41.785914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:48.770365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27199,"num_write_kb":317974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'5494","reported_seq":5518,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.804206+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:23:41.804206+0000","last_peered":"2024-05-09T05:23:41.804206+0000","last_clean":"2024-05-09T05:23:41.804206+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:23:41.804206+0000","last_undegraded":"2024-05-09T05:23:41.804206+0000","last_fullsized":"2024-05-09T05:23:41.804206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28054,"num_write_kb":296869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'7058","reported_seq":7081,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790501+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:23:41.790501+0000","last_peered":"2024-05-09T05:23:41.790501+0000","last_clean":"2024-05-09T05:23:41.790501+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:23:41.790501+0000","last_undegraded":"2024-05-09T05:23:41.790501+0000","last_fullsized":"2024-05-09T05:23:41.790501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34281,"num_write_kb":379027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'4948","reported_seq":4971,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.889531+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:23:41.889531+0000","last_peered":"2024-05-09T05:23:41.889531+0000","last_clean":"2024-05-09T05:23:41.889531+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:23:41.889531+0000","last_undegraded":"2024-05-09T05:23:41.889531+0000","last_fullsized":"2024-05-09T05:23:41.889531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24780,"num_write_kb":275405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'5454","reported_seq":5477,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.811426+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:23:41.811426+0000","last_peered":"2024-05-09T05:23:41.811426+0000","last_clean":"2024-05-09T05:23:41.811426+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:23:41.811426+0000","last_undegraded":"2024-05-09T05:23:41.811426+0000","last_fullsized":"2024-05-09T05:23:41.811426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27432,"num_write_kb":306787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'6053","reported_seq":6076,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.840232+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:23:41.840232+0000","last_peered":"2024-05-09T05:23:41.840232+0000","last_clean":"2024-05-09T05:23:41.840232+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:23:41.840232+0000","last_undegraded":"2024-05-09T05:23:41.840232+0000","last_fullsized":"2024-05-09T05:23:41.840232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6053,"log_dups_size":0,"ondisk_log_size":6053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29445,"num_write_kb":339212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"23'7129","reported_seq":7152,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.913522+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:23:41.913522+0000","last_peered":"2024-05-09T05:23:41.913522+0000","last_clean":"2024-05-09T05:23:41.913522+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:23:41.913522+0000","last_undegraded":"2024-05-09T05:23:41.913522+0000","last_fullsized":"2024-05-09T05:23:41.913522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36611,"num_write_kb":392539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'6358","reported_seq":6381,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.896558+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:23:41.896558+0000","last_peered":"2024-05-09T05:23:41.896558+0000","last_clean":"2024-05-09T05:23:41.896558+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:23:41.896558+0000","last_undegraded":"2024-05-09T05:23:41.896558+0000","last_fullsized":"2024-05-09T05:23:41.896558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32647,"num_write_kb":360281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'4747","reported_seq":4770,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.882769+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:23:41.882769+0000","last_peered":"2024-05-09T05:23:41.882769+0000","last_clean":"2024-05-09T05:23:41.882769+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:23:41.882769+0000","last_undegraded":"2024-05-09T05:23:41.882769+0000","last_fullsized":"2024-05-09T05:23:41.882769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22850,"num_write_kb":288147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'4814","reported_seq":4837,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.916891+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:23:41.916891+0000","last_peered":"2024-05-09T05:23:41.916891+0000","last_clean":"2024-05-09T05:23:41.916891+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:23:41.916891+0000","last_undegraded":"2024-05-09T05:23:41.916891+0000","last_fullsized":"2024-05-09T05:23:41.916891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":267591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'5105","reported_seq":5128,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.762396+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:23:41.762396+0000","last_peered":"2024-05-09T05:23:41.762396+0000","last_clean":"2024-05-09T05:23:41.762396+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:23:41.762396+0000","last_undegraded":"2024-05-09T05:23:41.762396+0000","last_fullsized":"2024-05-09T05:23:41.762396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25963,"num_write_kb":288363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'5957","reported_seq":5980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.819716+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:23:41.819716+0000","last_peered":"2024-05-09T05:23:41.819716+0000","last_clean":"2024-05-09T05:23:41.819716+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:23:41.819716+0000","last_undegraded":"2024-05-09T05:23:41.819716+0000","last_fullsized":"2024-05-09T05:23:41.819716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:37.258923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29976,"num_write_kb":329157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'5739","reported_seq":5762,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.843331+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:23:41.843331+0000","last_peered":"2024-05-09T05:23:41.843331+0000","last_clean":"2024-05-09T05:23:41.843331+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:23:41.843331+0000","last_undegraded":"2024-05-09T05:23:41.843331+0000","last_fullsized":"2024-05-09T05:23:41.843331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28226,"num_write_kb":322058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'5341","reported_seq":5364,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.852751+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:23:41.852751+0000","last_peered":"2024-05-09T05:23:41.852751+0000","last_clean":"2024-05-09T05:23:41.852751+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:23:41.852751+0000","last_undegraded":"2024-05-09T05:23:41.852751+0000","last_fullsized":"2024-05-09T05:23:41.852751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27307,"num_write_kb":289793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"23'6965","reported_seq":6982,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803206+0000","last_change":"2024-05-09T05:07:01.750820+0000","last_active":"2024-05-09T05:23:41.803206+0000","last_peered":"2024-05-09T05:23:41.803206+0000","last_clean":"2024-05-09T05:23:41.803206+0000","last_became_active":"2024-05-09T05:07:01.750492+0000","last_became_peered":"2024-05-09T05:07:01.750492+0000","last_unstale":"2024-05-09T05:23:41.803206+0000","last_undegraded":"2024-05-09T05:23:41.803206+0000","last_fullsized":"2024-05-09T05:23:41.803206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6965,"log_dups_size":0,"ondisk_log_size":6965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:23.141520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34877,"num_write_kb":385595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"23'4549","reported_seq":4572,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.791997+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:23:41.791997+0000","last_peered":"2024-05-09T05:23:41.791997+0000","last_clean":"2024-05-09T05:23:41.791997+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:23:41.791997+0000","last_undegraded":"2024-05-09T05:23:41.791997+0000","last_fullsized":"2024-05-09T05:23:41.791997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22029,"num_write_kb":252128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'5335","reported_seq":5358,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.781003+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:23:41.781003+0000","last_peered":"2024-05-09T05:23:41.781003+0000","last_clean":"2024-05-09T05:23:41.781003+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:23:41.781003+0000","last_undegraded":"2024-05-09T05:23:41.781003+0000","last_fullsized":"2024-05-09T05:23:41.781003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26106,"num_write_kb":309468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'5370","reported_seq":5393,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.861348+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:23:41.861348+0000","last_peered":"2024-05-09T05:23:41.861348+0000","last_clean":"2024-05-09T05:23:41.861348+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:23:41.861348+0000","last_undegraded":"2024-05-09T05:23:41.861348+0000","last_fullsized":"2024-05-09T05:23:41.861348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26990,"num_write_kb":308937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'5403","reported_seq":5426,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.859039+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:23:41.859039+0000","last_peered":"2024-05-09T05:23:41.859039+0000","last_clean":"2024-05-09T05:23:41.859039+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:23:41.859039+0000","last_undegraded":"2024-05-09T05:23:41.859039+0000","last_fullsized":"2024-05-09T05:23:41.859039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:02.665344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26698,"num_write_kb":315887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'6082","reported_seq":6105,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.904062+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:23:41.904062+0000","last_peered":"2024-05-09T05:23:41.904062+0000","last_clean":"2024-05-09T05:23:41.904062+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:23:41.904062+0000","last_undegraded":"2024-05-09T05:23:41.904062+0000","last_fullsized":"2024-05-09T05:23:41.904062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30462,"num_write_kb":343332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"23'5512","reported_seq":5535,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.774669+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:23:41.774669+0000","last_peered":"2024-05-09T05:23:41.774669+0000","last_clean":"2024-05-09T05:23:41.774669+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:23:41.774669+0000","last_undegraded":"2024-05-09T05:23:41.774669+0000","last_fullsized":"2024-05-09T05:23:41.774669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27471,"num_write_kb":310767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'5065","reported_seq":5089,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.839542+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:23:41.839542+0000","last_peered":"2024-05-09T05:23:41.839542+0000","last_clean":"2024-05-09T05:23:41.839542+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:23:41.839542+0000","last_undegraded":"2024-05-09T05:23:41.839542+0000","last_fullsized":"2024-05-09T05:23:41.839542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24983,"num_write_kb":280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'6632","reported_seq":6656,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.735021+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:23:41.735021+0000","last_peered":"2024-05-09T05:23:41.735021+0000","last_clean":"2024-05-09T05:23:41.735021+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:23:41.735021+0000","last_undegraded":"2024-05-09T05:23:41.735021+0000","last_fullsized":"2024-05-09T05:23:41.735021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:25:28.166958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33114,"num_write_kb":370371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'5121","reported_seq":5150,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.872969+0000","last_change":"2024-05-09T05:07:01.662425+0000","last_active":"2024-05-09T05:23:41.872969+0000","last_peered":"2024-05-09T05:23:41.872969+0000","last_clean":"2024-05-09T05:23:41.872969+0000","last_became_active":"2024-05-09T05:07:01.662096+0000","last_became_peered":"2024-05-09T05:07:01.662096+0000","last_unstale":"2024-05-09T05:23:41.872969+0000","last_undegraded":"2024-05-09T05:23:41.872969+0000","last_fullsized":"2024-05-09T05:23:41.872969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:50.949207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25567,"num_write_kb":287181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'6014","reported_seq":6038,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.767615+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:23:41.767615+0000","last_peered":"2024-05-09T05:23:41.767615+0000","last_clean":"2024-05-09T05:23:41.767615+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:23:41.767615+0000","last_undegraded":"2024-05-09T05:23:41.767615+0000","last_fullsized":"2024-05-09T05:23:41.767615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30657,"num_write_kb":340145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'5132","reported_seq":5149,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.864977+0000","last_change":"2024-05-09T05:07:01.723062+0000","last_active":"2024-05-09T05:23:41.864977+0000","last_peered":"2024-05-09T05:23:41.864977+0000","last_clean":"2024-05-09T05:23:41.864977+0000","last_became_active":"2024-05-09T05:07:01.722736+0000","last_became_peered":"2024-05-09T05:07:01.722736+0000","last_unstale":"2024-05-09T05:23:41.864977+0000","last_undegraded":"2024-05-09T05:23:41.864977+0000","last_fullsized":"2024-05-09T05:23:41.864977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:50.949207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25502,"num_write_kb":286500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"23'4664","reported_seq":4688,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.846625+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:23:41.846625+0000","last_peered":"2024-05-09T05:23:41.846625+0000","last_clean":"2024-05-09T05:23:41.846625+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:23:41.846625+0000","last_undegraded":"2024-05-09T05:23:41.846625+0000","last_fullsized":"2024-05-09T05:23:41.846625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23461,"num_write_kb":270138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'6545","reported_seq":6569,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.857347+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:23:41.857347+0000","last_peered":"2024-05-09T05:23:41.857347+0000","last_clean":"2024-05-09T05:23:41.857347+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:23:41.857347+0000","last_undegraded":"2024-05-09T05:23:41.857347+0000","last_fullsized":"2024-05-09T05:23:41.857347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6545,"log_dups_size":0,"ondisk_log_size":6545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32981,"num_write_kb":377918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'6387","reported_seq":6411,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.809273+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:23:41.809273+0000","last_peered":"2024-05-09T05:23:41.809273+0000","last_clean":"2024-05-09T05:23:41.809273+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:23:41.809273+0000","last_undegraded":"2024-05-09T05:23:41.809273+0000","last_fullsized":"2024-05-09T05:23:41.809273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32133,"num_write_kb":346002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'7361","reported_seq":7385,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.851736+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:23:41.851736+0000","last_peered":"2024-05-09T05:23:41.851736+0000","last_clean":"2024-05-09T05:23:41.851736+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:23:41.851736+0000","last_undegraded":"2024-05-09T05:23:41.851736+0000","last_fullsized":"2024-05-09T05:23:41.851736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36142,"num_write_kb":398870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'5228","reported_seq":5251,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803020+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:23:41.803020+0000","last_peered":"2024-05-09T05:23:41.803020+0000","last_clean":"2024-05-09T05:23:41.803020+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:23:41.803020+0000","last_undegraded":"2024-05-09T05:23:41.803020+0000","last_fullsized":"2024-05-09T05:23:41.803020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25537,"num_write_kb":278419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'577","reported_seq":602,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:36.679365+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:23:36.679365+0000","last_peered":"2024-05-09T05:23:36.679365+0000","last_clean":"2024-05-09T05:23:36.679365+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:23:36.679365+0000","last_undegraded":"2024-05-09T05:23:36.679365+0000","last_fullsized":"2024-05-09T05:23:36.679365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":573,"num_write_kb":4218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'4521","reported_seq":4544,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.926128+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:23:41.926128+0000","last_peered":"2024-05-09T05:23:41.926128+0000","last_clean":"2024-05-09T05:23:41.926128+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:23:41.926128+0000","last_undegraded":"2024-05-09T05:23:41.926128+0000","last_fullsized":"2024-05-09T05:23:41.926128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22336,"num_write_kb":249223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'111","reported_seq":136,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.634306+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:23:41.634306+0000","last_peered":"2024-05-09T05:23:41.634306+0000","last_clean":"2024-05-09T05:23:41.634306+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:23:41.634306+0000","last_undegraded":"2024-05-09T05:23:41.634306+0000","last_fullsized":"2024-05-09T05:23:41.634306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1715788,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":1741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.829290+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:22:54.829290+0000","last_peered":"2024-05-09T05:22:54.829290+0000","last_clean":"2024-05-09T05:22:54.829290+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:22:54.829290+0000","last_undegraded":"2024-05-09T05:22:54.829290+0000","last_fullsized":"2024-05-09T05:22:54.829290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'5503","reported_seq":5526,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.842314+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:23:41.842314+0000","last_peered":"2024-05-09T05:23:41.842314+0000","last_clean":"2024-05-09T05:23:41.842314+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:23:41.842314+0000","last_undegraded":"2024-05-09T05:23:41.842314+0000","last_fullsized":"2024-05-09T05:23:41.842314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27748,"num_write_kb":311965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.156887+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:22:59.156887+0000","last_peered":"2024-05-09T05:22:59.156887+0000","last_clean":"2024-05-09T05:22:59.156887+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:22:59.156887+0000","last_undegraded":"2024-05-09T05:22:59.156887+0000","last_fullsized":"2024-05-09T05:22:59.156887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'6912","reported_seq":6935,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.823969+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:23:41.823969+0000","last_peered":"2024-05-09T05:23:41.823969+0000","last_clean":"2024-05-09T05:23:41.823969+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:23:41.823969+0000","last_undegraded":"2024-05-09T05:23:41.823969+0000","last_fullsized":"2024-05-09T05:23:41.823969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34348,"num_write_kb":375438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'9","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.705733+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:23:41.705733+0000","last_peered":"2024-05-09T05:23:41.705733+0000","last_clean":"2024-05-09T05:23:41.705733+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:23:41.705733+0000","last_undegraded":"2024-05-09T05:23:41.705733+0000","last_fullsized":"2024-05-09T05:23:41.705733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'5546","reported_seq":5569,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.742360+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:23:41.742360+0000","last_peered":"2024-05-09T05:23:41.742360+0000","last_clean":"2024-05-09T05:23:41.742360+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:23:41.742360+0000","last_undegraded":"2024-05-09T05:23:41.742360+0000","last_fullsized":"2024-05-09T05:23:41.742360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5546,"log_dups_size":0,"ondisk_log_size":5546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27252,"num_write_kb":314086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.148513+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:22:59.148513+0000","last_peered":"2024-05-09T05:22:59.148513+0000","last_clean":"2024-05-09T05:22:59.148513+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:22:59.148513+0000","last_undegraded":"2024-05-09T05:22:59.148513+0000","last_fullsized":"2024-05-09T05:22:59.148513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'5049","reported_seq":5072,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.871865+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:23:41.871865+0000","last_peered":"2024-05-09T05:23:41.871865+0000","last_clean":"2024-05-09T05:23:41.871865+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:23:41.871865+0000","last_undegraded":"2024-05-09T05:23:41.871865+0000","last_fullsized":"2024-05-09T05:23:41.871865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:26.607004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25089,"num_write_kb":276787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'25","reported_seq":50,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.204360+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:22:59.204360+0000","last_peered":"2024-05-09T05:22:59.204360+0000","last_clean":"2024-05-09T05:22:59.204360+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:22:59.204360+0000","last_undegraded":"2024-05-09T05:22:59.204360+0000","last_fullsized":"2024-05-09T05:22:59.204360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'7381","reported_seq":7404,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.917166+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:23:41.917166+0000","last_peered":"2024-05-09T05:23:41.917166+0000","last_clean":"2024-05-09T05:23:41.917166+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:23:41.917166+0000","last_undegraded":"2024-05-09T05:23:41.917166+0000","last_fullsized":"2024-05-09T05:23:41.917166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7381,"log_dups_size":0,"ondisk_log_size":7381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36744,"num_write_kb":418024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'333","reported_seq":358,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.155427+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:22:59.155427+0000","last_peered":"2024-05-09T05:22:59.155427+0000","last_clean":"2024-05-09T05:22:59.155427+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:22:59.155427+0000","last_undegraded":"2024-05-09T05:22:59.155427+0000","last_fullsized":"2024-05-09T05:22:59.155427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":331,"num_write_kb":4250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'5538","reported_seq":5561,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.787753+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:23:41.787753+0000","last_peered":"2024-05-09T05:23:41.787753+0000","last_clean":"2024-05-09T05:23:41.787753+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:23:41.787753+0000","last_undegraded":"2024-05-09T05:23:41.787753+0000","last_fullsized":"2024-05-09T05:23:41.787753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27557,"num_write_kb":302974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"23'15","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.113005+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:22:59.113005+0000","last_peered":"2024-05-09T05:22:59.113005+0000","last_clean":"2024-05-09T05:22:59.113005+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:22:59.113005+0000","last_undegraded":"2024-05-09T05:22:59.113005+0000","last_fullsized":"2024-05-09T05:22:59.113005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'5579","reported_seq":5602,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.833702+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:23:41.833702+0000","last_peered":"2024-05-09T05:23:41.833702+0000","last_clean":"2024-05-09T05:23:41.833702+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:23:41.833702+0000","last_undegraded":"2024-05-09T05:23:41.833702+0000","last_fullsized":"2024-05-09T05:23:41.833702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:41.283927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27823,"num_write_kb":303973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'10","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.117627+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:22:59.117627+0000","last_peered":"2024-05-09T05:22:59.117627+0000","last_clean":"2024-05-09T05:22:59.117627+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:22:59.117627+0000","last_undegraded":"2024-05-09T05:22:59.117627+0000","last_fullsized":"2024-05-09T05:22:59.117627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'6303","reported_seq":6326,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.912348+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:23:41.912348+0000","last_peered":"2024-05-09T05:23:41.912348+0000","last_clean":"2024-05-09T05:23:41.912348+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:23:41.912348+0000","last_undegraded":"2024-05-09T05:23:41.912348+0000","last_fullsized":"2024-05-09T05:23:41.912348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30554,"num_write_kb":351862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'13","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.160623+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:22:59.160623+0000","last_peered":"2024-05-09T05:22:59.160623+0000","last_clean":"2024-05-09T05:22:59.160623+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:22:59.160623+0000","last_undegraded":"2024-05-09T05:22:59.160623+0000","last_fullsized":"2024-05-09T05:22:59.160623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'5877","reported_seq":5900,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.800435+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:23:41.800435+0000","last_peered":"2024-05-09T05:23:41.800435+0000","last_clean":"2024-05-09T05:23:41.800435+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:23:41.800435+0000","last_undegraded":"2024-05-09T05:23:41.800435+0000","last_fullsized":"2024-05-09T05:23:41.800435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29217,"num_write_kb":334206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"23'21","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.141090+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:22:59.141090+0000","last_peered":"2024-05-09T05:22:59.141090+0000","last_clean":"2024-05-09T05:22:59.141090+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:22:59.141090+0000","last_undegraded":"2024-05-09T05:22:59.141090+0000","last_fullsized":"2024-05-09T05:22:59.141090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'5443","reported_seq":5466,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.843005+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:23:41.843005+0000","last_peered":"2024-05-09T05:23:41.843005+0000","last_clean":"2024-05-09T05:23:41.843005+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:23:41.843005+0000","last_undegraded":"2024-05-09T05:23:41.843005+0000","last_fullsized":"2024-05-09T05:23:41.843005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27739,"num_write_kb":311936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.151151+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:22:59.151151+0000","last_peered":"2024-05-09T05:22:59.151151+0000","last_clean":"2024-05-09T05:22:59.151151+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:22:59.151151+0000","last_undegraded":"2024-05-09T05:22:59.151151+0000","last_fullsized":"2024-05-09T05:22:59.151151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'5295","reported_seq":5318,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.754503+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:23:41.754503+0000","last_peered":"2024-05-09T05:23:41.754503+0000","last_clean":"2024-05-09T05:23:41.754503+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:23:41.754503+0000","last_undegraded":"2024-05-09T05:23:41.754503+0000","last_fullsized":"2024-05-09T05:23:41.754503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25732,"num_write_kb":299773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'9","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.132085+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:22:59.132085+0000","last_peered":"2024-05-09T05:22:59.132085+0000","last_clean":"2024-05-09T05:22:59.132085+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:22:59.132085+0000","last_undegraded":"2024-05-09T05:22:59.132085+0000","last_fullsized":"2024-05-09T05:22:59.132085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'6402","reported_seq":6425,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.913813+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:23:41.913813+0000","last_peered":"2024-05-09T05:23:41.913813+0000","last_clean":"2024-05-09T05:23:41.913813+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:23:41.913813+0000","last_undegraded":"2024-05-09T05:23:41.913813+0000","last_fullsized":"2024-05-09T05:23:41.913813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31156,"num_write_kb":362485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:21:51.575458+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:21:51.575458+0000","last_peered":"2024-05-09T05:21:51.575458+0000","last_clean":"2024-05-09T05:21:51.575458+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:21:51.575458+0000","last_undegraded":"2024-05-09T05:21:51.575458+0000","last_fullsized":"2024-05-09T05:21:51.575458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'5559","reported_seq":5582,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.794865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:23:41.794865+0000","last_peered":"2024-05-09T05:23:41.794865+0000","last_clean":"2024-05-09T05:23:41.794865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:23:41.794865+0000","last_undegraded":"2024-05-09T05:23:41.794865+0000","last_fullsized":"2024-05-09T05:23:41.794865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28130,"num_write_kb":325652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'10","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.117825+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:22:59.117825+0000","last_peered":"2024-05-09T05:22:59.117825+0000","last_clean":"2024-05-09T05:22:59.117825+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:22:59.117825+0000","last_undegraded":"2024-05-09T05:22:59.117825+0000","last_fullsized":"2024-05-09T05:22:59.117825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'5794","reported_seq":5817,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.896072+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:23:41.896072+0000","last_peered":"2024-05-09T05:23:41.896072+0000","last_clean":"2024-05-09T05:23:41.896072+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:23:41.896072+0000","last_undegraded":"2024-05-09T05:23:41.896072+0000","last_fullsized":"2024-05-09T05:23:41.896072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28847,"num_write_kb":320571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":122,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:19:52.665112+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:19:52.665112+0000","last_peered":"2024-05-09T05:19:52.665112+0000","last_clean":"2024-05-09T05:19:52.665112+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:19:52.665112+0000","last_undegraded":"2024-05-09T05:19:52.665112+0000","last_fullsized":"2024-05-09T05:19:52.665112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'7","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:20:42.700037+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:20:42.700037+0000","last_peered":"2024-05-09T05:20:42.700037+0000","last_clean":"2024-05-09T05:20:42.700037+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:20:42.700037+0000","last_undegraded":"2024-05-09T05:20:42.700037+0000","last_fullsized":"2024-05-09T05:20:42.700037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'5510","reported_seq":5534,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.818585+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:23:41.818585+0000","last_peered":"2024-05-09T05:23:41.818585+0000","last_clean":"2024-05-09T05:23:41.818585+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:23:41.818585+0000","last_undegraded":"2024-05-09T05:23:41.818585+0000","last_fullsized":"2024-05-09T05:23:41.818585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5510,"log_dups_size":0,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27376,"num_write_kb":322728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'9","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.151152+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:22:59.151152+0000","last_peered":"2024-05-09T05:22:59.151152+0000","last_clean":"2024-05-09T05:22:59.151152+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:22:59.151152+0000","last_undegraded":"2024-05-09T05:22:59.151152+0000","last_fullsized":"2024-05-09T05:22:59.151152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'5383","reported_seq":5407,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.820369+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:23:41.820369+0000","last_peered":"2024-05-09T05:23:41.820369+0000","last_clean":"2024-05-09T05:23:41.820369+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:23:41.820369+0000","last_undegraded":"2024-05-09T05:23:41.820369+0000","last_fullsized":"2024-05-09T05:23:41.820369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:23.362012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27075,"num_write_kb":305200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'45","reported_seq":107,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.161683+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:22:59.161683+0000","last_peered":"2024-05-09T05:22:59.161683+0000","last_clean":"2024-05-09T05:22:59.161683+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:22:59.161683+0000","last_undegraded":"2024-05-09T05:22:59.161683+0000","last_fullsized":"2024-05-09T05:22:59.161683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88293,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":43,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'5441","reported_seq":5465,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.775191+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:23:41.775191+0000","last_peered":"2024-05-09T05:23:41.775191+0000","last_clean":"2024-05-09T05:23:41.775191+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:23:41.775191+0000","last_undegraded":"2024-05-09T05:23:41.775191+0000","last_fullsized":"2024-05-09T05:23:41.775191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27385,"num_write_kb":302253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.133458+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:22:59.133458+0000","last_peered":"2024-05-09T05:22:59.133458+0000","last_clean":"2024-05-09T05:22:59.133458+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:22:59.133458+0000","last_undegraded":"2024-05-09T05:22:59.133458+0000","last_fullsized":"2024-05-09T05:22:59.133458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'5472","reported_seq":5496,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.928867+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:23:41.928867+0000","last_peered":"2024-05-09T05:23:41.928867+0000","last_clean":"2024-05-09T05:23:41.928867+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:23:41.928867+0000","last_undegraded":"2024-05-09T05:23:41.928867+0000","last_fullsized":"2024-05-09T05:23:41.928867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28049,"num_write_kb":304974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'6","reported_seq":32,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:55.531860+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:22:55.531860+0000","last_peered":"2024-05-09T05:22:55.531860+0000","last_clean":"2024-05-09T05:22:55.531860+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:22:55.531860+0000","last_undegraded":"2024-05-09T05:22:55.531860+0000","last_fullsized":"2024-05-09T05:22:55.531860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'4678","reported_seq":4702,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.919111+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:23:41.919111+0000","last_peered":"2024-05-09T05:23:41.919111+0000","last_clean":"2024-05-09T05:23:41.919111+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:23:41.919111+0000","last_undegraded":"2024-05-09T05:23:41.919111+0000","last_fullsized":"2024-05-09T05:23:41.919111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23621,"num_write_kb":274908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"23'8","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.841082+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:22:54.841082+0000","last_peered":"2024-05-09T05:22:54.841082+0000","last_clean":"2024-05-09T05:22:54.841082+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:22:54.841082+0000","last_undegraded":"2024-05-09T05:22:54.841082+0000","last_fullsized":"2024-05-09T05:22:54.841082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'5941","reported_seq":5965,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.780983+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:23:41.780983+0000","last_peered":"2024-05-09T05:23:41.780983+0000","last_clean":"2024-05-09T05:23:41.780983+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:23:41.780983+0000","last_undegraded":"2024-05-09T05:23:41.780983+0000","last_fullsized":"2024-05-09T05:23:41.780983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":328170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'609","reported_seq":635,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:00.281854+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:23:00.281854+0000","last_peered":"2024-05-09T05:23:00.281854+0000","last_clean":"2024-05-09T05:23:00.281854+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:23:00.281854+0000","last_undegraded":"2024-05-09T05:23:00.281854+0000","last_fullsized":"2024-05-09T05:23:00.281854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'5503","reported_seq":5527,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.773405+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:23:41.773405+0000","last_peered":"2024-05-09T05:23:41.773405+0000","last_clean":"2024-05-09T05:23:41.773405+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:23:41.773405+0000","last_undegraded":"2024-05-09T05:23:41.773405+0000","last_fullsized":"2024-05-09T05:23:41.773405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:33.840344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26876,"num_write_kb":308483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.130335+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:22:59.130335+0000","last_peered":"2024-05-09T05:22:59.130335+0000","last_clean":"2024-05-09T05:22:59.130335+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:22:59.130335+0000","last_undegraded":"2024-05-09T05:22:59.130335+0000","last_fullsized":"2024-05-09T05:22:59.130335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'4642","reported_seq":4666,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.787942+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:23:41.787942+0000","last_peered":"2024-05-09T05:23:41.787942+0000","last_clean":"2024-05-09T05:23:41.787942+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:23:41.787942+0000","last_undegraded":"2024-05-09T05:23:41.787942+0000","last_fullsized":"2024-05-09T05:23:41.787942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:50.236344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22705,"num_write_kb":262889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'5","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.155406+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:22:59.155406+0000","last_peered":"2024-05-09T05:22:59.155406+0000","last_clean":"2024-05-09T05:22:59.155406+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:22:59.155406+0000","last_undegraded":"2024-05-09T05:22:59.155406+0000","last_fullsized":"2024-05-09T05:22:59.155406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'5215","reported_seq":5239,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790504+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:23:41.790504+0000","last_peered":"2024-05-09T05:23:41.790504+0000","last_clean":"2024-05-09T05:23:41.790504+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:23:41.790504+0000","last_undegraded":"2024-05-09T05:23:41.790504+0000","last_fullsized":"2024-05-09T05:23:41.790504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25185,"num_write_kb":293431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.138117+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:22:59.138117+0000","last_peered":"2024-05-09T05:22:59.138117+0000","last_clean":"2024-05-09T05:22:59.138117+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:22:59.138117+0000","last_undegraded":"2024-05-09T05:22:59.138117+0000","last_fullsized":"2024-05-09T05:22:59.138117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'5538","reported_seq":5562,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803051+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:23:41.803051+0000","last_peered":"2024-05-09T05:23:41.803051+0000","last_clean":"2024-05-09T05:23:41.803051+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:23:41.803051+0000","last_undegraded":"2024-05-09T05:23:41.803051+0000","last_fullsized":"2024-05-09T05:23:41.803051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27899,"num_write_kb":316727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"23'6","reported_seq":32,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.135039+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:22:59.135039+0000","last_peered":"2024-05-09T05:22:59.135039+0000","last_clean":"2024-05-09T05:22:59.135039+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:22:59.135039+0000","last_undegraded":"2024-05-09T05:22:59.135039+0000","last_fullsized":"2024-05-09T05:22:59.135039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'5078","reported_seq":5101,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.813122+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:23:41.813122+0000","last_peered":"2024-05-09T05:23:41.813122+0000","last_clean":"2024-05-09T05:23:41.813122+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:23:41.813122+0000","last_undegraded":"2024-05-09T05:23:41.813122+0000","last_fullsized":"2024-05-09T05:23:41.813122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:58:35.651549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24494,"num_write_kb":270229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1097","reported_seq":1122,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.133596+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:22:59.133596+0000","last_peered":"2024-05-09T05:22:59.133596+0000","last_clean":"2024-05-09T05:22:59.133596+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:22:59.133596+0000","last_undegraded":"2024-05-09T05:22:59.133596+0000","last_fullsized":"2024-05-09T05:22:59.133596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1089,"num_write_kb":4419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'4956","reported_seq":4980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.764163+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:23:41.764163+0000","last_peered":"2024-05-09T05:23:41.764163+0000","last_clean":"2024-05-09T05:23:41.764163+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:23:41.764163+0000","last_undegraded":"2024-05-09T05:23:41.764163+0000","last_fullsized":"2024-05-09T05:23:41.764163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24725,"num_write_kb":283470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'6745","reported_seq":6768,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.926010+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:23:41.926010+0000","last_peered":"2024-05-09T05:23:41.926010+0000","last_clean":"2024-05-09T05:23:41.926010+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:23:41.926010+0000","last_undegraded":"2024-05-09T05:23:41.926010+0000","last_fullsized":"2024-05-09T05:23:41.926010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33228,"num_write_kb":379009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.163507+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:22:59.163507+0000","last_peered":"2024-05-09T05:22:59.163507+0000","last_clean":"2024-05-09T05:22:59.163507+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:22:59.163507+0000","last_undegraded":"2024-05-09T05:22:59.163507+0000","last_fullsized":"2024-05-09T05:22:59.163507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'4185","reported_seq":4201,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.772240+0000","last_change":"2024-05-09T05:07:01.723135+0000","last_active":"2024-05-09T05:23:41.772240+0000","last_peered":"2024-05-09T05:23:41.772240+0000","last_clean":"2024-05-09T05:23:41.772240+0000","last_became_active":"2024-05-09T05:07:01.722785+0000","last_became_peered":"2024-05-09T05:07:01.722785+0000","last_unstale":"2024-05-09T05:23:41.772240+0000","last_undegraded":"2024-05-09T05:23:41.772240+0000","last_fullsized":"2024-05-09T05:23:41.772240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:57:53.799866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20471,"num_write_kb":229445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'21","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.166888+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:22:59.166888+0000","last_peered":"2024-05-09T05:22:59.166888+0000","last_clean":"2024-05-09T05:22:59.166888+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:22:59.166888+0000","last_undegraded":"2024-05-09T05:22:59.166888+0000","last_fullsized":"2024-05-09T05:22:59.166888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'6872","reported_seq":6896,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.883263+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:23:41.883263+0000","last_peered":"2024-05-09T05:23:41.883263+0000","last_clean":"2024-05-09T05:23:41.883263+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:23:41.883263+0000","last_undegraded":"2024-05-09T05:23:41.883263+0000","last_fullsized":"2024-05-09T05:23:41.883263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:42.457145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34844,"num_write_kb":385600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'9","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.127515+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:22:59.127515+0000","last_peered":"2024-05-09T05:22:59.127515+0000","last_clean":"2024-05-09T05:22:59.127515+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:22:59.127515+0000","last_undegraded":"2024-05-09T05:22:59.127515+0000","last_fullsized":"2024-05-09T05:22:59.127515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'5811","reported_seq":5835,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790466+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:23:41.790466+0000","last_peered":"2024-05-09T05:23:41.790466+0000","last_clean":"2024-05-09T05:23:41.790466+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:23:41.790466+0000","last_undegraded":"2024-05-09T05:23:41.790466+0000","last_fullsized":"2024-05-09T05:23:41.790466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:32:29.331565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29692,"num_write_kb":336226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'740","reported_seq":766,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:43.840338+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:22:43.840338+0000","last_peered":"2024-05-09T05:22:43.840338+0000","last_clean":"2024-05-09T05:22:43.840338+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:22:43.840338+0000","last_undegraded":"2024-05-09T05:22:43.840338+0000","last_fullsized":"2024-05-09T05:22:43.840338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":739,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1808567,"num_write_kb":20359177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":362904,"ondisk_log_size":362904,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18590025,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":4539,"num_write_kb":19914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37421056,"data_stored":37180050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3759,"ondisk_log_size":3759,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509657,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729492,"kb_used_data":13132,"kb_used_omap":1,"kb_used_meta":716350,"kb_avail":93642348,"statfs":{"total":96636764160,"available":95889764352,"internally_reserved":0,"allocated":13447168,"data_stored":13261681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":733542858},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509658,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702748,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":702462,"kb_avail":93669092,"statfs":{"total":96636764160,"available":95917150208,"internally_reserved":0,"allocated":282624,"data_stored":90233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719321541},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542362,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384208,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":378174,"kb_avail":93987632,"statfs":{"total":96636764160,"available":96243335168,"internally_reserved":0,"allocated":6168576,"data_stored":5997855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387250630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542363,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698768,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":692734,"kb_avail":93673072,"statfs":{"total":96636764160,"available":95921225728,"internally_reserved":0,"allocated":6168576,"data_stored":5997723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":709360069},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542363,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679136,"kb_used_data":8536,"kb_used_omap":1,"kb_used_meta":670590,"kb_avail":93692704,"statfs":{"total":96636764160,"available":95941328896,"internally_reserved":0,"allocated":8740864,"data_stored":8564586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":686684616},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542364,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400676,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":400318,"kb_avail":93971164,"statfs":{"total":96636764160,"available":96226471936,"internally_reserved":0,"allocated":356352,"data_stored":175944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":409926085},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542363,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":446014,"kb_avail":93925560,"statfs":{"total":96636764160,"available":96179773440,"internally_reserved":0,"allocated":262144,"data_stored":87251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456718789},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542362,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675468,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":670526,"kb_avail":93696372,"statfs":{"total":96636764160,"available":95945084928,"internally_reserved":0,"allocated":5050368,"data_stored":4871265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":686619077},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4222976,"data_stored":4195178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":90225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8507392,"data_stored":8478867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12619776,"data_stored":12585594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:23:48.751 INFO:tasks.ceph.ceph_manager.ceph:clean! 2024-05-09T05:23:48.751 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:23:48.901 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.904+0000 7f33b956b640 1 Processor -- start 2024-05-09T05:23:48.902 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.905+0000 7f33b956b640 1 -- start start 2024-05-09T05:23:48.903 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b956b640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33b4070930 0x7f33b415d250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b956b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f33b40c4c30 0x7f33b40b37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b956b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f33b40cc040 con 0x7f33b40c4820 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b956b640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f33b415d790 con 0x7f33b4070930 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b956b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f33b406fe60 con 0x7f33b40c4c30 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b2ffd640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f33b4070930 0x7f33b415d250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b27fc640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f33b40c4820 0x7f33b40cb930 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58802/0 (socket says 172.21.15.1:58802) 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b27fc640 1 -- 172.21.15.1:0/2581013910 learned_addr learned my addr 172.21.15.1:0/2581013910 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.906+0000 7f33b37fe640 1 --2- 172.21.15.1:0/2581013910 >> v2:172.21.15.196:3300/0 conn(0x7f33b40c4c30 0x7f33b40b37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b2ffd640 1 -- 172.21.15.1:0/2581013910 >> v2:172.21.15.196:3300/0 conn(0x7f33b40c4c30 msgr2=0x7f33b40b37a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b2ffd640 1 --2- 172.21.15.1:0/2581013910 >> v2:172.21.15.196:3300/0 conn(0x7f33b40c4c30 0x7f33b40b37a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b2ffd640 1 -- 172.21.15.1:0/2581013910 >> v1:172.21.15.1:6789/0 conn(0x7f33b40c4820 legacy=0x7f33b40cb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b2ffd640 1 -- 172.21.15.1:0/2581013910 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b4070140 con 0x7f33b4070930 2024-05-09T05:23:48.904 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b1ffb640 1 -- 172.21.15.1:0/2581013910 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 147227336 0 0) 0x7f33b40cc040 con 0x7f33b40c4820 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b2ffd640 1 --2- 172.21.15.1:0/2581013910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b415d250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f33b40c5c80 tx=0x7f33a402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c51e6086062fd36b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b1ffb640 1 -- 172.21.15.1:0/2581013910 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4007740 con 0x7f33b4070930 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b1ffb640 1 -- 172.21.15.1:0/2581013910 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a40078e0 con 0x7f33b4070930 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.907+0000 7f33b1ffb640 1 -- 172.21.15.1:0/2581013910 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4007cb0 con 0x7f33b4070930 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.908+0000 7f33b956b640 1 -- 172.21.15.1:0/2581013910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 msgr2=0x7f33b415d250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.908+0000 7f33b956b640 1 --2- 172.21.15.1:0/2581013910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b415d250 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f33b40c5c80 tx=0x7f33a402a710 comp rx=0 tx=0).stop 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.908+0000 7f33b956b640 1 -- 172.21.15.1:0/2581013910 shutdown_connections 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.908+0000 7f33b956b640 1 --2- 172.21.15.1:0/2581013910 >> v2:172.21.15.196:3300/0 conn(0x7f33b40c4c30 0x7f33b40b37a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.908+0000 7f33b956b640 1 --2- 172.21.15.1:0/2581013910 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b415d250 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.908+0000 7f33b956b640 1 -- 172.21.15.1:0/2581013910 >> 172.21.15.1:0/2581013910 conn(0x7f33b4071b40 msgr2=0x7f33b4071f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.908+0000 7f33b956b640 1 -- 172.21.15.1:0/2581013910 shutdown_connections 2024-05-09T05:23:48.906 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.908+0000 7f33b956b640 1 -- 172.21.15.1:0/2581013910 wait complete. 2024-05-09T05:23:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.910+0000 7f33b956b640 1 Processor -- start 2024-05-09T05:23:48.907 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.910+0000 7f33b956b640 1 -- start start 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.911+0000 7f33b956b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b41f23b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.911+0000 7f33b956b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33b40c4820 0x7f33b4155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.911+0000 7f33b956b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33b40c4c30 0x7f33b41507d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.911+0000 7f33b956b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f33b4160410 con 0x7f33b4070930 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.911+0000 7f33b956b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f33b4160130 con 0x7f33b40c4c30 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.911+0000 7f33b956b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f33b41606f0 con 0x7f33b40c4820 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.911+0000 7f33b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b41f23b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b2ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b41f23b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60328/0 (socket says 172.21.15.1:60328) 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b2ffd640 1 -- 172.21.15.1:0/1024216530 learned_addr learned my addr 172.21.15.1:0/1024216530 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b37fe640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33b40c4c30 0x7f33b41507d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.910 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b27fc640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33b40c4820 0x7f33b4155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b2ffd640 1 -- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33b40c4820 msgr2=0x7f33b4155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b2ffd640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33b40c4820 0x7f33b4155c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b2ffd640 1 -- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33b40c4c30 msgr2=0x7f33b41507d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b2ffd640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33b40c4c30 0x7f33b41507d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b2ffd640 1 -- 172.21.15.1:0/1024216530 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33b41f28f0 con 0x7f33b4070930 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b2ffd640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b41f23b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009e80 tx=0x7f33a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33ab7fe640 1 -- 172.21.15.1:0/1024216530 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a403f070 con 0x7f33b4070930 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33ab7fe640 1 -- 172.21.15.1:0/1024216530 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f33a4031c40 con 0x7f33b4070930 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33b415f1b0 con 0x7f33b4070930 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33ab7fe640 1 -- 172.21.15.1:0/1024216530 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33a4044460 con 0x7f33b4070930 2024-05-09T05:23:48.911 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.912+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33b415f770 con 0x7f33b4070930 2024-05-09T05:23:48.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.913+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f33b4156f50 con 0x7f33b4070930 2024-05-09T05:23:48.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.914+0000 7f33ab7fe640 1 -- 172.21.15.1:0/1024216530 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f33a403b410 con 0x7f33b4070930 2024-05-09T05:23:48.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.915+0000 7f33ab7fe640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3394076300 0x7f3394078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:48.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.915+0000 7f33ab7fe640 1 -- 172.21.15.1:0/1024216530 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f33a40b8c10 con 0x7f33b4070930 2024-05-09T05:23:48.912 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.915+0000 7f33b27fc640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3394076300 0x7f3394078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:48.913 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.915+0000 7f33b27fc640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3394076300 0x7f3394078810 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f33b40b3a70 tx=0x7f339c009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:48.916 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:48.920+0000 7f33ab7fe640 1 -- 172.21.15.1:0/1024216530 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f33a4049060 con 0x7f33b4070930 2024-05-09T05:23:49.105 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.109+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f33b406e9d0 con 0x7f3394076300 2024-05-09T05:23:49.112 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.115+0000 7f33ab7fe640 1 -- 172.21.15.1:0/1024216530 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+253608 (secure 0 0 0) 0x7f33b406e9d0 con 0x7f3394076300 2024-05-09T05:23:49.114 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:23:49.116 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:23:49.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3394076300 msgr2=0x7f3394078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3394076300 0x7f3394078810 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f33b40b3a70 tx=0x7f339c009040 comp rx=0 tx=0).stop 2024-05-09T05:23:49.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 msgr2=0x7f33b41f23b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b41f23b0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f33a4009e80 tx=0x7f33a4004a40 comp rx=0 tx=0).stop 2024-05-09T05:23:49.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 shutdown_connections 2024-05-09T05:23:49.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3394076300 0x7f3394078810 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.117 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f33b40c4c30 0x7f33b41507d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f33b40c4820 0x7f33b4155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 --2- 172.21.15.1:0/1024216530 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f33b4070930 0x7f33b41f23b0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 >> 172.21.15.1:0/1024216530 conn(0x7f33b4071b40 msgr2=0x7f33b40ca220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:49.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 shutdown_connections 2024-05-09T05:23:49.118 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.118+0000 7f33b956b640 1 -- 172.21.15.1:0/1024216530 wait complete. 2024-05-09T05:23:49.127 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":554,"stamp":"2024-05-09T05:23:47.833121+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19180393,"num_objects":105,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":102,"num_read_kb":167,"num_write":1813207,"num_write_kb":20380719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366735,"ondisk_log_size":366735,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4716776,"kb_used_data":39528,"kb_used_omap":12,"kb_used_meta":4677171,"kb_avail":750257944,"statfs":{"total":773094113280,"available":768264134656,"internally_reserved":0,"allocated":40476672,"data_stored":39046538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4789423665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":672,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003605"},"pg_stats":[{"pgid":"3.3f","version":"23'5187","reported_seq":5211,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.889349+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:23:41.889349+0000","last_peered":"2024-05-09T05:23:41.889349+0000","last_clean":"2024-05-09T05:23:41.889349+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:23:41.889349+0000","last_undegraded":"2024-05-09T05:23:41.889349+0000","last_fullsized":"2024-05-09T05:23:41.889349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26135,"num_write_kb":280897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'6912","reported_seq":6936,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.786450+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:23:41.786450+0000","last_peered":"2024-05-09T05:23:41.786450+0000","last_clean":"2024-05-09T05:23:41.786450+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:23:41.786450+0000","last_undegraded":"2024-05-09T05:23:41.786450+0000","last_fullsized":"2024-05-09T05:23:41.786450+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:28:11.078495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35479,"num_write_kb":400336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'5571","reported_seq":5595,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.785914+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:23:41.785914+0000","last_peered":"2024-05-09T05:23:41.785914+0000","last_clean":"2024-05-09T05:23:41.785914+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:23:41.785914+0000","last_undegraded":"2024-05-09T05:23:41.785914+0000","last_fullsized":"2024-05-09T05:23:41.785914+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:00:48.770365+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27199,"num_write_kb":317974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'5494","reported_seq":5518,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.804206+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:23:41.804206+0000","last_peered":"2024-05-09T05:23:41.804206+0000","last_clean":"2024-05-09T05:23:41.804206+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:23:41.804206+0000","last_undegraded":"2024-05-09T05:23:41.804206+0000","last_fullsized":"2024-05-09T05:23:41.804206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28054,"num_write_kb":296869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'7058","reported_seq":7081,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790501+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:23:41.790501+0000","last_peered":"2024-05-09T05:23:41.790501+0000","last_clean":"2024-05-09T05:23:41.790501+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:23:41.790501+0000","last_undegraded":"2024-05-09T05:23:41.790501+0000","last_fullsized":"2024-05-09T05:23:41.790501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34281,"num_write_kb":379027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'4948","reported_seq":4971,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.889531+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:23:41.889531+0000","last_peered":"2024-05-09T05:23:41.889531+0000","last_clean":"2024-05-09T05:23:41.889531+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:23:41.889531+0000","last_undegraded":"2024-05-09T05:23:41.889531+0000","last_fullsized":"2024-05-09T05:23:41.889531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24780,"num_write_kb":275405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'5454","reported_seq":5477,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.811426+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:23:41.811426+0000","last_peered":"2024-05-09T05:23:41.811426+0000","last_clean":"2024-05-09T05:23:41.811426+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:23:41.811426+0000","last_undegraded":"2024-05-09T05:23:41.811426+0000","last_fullsized":"2024-05-09T05:23:41.811426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27432,"num_write_kb":306787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'6053","reported_seq":6076,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.840232+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:23:41.840232+0000","last_peered":"2024-05-09T05:23:41.840232+0000","last_clean":"2024-05-09T05:23:41.840232+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:23:41.840232+0000","last_undegraded":"2024-05-09T05:23:41.840232+0000","last_fullsized":"2024-05-09T05:23:41.840232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6053,"log_dups_size":0,"ondisk_log_size":6053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29445,"num_write_kb":339212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"23'7129","reported_seq":7152,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.913522+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:23:41.913522+0000","last_peered":"2024-05-09T05:23:41.913522+0000","last_clean":"2024-05-09T05:23:41.913522+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:23:41.913522+0000","last_undegraded":"2024-05-09T05:23:41.913522+0000","last_fullsized":"2024-05-09T05:23:41.913522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36611,"num_write_kb":392539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'6358","reported_seq":6381,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.896558+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:23:41.896558+0000","last_peered":"2024-05-09T05:23:41.896558+0000","last_clean":"2024-05-09T05:23:41.896558+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:23:41.896558+0000","last_undegraded":"2024-05-09T05:23:41.896558+0000","last_fullsized":"2024-05-09T05:23:41.896558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32647,"num_write_kb":360281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'4747","reported_seq":4770,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.882769+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:23:41.882769+0000","last_peered":"2024-05-09T05:23:41.882769+0000","last_clean":"2024-05-09T05:23:41.882769+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:23:41.882769+0000","last_undegraded":"2024-05-09T05:23:41.882769+0000","last_fullsized":"2024-05-09T05:23:41.882769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22850,"num_write_kb":288147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'4814","reported_seq":4837,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.916891+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:23:41.916891+0000","last_peered":"2024-05-09T05:23:41.916891+0000","last_clean":"2024-05-09T05:23:41.916891+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:23:41.916891+0000","last_undegraded":"2024-05-09T05:23:41.916891+0000","last_fullsized":"2024-05-09T05:23:41.916891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":267591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'5105","reported_seq":5128,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.762396+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:23:41.762396+0000","last_peered":"2024-05-09T05:23:41.762396+0000","last_clean":"2024-05-09T05:23:41.762396+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:23:41.762396+0000","last_undegraded":"2024-05-09T05:23:41.762396+0000","last_fullsized":"2024-05-09T05:23:41.762396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25963,"num_write_kb":288363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'5957","reported_seq":5980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.819716+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:23:41.819716+0000","last_peered":"2024-05-09T05:23:41.819716+0000","last_clean":"2024-05-09T05:23:41.819716+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:23:41.819716+0000","last_undegraded":"2024-05-09T05:23:41.819716+0000","last_fullsized":"2024-05-09T05:23:41.819716+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:47:37.258923+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29976,"num_write_kb":329157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'5739","reported_seq":5762,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.843331+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:23:41.843331+0000","last_peered":"2024-05-09T05:23:41.843331+0000","last_clean":"2024-05-09T05:23:41.843331+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:23:41.843331+0000","last_undegraded":"2024-05-09T05:23:41.843331+0000","last_fullsized":"2024-05-09T05:23:41.843331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28226,"num_write_kb":322058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'5341","reported_seq":5364,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.852751+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:23:41.852751+0000","last_peered":"2024-05-09T05:23:41.852751+0000","last_clean":"2024-05-09T05:23:41.852751+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:23:41.852751+0000","last_undegraded":"2024-05-09T05:23:41.852751+0000","last_fullsized":"2024-05-09T05:23:41.852751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27307,"num_write_kb":289793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"23'6965","reported_seq":6982,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803206+0000","last_change":"2024-05-09T05:07:01.750820+0000","last_active":"2024-05-09T05:23:41.803206+0000","last_peered":"2024-05-09T05:23:41.803206+0000","last_clean":"2024-05-09T05:23:41.803206+0000","last_became_active":"2024-05-09T05:07:01.750492+0000","last_became_peered":"2024-05-09T05:07:01.750492+0000","last_unstale":"2024-05-09T05:23:41.803206+0000","last_undegraded":"2024-05-09T05:23:41.803206+0000","last_fullsized":"2024-05-09T05:23:41.803206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6965,"log_dups_size":0,"ondisk_log_size":6965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:23.141520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34877,"num_write_kb":385595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"23'4549","reported_seq":4572,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.791997+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:23:41.791997+0000","last_peered":"2024-05-09T05:23:41.791997+0000","last_clean":"2024-05-09T05:23:41.791997+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:23:41.791997+0000","last_undegraded":"2024-05-09T05:23:41.791997+0000","last_fullsized":"2024-05-09T05:23:41.791997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22029,"num_write_kb":252128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'5335","reported_seq":5358,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.781003+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:23:41.781003+0000","last_peered":"2024-05-09T05:23:41.781003+0000","last_clean":"2024-05-09T05:23:41.781003+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:23:41.781003+0000","last_undegraded":"2024-05-09T05:23:41.781003+0000","last_fullsized":"2024-05-09T05:23:41.781003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26106,"num_write_kb":309468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'5370","reported_seq":5393,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.861348+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:23:41.861348+0000","last_peered":"2024-05-09T05:23:41.861348+0000","last_clean":"2024-05-09T05:23:41.861348+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:23:41.861348+0000","last_undegraded":"2024-05-09T05:23:41.861348+0000","last_fullsized":"2024-05-09T05:23:41.861348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26990,"num_write_kb":308937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'5403","reported_seq":5426,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.859039+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:23:41.859039+0000","last_peered":"2024-05-09T05:23:41.859039+0000","last_clean":"2024-05-09T05:23:41.859039+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:23:41.859039+0000","last_undegraded":"2024-05-09T05:23:41.859039+0000","last_fullsized":"2024-05-09T05:23:41.859039+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:02.665344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26698,"num_write_kb":315887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'6082","reported_seq":6105,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.904062+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:23:41.904062+0000","last_peered":"2024-05-09T05:23:41.904062+0000","last_clean":"2024-05-09T05:23:41.904062+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:23:41.904062+0000","last_undegraded":"2024-05-09T05:23:41.904062+0000","last_fullsized":"2024-05-09T05:23:41.904062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30462,"num_write_kb":343332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"23'5512","reported_seq":5535,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.774669+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:23:41.774669+0000","last_peered":"2024-05-09T05:23:41.774669+0000","last_clean":"2024-05-09T05:23:41.774669+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:23:41.774669+0000","last_undegraded":"2024-05-09T05:23:41.774669+0000","last_fullsized":"2024-05-09T05:23:41.774669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27471,"num_write_kb":310767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'5065","reported_seq":5089,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.839542+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:23:41.839542+0000","last_peered":"2024-05-09T05:23:41.839542+0000","last_clean":"2024-05-09T05:23:41.839542+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:23:41.839542+0000","last_undegraded":"2024-05-09T05:23:41.839542+0000","last_fullsized":"2024-05-09T05:23:41.839542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24983,"num_write_kb":280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'6632","reported_seq":6656,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.735021+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:23:41.735021+0000","last_peered":"2024-05-09T05:23:41.735021+0000","last_clean":"2024-05-09T05:23:41.735021+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:23:41.735021+0000","last_undegraded":"2024-05-09T05:23:41.735021+0000","last_fullsized":"2024-05-09T05:23:41.735021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:25:28.166958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33114,"num_write_kb":370371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'5121","reported_seq":5150,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.872969+0000","last_change":"2024-05-09T05:07:01.662425+0000","last_active":"2024-05-09T05:23:41.872969+0000","last_peered":"2024-05-09T05:23:41.872969+0000","last_clean":"2024-05-09T05:23:41.872969+0000","last_became_active":"2024-05-09T05:07:01.662096+0000","last_became_peered":"2024-05-09T05:07:01.662096+0000","last_unstale":"2024-05-09T05:23:41.872969+0000","last_undegraded":"2024-05-09T05:23:41.872969+0000","last_fullsized":"2024-05-09T05:23:41.872969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:50.949207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25567,"num_write_kb":287181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'6014","reported_seq":6038,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.767615+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:23:41.767615+0000","last_peered":"2024-05-09T05:23:41.767615+0000","last_clean":"2024-05-09T05:23:41.767615+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:23:41.767615+0000","last_undegraded":"2024-05-09T05:23:41.767615+0000","last_fullsized":"2024-05-09T05:23:41.767615+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30657,"num_write_kb":340145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'5132","reported_seq":5149,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.864977+0000","last_change":"2024-05-09T05:07:01.723062+0000","last_active":"2024-05-09T05:23:41.864977+0000","last_peered":"2024-05-09T05:23:41.864977+0000","last_clean":"2024-05-09T05:23:41.864977+0000","last_became_active":"2024-05-09T05:07:01.722736+0000","last_became_peered":"2024-05-09T05:07:01.722736+0000","last_unstale":"2024-05-09T05:23:41.864977+0000","last_undegraded":"2024-05-09T05:23:41.864977+0000","last_fullsized":"2024-05-09T05:23:41.864977+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:50.949207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25502,"num_write_kb":286500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"23'4664","reported_seq":4688,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.846625+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:23:41.846625+0000","last_peered":"2024-05-09T05:23:41.846625+0000","last_clean":"2024-05-09T05:23:41.846625+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:23:41.846625+0000","last_undegraded":"2024-05-09T05:23:41.846625+0000","last_fullsized":"2024-05-09T05:23:41.846625+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23461,"num_write_kb":270138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'6545","reported_seq":6569,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.857347+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:23:41.857347+0000","last_peered":"2024-05-09T05:23:41.857347+0000","last_clean":"2024-05-09T05:23:41.857347+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:23:41.857347+0000","last_undegraded":"2024-05-09T05:23:41.857347+0000","last_fullsized":"2024-05-09T05:23:41.857347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6545,"log_dups_size":0,"ondisk_log_size":6545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32981,"num_write_kb":377918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'6387","reported_seq":6411,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.809273+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:23:41.809273+0000","last_peered":"2024-05-09T05:23:41.809273+0000","last_clean":"2024-05-09T05:23:41.809273+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:23:41.809273+0000","last_undegraded":"2024-05-09T05:23:41.809273+0000","last_fullsized":"2024-05-09T05:23:41.809273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32133,"num_write_kb":346002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'7361","reported_seq":7385,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.851736+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:23:41.851736+0000","last_peered":"2024-05-09T05:23:41.851736+0000","last_clean":"2024-05-09T05:23:41.851736+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:23:41.851736+0000","last_undegraded":"2024-05-09T05:23:41.851736+0000","last_fullsized":"2024-05-09T05:23:41.851736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36142,"num_write_kb":398870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'5228","reported_seq":5251,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803020+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:23:41.803020+0000","last_peered":"2024-05-09T05:23:41.803020+0000","last_clean":"2024-05-09T05:23:41.803020+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:23:41.803020+0000","last_undegraded":"2024-05-09T05:23:41.803020+0000","last_fullsized":"2024-05-09T05:23:41.803020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25537,"num_write_kb":278419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'577","reported_seq":602,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:36.679365+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:23:36.679365+0000","last_peered":"2024-05-09T05:23:36.679365+0000","last_clean":"2024-05-09T05:23:36.679365+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:23:36.679365+0000","last_undegraded":"2024-05-09T05:23:36.679365+0000","last_fullsized":"2024-05-09T05:23:36.679365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":573,"num_write_kb":4218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'4521","reported_seq":4544,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.926128+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:23:41.926128+0000","last_peered":"2024-05-09T05:23:41.926128+0000","last_clean":"2024-05-09T05:23:41.926128+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:23:41.926128+0000","last_undegraded":"2024-05-09T05:23:41.926128+0000","last_fullsized":"2024-05-09T05:23:41.926128+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22336,"num_write_kb":249223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'111","reported_seq":136,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.634306+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:23:41.634306+0000","last_peered":"2024-05-09T05:23:41.634306+0000","last_clean":"2024-05-09T05:23:41.634306+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:23:41.634306+0000","last_undegraded":"2024-05-09T05:23:41.634306+0000","last_fullsized":"2024-05-09T05:23:41.634306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1715788,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":1741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.829290+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:22:54.829290+0000","last_peered":"2024-05-09T05:22:54.829290+0000","last_clean":"2024-05-09T05:22:54.829290+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:22:54.829290+0000","last_undegraded":"2024-05-09T05:22:54.829290+0000","last_fullsized":"2024-05-09T05:22:54.829290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'5503","reported_seq":5526,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.842314+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:23:41.842314+0000","last_peered":"2024-05-09T05:23:41.842314+0000","last_clean":"2024-05-09T05:23:41.842314+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:23:41.842314+0000","last_undegraded":"2024-05-09T05:23:41.842314+0000","last_fullsized":"2024-05-09T05:23:41.842314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27748,"num_write_kb":311965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.156887+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:22:59.156887+0000","last_peered":"2024-05-09T05:22:59.156887+0000","last_clean":"2024-05-09T05:22:59.156887+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:22:59.156887+0000","last_undegraded":"2024-05-09T05:22:59.156887+0000","last_fullsized":"2024-05-09T05:22:59.156887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'6912","reported_seq":6935,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.823969+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:23:41.823969+0000","last_peered":"2024-05-09T05:23:41.823969+0000","last_clean":"2024-05-09T05:23:41.823969+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:23:41.823969+0000","last_undegraded":"2024-05-09T05:23:41.823969+0000","last_fullsized":"2024-05-09T05:23:41.823969+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34348,"num_write_kb":375438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'9","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.705733+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:23:41.705733+0000","last_peered":"2024-05-09T05:23:41.705733+0000","last_clean":"2024-05-09T05:23:41.705733+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:23:41.705733+0000","last_undegraded":"2024-05-09T05:23:41.705733+0000","last_fullsized":"2024-05-09T05:23:41.705733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'5546","reported_seq":5569,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.742360+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:23:41.742360+0000","last_peered":"2024-05-09T05:23:41.742360+0000","last_clean":"2024-05-09T05:23:41.742360+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:23:41.742360+0000","last_undegraded":"2024-05-09T05:23:41.742360+0000","last_fullsized":"2024-05-09T05:23:41.742360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5546,"log_dups_size":0,"ondisk_log_size":5546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27252,"num_write_kb":314086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.148513+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:22:59.148513+0000","last_peered":"2024-05-09T05:22:59.148513+0000","last_clean":"2024-05-09T05:22:59.148513+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:22:59.148513+0000","last_undegraded":"2024-05-09T05:22:59.148513+0000","last_fullsized":"2024-05-09T05:22:59.148513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'5049","reported_seq":5072,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.871865+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:23:41.871865+0000","last_peered":"2024-05-09T05:23:41.871865+0000","last_clean":"2024-05-09T05:23:41.871865+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:23:41.871865+0000","last_undegraded":"2024-05-09T05:23:41.871865+0000","last_fullsized":"2024-05-09T05:23:41.871865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:31:26.607004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25089,"num_write_kb":276787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'25","reported_seq":50,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.204360+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:22:59.204360+0000","last_peered":"2024-05-09T05:22:59.204360+0000","last_clean":"2024-05-09T05:22:59.204360+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:22:59.204360+0000","last_undegraded":"2024-05-09T05:22:59.204360+0000","last_fullsized":"2024-05-09T05:22:59.204360+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'7381","reported_seq":7404,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.917166+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:23:41.917166+0000","last_peered":"2024-05-09T05:23:41.917166+0000","last_clean":"2024-05-09T05:23:41.917166+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:23:41.917166+0000","last_undegraded":"2024-05-09T05:23:41.917166+0000","last_fullsized":"2024-05-09T05:23:41.917166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7381,"log_dups_size":0,"ondisk_log_size":7381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36744,"num_write_kb":418024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'333","reported_seq":358,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.155427+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:22:59.155427+0000","last_peered":"2024-05-09T05:22:59.155427+0000","last_clean":"2024-05-09T05:22:59.155427+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:22:59.155427+0000","last_undegraded":"2024-05-09T05:22:59.155427+0000","last_fullsized":"2024-05-09T05:22:59.155427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":331,"num_write_kb":4250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'5538","reported_seq":5561,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.787753+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:23:41.787753+0000","last_peered":"2024-05-09T05:23:41.787753+0000","last_clean":"2024-05-09T05:23:41.787753+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:23:41.787753+0000","last_undegraded":"2024-05-09T05:23:41.787753+0000","last_fullsized":"2024-05-09T05:23:41.787753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27557,"num_write_kb":302974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"23'15","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.113005+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:22:59.113005+0000","last_peered":"2024-05-09T05:22:59.113005+0000","last_clean":"2024-05-09T05:22:59.113005+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:22:59.113005+0000","last_undegraded":"2024-05-09T05:22:59.113005+0000","last_fullsized":"2024-05-09T05:22:59.113005+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:50:29.834698+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'5579","reported_seq":5602,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.833702+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:23:41.833702+0000","last_peered":"2024-05-09T05:23:41.833702+0000","last_clean":"2024-05-09T05:23:41.833702+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:23:41.833702+0000","last_undegraded":"2024-05-09T05:23:41.833702+0000","last_fullsized":"2024-05-09T05:23:41.833702+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:41.283927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27823,"num_write_kb":303973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'10","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.117627+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:22:59.117627+0000","last_peered":"2024-05-09T05:22:59.117627+0000","last_clean":"2024-05-09T05:22:59.117627+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:22:59.117627+0000","last_undegraded":"2024-05-09T05:22:59.117627+0000","last_fullsized":"2024-05-09T05:22:59.117627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'6303","reported_seq":6326,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.912348+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:23:41.912348+0000","last_peered":"2024-05-09T05:23:41.912348+0000","last_clean":"2024-05-09T05:23:41.912348+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:23:41.912348+0000","last_undegraded":"2024-05-09T05:23:41.912348+0000","last_fullsized":"2024-05-09T05:23:41.912348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30554,"num_write_kb":351862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'13","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.160623+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:22:59.160623+0000","last_peered":"2024-05-09T05:22:59.160623+0000","last_clean":"2024-05-09T05:22:59.160623+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:22:59.160623+0000","last_undegraded":"2024-05-09T05:22:59.160623+0000","last_fullsized":"2024-05-09T05:22:59.160623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'5877","reported_seq":5900,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.800435+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:23:41.800435+0000","last_peered":"2024-05-09T05:23:41.800435+0000","last_clean":"2024-05-09T05:23:41.800435+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:23:41.800435+0000","last_undegraded":"2024-05-09T05:23:41.800435+0000","last_fullsized":"2024-05-09T05:23:41.800435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29217,"num_write_kb":334206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"23'21","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.141090+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:22:59.141090+0000","last_peered":"2024-05-09T05:22:59.141090+0000","last_clean":"2024-05-09T05:22:59.141090+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:22:59.141090+0000","last_undegraded":"2024-05-09T05:22:59.141090+0000","last_fullsized":"2024-05-09T05:22:59.141090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'5443","reported_seq":5466,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.843005+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:23:41.843005+0000","last_peered":"2024-05-09T05:23:41.843005+0000","last_clean":"2024-05-09T05:23:41.843005+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:23:41.843005+0000","last_undegraded":"2024-05-09T05:23:41.843005+0000","last_fullsized":"2024-05-09T05:23:41.843005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27739,"num_write_kb":311936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.151151+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:22:59.151151+0000","last_peered":"2024-05-09T05:22:59.151151+0000","last_clean":"2024-05-09T05:22:59.151151+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:22:59.151151+0000","last_undegraded":"2024-05-09T05:22:59.151151+0000","last_fullsized":"2024-05-09T05:22:59.151151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'5295","reported_seq":5318,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.754503+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:23:41.754503+0000","last_peered":"2024-05-09T05:23:41.754503+0000","last_clean":"2024-05-09T05:23:41.754503+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:23:41.754503+0000","last_undegraded":"2024-05-09T05:23:41.754503+0000","last_fullsized":"2024-05-09T05:23:41.754503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25732,"num_write_kb":299773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'9","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.132085+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:22:59.132085+0000","last_peered":"2024-05-09T05:22:59.132085+0000","last_clean":"2024-05-09T05:22:59.132085+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:22:59.132085+0000","last_undegraded":"2024-05-09T05:22:59.132085+0000","last_fullsized":"2024-05-09T05:22:59.132085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'6402","reported_seq":6425,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.913813+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:23:41.913813+0000","last_peered":"2024-05-09T05:23:41.913813+0000","last_clean":"2024-05-09T05:23:41.913813+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:23:41.913813+0000","last_undegraded":"2024-05-09T05:23:41.913813+0000","last_fullsized":"2024-05-09T05:23:41.913813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31156,"num_write_kb":362485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"23'4","reported_seq":29,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:21:51.575458+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:21:51.575458+0000","last_peered":"2024-05-09T05:21:51.575458+0000","last_clean":"2024-05-09T05:21:51.575458+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:21:51.575458+0000","last_undegraded":"2024-05-09T05:21:51.575458+0000","last_fullsized":"2024-05-09T05:21:51.575458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'5559","reported_seq":5582,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.794865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:23:41.794865+0000","last_peered":"2024-05-09T05:23:41.794865+0000","last_clean":"2024-05-09T05:23:41.794865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:23:41.794865+0000","last_undegraded":"2024-05-09T05:23:41.794865+0000","last_fullsized":"2024-05-09T05:23:41.794865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28130,"num_write_kb":325652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'10","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.117825+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:22:59.117825+0000","last_peered":"2024-05-09T05:22:59.117825+0000","last_clean":"2024-05-09T05:22:59.117825+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:22:59.117825+0000","last_undegraded":"2024-05-09T05:22:59.117825+0000","last_fullsized":"2024-05-09T05:22:59.117825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'5794","reported_seq":5817,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.896072+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:23:41.896072+0000","last_peered":"2024-05-09T05:23:41.896072+0000","last_clean":"2024-05-09T05:23:41.896072+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:23:41.896072+0000","last_undegraded":"2024-05-09T05:23:41.896072+0000","last_fullsized":"2024-05-09T05:23:41.896072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28847,"num_write_kb":320571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":122,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:19:52.665112+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:19:52.665112+0000","last_peered":"2024-05-09T05:19:52.665112+0000","last_clean":"2024-05-09T05:19:52.665112+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:19:52.665112+0000","last_undegraded":"2024-05-09T05:19:52.665112+0000","last_fullsized":"2024-05-09T05:19:52.665112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'7","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:20:42.700037+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:20:42.700037+0000","last_peered":"2024-05-09T05:20:42.700037+0000","last_clean":"2024-05-09T05:20:42.700037+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:20:42.700037+0000","last_undegraded":"2024-05-09T05:20:42.700037+0000","last_fullsized":"2024-05-09T05:20:42.700037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'5510","reported_seq":5534,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.818585+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:23:41.818585+0000","last_peered":"2024-05-09T05:23:41.818585+0000","last_clean":"2024-05-09T05:23:41.818585+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:23:41.818585+0000","last_undegraded":"2024-05-09T05:23:41.818585+0000","last_fullsized":"2024-05-09T05:23:41.818585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5510,"log_dups_size":0,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27376,"num_write_kb":322728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'9","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.151152+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:22:59.151152+0000","last_peered":"2024-05-09T05:22:59.151152+0000","last_clean":"2024-05-09T05:22:59.151152+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:22:59.151152+0000","last_undegraded":"2024-05-09T05:22:59.151152+0000","last_fullsized":"2024-05-09T05:22:59.151152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'5383","reported_seq":5407,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.820369+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:23:41.820369+0000","last_peered":"2024-05-09T05:23:41.820369+0000","last_clean":"2024-05-09T05:23:41.820369+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:23:41.820369+0000","last_undegraded":"2024-05-09T05:23:41.820369+0000","last_fullsized":"2024-05-09T05:23:41.820369+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:23.362012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27075,"num_write_kb":305200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'45","reported_seq":107,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.161683+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:22:59.161683+0000","last_peered":"2024-05-09T05:22:59.161683+0000","last_clean":"2024-05-09T05:22:59.161683+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:22:59.161683+0000","last_undegraded":"2024-05-09T05:22:59.161683+0000","last_fullsized":"2024-05-09T05:22:59.161683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88293,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":43,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'5441","reported_seq":5465,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.775191+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:23:41.775191+0000","last_peered":"2024-05-09T05:23:41.775191+0000","last_clean":"2024-05-09T05:23:41.775191+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:23:41.775191+0000","last_undegraded":"2024-05-09T05:23:41.775191+0000","last_fullsized":"2024-05-09T05:23:41.775191+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27385,"num_write_kb":302253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.133458+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:22:59.133458+0000","last_peered":"2024-05-09T05:22:59.133458+0000","last_clean":"2024-05-09T05:22:59.133458+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:22:59.133458+0000","last_undegraded":"2024-05-09T05:22:59.133458+0000","last_fullsized":"2024-05-09T05:22:59.133458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'5472","reported_seq":5496,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.928867+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:23:41.928867+0000","last_peered":"2024-05-09T05:23:41.928867+0000","last_clean":"2024-05-09T05:23:41.928867+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:23:41.928867+0000","last_undegraded":"2024-05-09T05:23:41.928867+0000","last_fullsized":"2024-05-09T05:23:41.928867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28049,"num_write_kb":304974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'6","reported_seq":32,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:55.531860+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:22:55.531860+0000","last_peered":"2024-05-09T05:22:55.531860+0000","last_clean":"2024-05-09T05:22:55.531860+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:22:55.531860+0000","last_undegraded":"2024-05-09T05:22:55.531860+0000","last_fullsized":"2024-05-09T05:22:55.531860+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'4678","reported_seq":4702,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.919111+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:23:41.919111+0000","last_peered":"2024-05-09T05:23:41.919111+0000","last_clean":"2024-05-09T05:23:41.919111+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:23:41.919111+0000","last_undegraded":"2024-05-09T05:23:41.919111+0000","last_fullsized":"2024-05-09T05:23:41.919111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23621,"num_write_kb":274908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"23'8","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.841082+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:22:54.841082+0000","last_peered":"2024-05-09T05:22:54.841082+0000","last_clean":"2024-05-09T05:22:54.841082+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:22:54.841082+0000","last_undegraded":"2024-05-09T05:22:54.841082+0000","last_fullsized":"2024-05-09T05:22:54.841082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'5941","reported_seq":5965,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.780983+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:23:41.780983+0000","last_peered":"2024-05-09T05:23:41.780983+0000","last_clean":"2024-05-09T05:23:41.780983+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:23:41.780983+0000","last_undegraded":"2024-05-09T05:23:41.780983+0000","last_fullsized":"2024-05-09T05:23:41.780983+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":328170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'609","reported_seq":635,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:00.281854+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:23:00.281854+0000","last_peered":"2024-05-09T05:23:00.281854+0000","last_clean":"2024-05-09T05:23:00.281854+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:23:00.281854+0000","last_undegraded":"2024-05-09T05:23:00.281854+0000","last_fullsized":"2024-05-09T05:23:00.281854+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:08.505935+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'5503","reported_seq":5527,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.773405+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:23:41.773405+0000","last_peered":"2024-05-09T05:23:41.773405+0000","last_clean":"2024-05-09T05:23:41.773405+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:23:41.773405+0000","last_undegraded":"2024-05-09T05:23:41.773405+0000","last_fullsized":"2024-05-09T05:23:41.773405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:33.840344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26876,"num_write_kb":308483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.130335+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:22:59.130335+0000","last_peered":"2024-05-09T05:22:59.130335+0000","last_clean":"2024-05-09T05:22:59.130335+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:22:59.130335+0000","last_undegraded":"2024-05-09T05:22:59.130335+0000","last_fullsized":"2024-05-09T05:22:59.130335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'4642","reported_seq":4666,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.787942+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:23:41.787942+0000","last_peered":"2024-05-09T05:23:41.787942+0000","last_clean":"2024-05-09T05:23:41.787942+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:23:41.787942+0000","last_undegraded":"2024-05-09T05:23:41.787942+0000","last_fullsized":"2024-05-09T05:23:41.787942+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:50.236344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22705,"num_write_kb":262889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'5","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.155406+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:22:59.155406+0000","last_peered":"2024-05-09T05:22:59.155406+0000","last_clean":"2024-05-09T05:22:59.155406+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:22:59.155406+0000","last_undegraded":"2024-05-09T05:22:59.155406+0000","last_fullsized":"2024-05-09T05:22:59.155406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'5215","reported_seq":5239,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790504+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:23:41.790504+0000","last_peered":"2024-05-09T05:23:41.790504+0000","last_clean":"2024-05-09T05:23:41.790504+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:23:41.790504+0000","last_undegraded":"2024-05-09T05:23:41.790504+0000","last_fullsized":"2024-05-09T05:23:41.790504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25185,"num_write_kb":293431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.138117+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:22:59.138117+0000","last_peered":"2024-05-09T05:22:59.138117+0000","last_clean":"2024-05-09T05:22:59.138117+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:22:59.138117+0000","last_undegraded":"2024-05-09T05:22:59.138117+0000","last_fullsized":"2024-05-09T05:22:59.138117+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'5538","reported_seq":5562,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803051+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:23:41.803051+0000","last_peered":"2024-05-09T05:23:41.803051+0000","last_clean":"2024-05-09T05:23:41.803051+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:23:41.803051+0000","last_undegraded":"2024-05-09T05:23:41.803051+0000","last_fullsized":"2024-05-09T05:23:41.803051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27899,"num_write_kb":316727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"23'6","reported_seq":32,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.135039+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:22:59.135039+0000","last_peered":"2024-05-09T05:22:59.135039+0000","last_clean":"2024-05-09T05:22:59.135039+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:22:59.135039+0000","last_undegraded":"2024-05-09T05:22:59.135039+0000","last_fullsized":"2024-05-09T05:22:59.135039+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:52.816983+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'5078","reported_seq":5101,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.813122+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:23:41.813122+0000","last_peered":"2024-05-09T05:23:41.813122+0000","last_clean":"2024-05-09T05:23:41.813122+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:23:41.813122+0000","last_undegraded":"2024-05-09T05:23:41.813122+0000","last_fullsized":"2024-05-09T05:23:41.813122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:58:35.651549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24494,"num_write_kb":270229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1097","reported_seq":1122,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.133596+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:22:59.133596+0000","last_peered":"2024-05-09T05:22:59.133596+0000","last_clean":"2024-05-09T05:22:59.133596+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:22:59.133596+0000","last_undegraded":"2024-05-09T05:22:59.133596+0000","last_fullsized":"2024-05-09T05:22:59.133596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1089,"num_write_kb":4419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'4956","reported_seq":4980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.764163+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:23:41.764163+0000","last_peered":"2024-05-09T05:23:41.764163+0000","last_clean":"2024-05-09T05:23:41.764163+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:23:41.764163+0000","last_undegraded":"2024-05-09T05:23:41.764163+0000","last_fullsized":"2024-05-09T05:23:41.764163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24725,"num_write_kb":283470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'6745","reported_seq":6768,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.926010+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:23:41.926010+0000","last_peered":"2024-05-09T05:23:41.926010+0000","last_clean":"2024-05-09T05:23:41.926010+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:23:41.926010+0000","last_undegraded":"2024-05-09T05:23:41.926010+0000","last_fullsized":"2024-05-09T05:23:41.926010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33228,"num_write_kb":379009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.163507+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:22:59.163507+0000","last_peered":"2024-05-09T05:22:59.163507+0000","last_clean":"2024-05-09T05:22:59.163507+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:22:59.163507+0000","last_undegraded":"2024-05-09T05:22:59.163507+0000","last_fullsized":"2024-05-09T05:22:59.163507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'4185","reported_seq":4201,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.772240+0000","last_change":"2024-05-09T05:07:01.723135+0000","last_active":"2024-05-09T05:23:41.772240+0000","last_peered":"2024-05-09T05:23:41.772240+0000","last_clean":"2024-05-09T05:23:41.772240+0000","last_became_active":"2024-05-09T05:07:01.722785+0000","last_became_peered":"2024-05-09T05:07:01.722785+0000","last_unstale":"2024-05-09T05:23:41.772240+0000","last_undegraded":"2024-05-09T05:23:41.772240+0000","last_fullsized":"2024-05-09T05:23:41.772240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:57:53.799866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20471,"num_write_kb":229445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'21","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.166888+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:22:59.166888+0000","last_peered":"2024-05-09T05:22:59.166888+0000","last_clean":"2024-05-09T05:22:59.166888+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:22:59.166888+0000","last_undegraded":"2024-05-09T05:22:59.166888+0000","last_fullsized":"2024-05-09T05:22:59.166888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'6872","reported_seq":6896,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.883263+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:23:41.883263+0000","last_peered":"2024-05-09T05:23:41.883263+0000","last_clean":"2024-05-09T05:23:41.883263+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:23:41.883263+0000","last_undegraded":"2024-05-09T05:23:41.883263+0000","last_fullsized":"2024-05-09T05:23:41.883263+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:32:42.457145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34844,"num_write_kb":385600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'9","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.127515+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:22:59.127515+0000","last_peered":"2024-05-09T05:22:59.127515+0000","last_clean":"2024-05-09T05:22:59.127515+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:22:59.127515+0000","last_undegraded":"2024-05-09T05:22:59.127515+0000","last_fullsized":"2024-05-09T05:22:59.127515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'5811","reported_seq":5835,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790466+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:23:41.790466+0000","last_peered":"2024-05-09T05:23:41.790466+0000","last_clean":"2024-05-09T05:23:41.790466+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:23:41.790466+0000","last_undegraded":"2024-05-09T05:23:41.790466+0000","last_fullsized":"2024-05-09T05:23:41.790466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:32:29.331565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29692,"num_write_kb":336226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'740","reported_seq":766,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:43.840338+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:22:43.840338+0000","last_peered":"2024-05-09T05:22:43.840338+0000","last_clean":"2024-05-09T05:22:43.840338+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:22:43.840338+0000","last_undegraded":"2024-05-09T05:22:43.840338+0000","last_fullsized":"2024-05-09T05:22:43.840338+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:10:29.471489+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":739,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1808567,"num_write_kb":20359177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":362904,"ondisk_log_size":362904,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18590025,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":4539,"num_write_kb":19914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37421056,"data_stored":37180050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3759,"ondisk_log_size":3759,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509657,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729492,"kb_used_data":13132,"kb_used_omap":1,"kb_used_meta":716350,"kb_avail":93642348,"statfs":{"total":96636764160,"available":95889764352,"internally_reserved":0,"allocated":13447168,"data_stored":13261681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":733542858},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509658,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702748,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":702462,"kb_avail":93669092,"statfs":{"total":96636764160,"available":95917150208,"internally_reserved":0,"allocated":282624,"data_stored":90233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719321541},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542362,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384208,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":378174,"kb_avail":93987632,"statfs":{"total":96636764160,"available":96243335168,"internally_reserved":0,"allocated":6168576,"data_stored":5997855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387250630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542363,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698768,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":692734,"kb_avail":93673072,"statfs":{"total":96636764160,"available":95921225728,"internally_reserved":0,"allocated":6168576,"data_stored":5997723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":709360069},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542363,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679136,"kb_used_data":8536,"kb_used_omap":1,"kb_used_meta":670590,"kb_avail":93692704,"statfs":{"total":96636764160,"available":95941328896,"internally_reserved":0,"allocated":8740864,"data_stored":8564586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":686684616},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542364,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400676,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":400318,"kb_avail":93971164,"statfs":{"total":96636764160,"available":96226471936,"internally_reserved":0,"allocated":356352,"data_stored":175944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":409926085},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542363,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":446014,"kb_avail":93925560,"statfs":{"total":96636764160,"available":96179773440,"internally_reserved":0,"allocated":262144,"data_stored":87251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456718789},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542362,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675468,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":670526,"kb_avail":93696372,"statfs":{"total":96636764160,"available":95945084928,"internally_reserved":0,"allocated":5050368,"data_stored":4871265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":686619077},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4222976,"data_stored":4195178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":90225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8507392,"data_stored":8478867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12619776,"data_stored":12585594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:23:49.131 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:23:49.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.286+0000 7fb47fc27640 1 Processor -- start 2024-05-09T05:23:49.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.286+0000 7fb47fc27640 1 -- start start 2024-05-09T05:23:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47fc27640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb47815ca90 0x7fb47815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47fc27640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb47806f6d0 con 0x7fb47806f1b0 2024-05-09T05:23:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47fc27640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fb4780cc4d0 con 0x7fb4780d16b0 2024-05-09T05:23:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47fc27640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fb4780cee50 con 0x7fb47815ca90 2024-05-09T05:23:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47e1c4640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fb47815ca90 0x7fb47815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47d1c2640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fb47806f1b0 0x7fb4780ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:58810/0 (socket says 172.21.15.1:58810) 2024-05-09T05:23:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47d1c2640 1 -- 172.21.15.1:0/706879424 learned_addr learned my addr 172.21.15.1:0/706879424 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:49.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47e1c4640 1 -- 172.21.15.1:0/706879424 >> v1:172.21.15.196:6789/0 conn(0x7fb4780d16b0 legacy=0x7fb4780d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47e1c4640 1 -- 172.21.15.1:0/706879424 >> v1:172.21.15.1:6789/0 conn(0x7fb47806f1b0 legacy=0x7fb4780ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.288+0000 7fb47e1c4640 1 -- 172.21.15.1:0/706879424 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4780cc7b0 con 0x7fb47815ca90 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.289+0000 7fb47e1c4640 1 --2- 172.21.15.1:0/706879424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb47815cea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb470009d90 tx=0x7fb47002a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1939b91254645260 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.289+0000 7fb47c9c1640 1 -- 172.21.15.1:0/706879424 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470037070 con 0x7fb47815ca90 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.289+0000 7fb47c9c1640 1 -- 172.21.15.1:0/706879424 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb470007740 con 0x7fb47815ca90 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.289+0000 7fb47c9c1640 1 -- 172.21.15.1:0/706879424 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470007b10 con 0x7fb47815ca90 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.290+0000 7fb47fc27640 1 -- 172.21.15.1:0/706879424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 msgr2=0x7fb47815cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.290+0000 7fb47fc27640 1 --2- 172.21.15.1:0/706879424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb47815cea0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb470009d90 tx=0x7fb47002a780 comp rx=0 tx=0).stop 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.290+0000 7fb47fc27640 1 -- 172.21.15.1:0/706879424 shutdown_connections 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.290+0000 7fb47fc27640 1 --2- 172.21.15.1:0/706879424 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb47815cea0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.290+0000 7fb47fc27640 1 -- 172.21.15.1:0/706879424 >> 172.21.15.1:0/706879424 conn(0x7fb4780b3660 msgr2=0x7fb4780b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:49.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.290+0000 7fb47fc27640 1 -- 172.21.15.1:0/706879424 shutdown_connections 2024-05-09T05:23:49.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.290+0000 7fb47fc27640 1 -- 172.21.15.1:0/706879424 wait complete. 2024-05-09T05:23:49.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.292+0000 7fb47fc27640 1 Processor -- start 2024-05-09T05:23:49.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.292+0000 7fb47fc27640 1 -- start start 2024-05-09T05:23:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47fc27640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb47806f1b0 0x7fb4780c9a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47fc27640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4780d16b0 0x7fb4781f5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47fc27640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb4781f62d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47fc27640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fb4780ccd70 con 0x7fb47815ca90 2024-05-09T05:23:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47fc27640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fb4780cd0e0 con 0x7fb4780d16b0 2024-05-09T05:23:49.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47fc27640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fb47815d3e0 con 0x7fb47806f1b0 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47e1c4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb4781f62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47e1c4640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb4781f62d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:60338/0 (socket says 172.21.15.1:60338) 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47e1c4640 1 -- 172.21.15.1:0/2414011292 learned_addr learned my addr 172.21.15.1:0/2414011292 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47d1c2640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4780d16b0 0x7fb4781f5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.293+0000 7fb47d9c3640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb47806f1b0 0x7fb4780c9a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb47e1c4640 1 -- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb47806f1b0 msgr2=0x7fb4780c9a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb47e1c4640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb47806f1b0 0x7fb4780c9a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb47e1c4640 1 -- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4780d16b0 msgr2=0x7fb4781f5d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb47e1c4640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4780d16b0 0x7fb4781f5d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb47e1c4640 1 -- 172.21.15.1:0/2414011292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4780c7e90 con 0x7fb47815ca90 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb47e1c4640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb4781f62d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb470009eb0 tx=0x7fb470004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb46a7fc640 1 -- 172.21.15.1:0/2414011292 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470042020 con 0x7fb47815ca90 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb46a7fc640 1 -- 172.21.15.1:0/2414011292 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fb470032990 con 0x7fb47815ca90 2024-05-09T05:23:49.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4780c9f70 con 0x7fb47815ca90 2024-05-09T05:23:49.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb46a7fc640 1 -- 172.21.15.1:0/2414011292 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb470037040 con 0x7fb47815ca90 2024-05-09T05:23:49.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.294+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4780c9230 con 0x7fb47815ca90 2024-05-09T05:23:49.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.295+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb478050870 con 0x7fb47815ca90 2024-05-09T05:23:49.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.296+0000 7fb46a7fc640 1 -- 172.21.15.1:0/2414011292 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fb470045020 con 0x7fb47815ca90 2024-05-09T05:23:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.296+0000 7fb46a7fc640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb45c076300 0x7fb45c078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.297+0000 7fb46a7fc640 1 -- 172.21.15.1:0/2414011292 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fb4700b8cc0 con 0x7fb47815ca90 2024-05-09T05:23:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.297+0000 7fb47d9c3640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb45c076300 0x7fb45c078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:49.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.297+0000 7fb47d9c3640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb45c076300 0x7fb45c078810 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb4740015e0 tx=0x7fb474009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:49.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.300+0000 7fb46a7fc640 1 -- 172.21.15.1:0/2414011292 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fb47004c030 con 0x7fb47815ca90 2024-05-09T05:23:49.490 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.493+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fb4780b5170 con 0x7fb47815ca90 2024-05-09T05:23:49.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.498+0000 7fb46a7fc640 1 -- 172.21.15.1:0/2414011292 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16795 (secure 0 0 0) 0x7fb470082680 con 0x7fb47815ca90 2024-05-09T05:23:49.495 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:23:49.495 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:07:01.648910+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:23:49.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb45c076300 msgr2=0x7fb45c078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb45c076300 0x7fb45c078810 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fb4740015e0 tx=0x7fb474009040 comp rx=0 tx=0).stop 2024-05-09T05:23:49.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 msgr2=0x7fb4781f62d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:49.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb4781f62d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb470009eb0 tx=0x7fb470004a40 comp rx=0 tx=0).stop 2024-05-09T05:23:49.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 shutdown_connections 2024-05-09T05:23:49.498 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fb45c076300 0x7fb45c078810 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fb4780d16b0 0x7fb4781f5d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fb47815ca90 0x7fb4781f62d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 --2- 172.21.15.1:0/2414011292 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fb47806f1b0 0x7fb4780c9a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.500+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 >> 172.21.15.1:0/2414011292 conn(0x7fb4780b3660 msgr2=0x7fb4780b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.501+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 shutdown_connections 2024-05-09T05:23:49.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:49.501+0000 7fb47fc27640 1 -- 172.21.15.1:0/2414011292 wait complete. 2024-05-09T05:23:50.510 INFO:tasks.ceph:Scrubbing osd.0 2024-05-09T05:23:50.511 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:23:50.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.634+0000 7f2e22ee0640 1 Processor -- start 2024-05-09T05:23:50.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.634+0000 7f2e22ee0640 1 -- start start 2024-05-09T05:23:50.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.635+0000 7f2e22ee0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0ca970 0x7f2e1c153730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.635+0000 7f2e22ee0640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e1c0c8a50 con 0x7f2e1c0c67b0 2024-05-09T05:23:50.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.635+0000 7f2e22ee0640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f2e1c0c8d10 con 0x7f2e1c0cae10 2024-05-09T05:23:50.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.635+0000 7f2e22ee0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1c0c9d00 con 0x7f2e1c0ca970 2024-05-09T05:23:50.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.635+0000 7f2e2147d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f2e1c0c67b0 0x7f2e1c163d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55252/0 (socket says 172.21.15.1:55252) 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.635+0000 7f2e2147d640 1 -- 172.21.15.1:0/4286903338 learned_addr learned my addr 172.21.15.1:0/4286903338 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.635+0000 7f2e20c7c640 1 --2- 172.21.15.1:0/4286903338 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0ca970 0x7f2e1c153730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.636+0000 7f2e1b7fe640 1 -- 172.21.15.1:0/4286903338 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2850519998 0 0) 0x7f2e1c0c8a50 con 0x7f2e1c0c67b0 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.636+0000 7f2e1b7fe640 1 -- 172.21.15.1:0/4286903338 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e0c003520 con 0x7f2e1c0c67b0 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.636+0000 7f2e1b7fe640 1 -- 172.21.15.1:0/4286903338 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1468591349 0 0) 0x7f2e1c0c8d10 con 0x7f2e1c0cae10 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.636+0000 7f2e1b7fe640 1 -- 172.21.15.1:0/4286903338 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f2e1c0c8a50 con 0x7f2e1c0cae10 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.636+0000 7f2e20c7c640 1 -- 172.21.15.1:0/4286903338 >> v1:172.21.15.196:6789/0 conn(0x7f2e1c0cae10 legacy=0x7f2e1c15c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.636+0000 7f2e20c7c640 1 -- 172.21.15.1:0/4286903338 >> v1:172.21.15.1:6789/0 conn(0x7f2e1c0c67b0 legacy=0x7f2e1c163d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.636+0000 7f2e1b7fe640 1 -- 172.21.15.1:0/4286903338 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3939243160 0 0) 0x7f2e0c003520 con 0x7f2e1c0c67b0 2024-05-09T05:23:50.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.636+0000 7f2e20c7c640 1 -- 172.21.15.1:0/4286903338 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c0c5e10 con 0x7f2e1c0ca970 2024-05-09T05:23:50.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.637+0000 7f2e20c7c640 1 --2- 172.21.15.1:0/4286903338 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0ca970 0x7f2e1c153730 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e100097d0 tx=0x7f2e1002a1a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49ca00de32d72c75 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:50.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.637+0000 7f2e1b7fe640 1 -- 172.21.15.1:0/4286903338 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e10037070 con 0x7f2e1c0ca970 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.637+0000 7f2e1b7fe640 1 -- 172.21.15.1:0/4286903338 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e1002ac30 con 0x7f2e1c0ca970 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.637+0000 7f2e1b7fe640 1 -- 172.21.15.1:0/4286903338 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e14001a70 con 0x7f2e1c0ca970 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.638+0000 7f2e22ee0640 1 -- 172.21.15.1:0/4286903338 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0ca970 msgr2=0x7f2e1c153730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.638+0000 7f2e22ee0640 1 --2- 172.21.15.1:0/4286903338 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0ca970 0x7f2e1c153730 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f2e100097d0 tx=0x7f2e1002a1a0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.638+0000 7f2e22ee0640 1 -- 172.21.15.1:0/4286903338 shutdown_connections 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.638+0000 7f2e22ee0640 1 --2- 172.21.15.1:0/4286903338 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0ca970 0x7f2e1c153730 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.638+0000 7f2e22ee0640 1 -- 172.21.15.1:0/4286903338 >> 172.21.15.1:0/4286903338 conn(0x7f2e1c0ce910 msgr2=0x7f2e1c0ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.638+0000 7f2e22ee0640 1 -- 172.21.15.1:0/4286903338 shutdown_connections 2024-05-09T05:23:50.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.638+0000 7f2e22ee0640 1 -- 172.21.15.1:0/4286903338 wait complete. 2024-05-09T05:23:50.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.640+0000 7f2e22ee0640 1 Processor -- start 2024-05-09T05:23:50.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.640+0000 7f2e22ee0640 1 -- start start 2024-05-09T05:23:50.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e22ee0640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c0c67b0 0x7f2e1c144e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e22ee0640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e1c0ca970 0x7f2e1c1e27d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e22ee0640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0cae10 0x7f2e1c1e2d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e22ee0640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c0c63d0 con 0x7f2e1c0ca970 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e22ee0640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f2e1c0c9870 con 0x7f2e1c0c67b0 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e22ee0640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f2e1c0c9500 con 0x7f2e1c0cae10 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e1bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e1c0ca970 0x7f2e1c1e27d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e1bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e1c0ca970 0x7f2e1c1e27d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37264/0 (socket says 172.21.15.1:37264) 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e1bfff640 1 -- 172.21.15.1:0/661849668 learned_addr learned my addr 172.21.15.1:0/661849668 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e2147d640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0cae10 0x7f2e1c1e2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.641+0000 7f2e20c7c640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c0c67b0 0x7f2e1c144e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e1bfff640 1 -- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0cae10 msgr2=0x7f2e1c1e2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e1bfff640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f2e1c0cae10 0x7f2e1c1e2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.640 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e1bfff640 1 -- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c0c67b0 msgr2=0x7f2e1c144e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e1bfff640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f2e1c0c67b0 0x7f2e1c144e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e1bfff640 1 -- 172.21.15.1:0/661849668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e1c1f6590 con 0x7f2e1c0ca970 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e1bfff640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e1c0ca970 0x7f2e1c1e27d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800c430 tx=0x7f2e0800c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e08017070 con 0x7f2e1c0ca970 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f2e080052f0 con 0x7f2e1c0ca970 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e1c154780 con 0x7f2e1c0ca970 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e08003830 con 0x7f2e1c0ca970 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.642+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e1c154cc0 con 0x7f2e1c0ca970 2024-05-09T05:23:50.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.643+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f2e1c147d30 con 0x7f2e1c0ca970 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.644+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f2e0801a070 con 0x7f2e1c0ca970 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e197fa640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2e000762d0 0x7f2e000787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f2e08097080 con 0x7f2e1c0ca970 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e197fa640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7f2e0007e6a0 0x7f2e00080b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 --> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2e0801dec0 con 0x7f2e0007e6a0 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f2e08097430 con 0x7f2e1c0ca970 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e20c7c640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2e000762d0 0x7f2e000787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e2147d640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7f2e0007e6a0 0x7f2e00080b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e2147d640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7f2e0007e6a0 0x7f2e00080b70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.645+0000 7f2e20c7c640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2e000762d0 0x7f2e000787e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2e10034000 tx=0x7f2e1002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:50.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.646+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 <== osd.0 v2:172.21.15.1:6824/1100742567 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2e0801dec0 con 0x7f2e0007e6a0 2024-05-09T05:23:50.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.659+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 --> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f2e1c050870 con 0x7f2e0007e6a0 2024-05-09T05:23:50.657 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e197fa640 1 -- 172.21.15.1:0/661849668 <== osd.0 v2:172.21.15.1:6824/1100742567 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f2e1c050870 con 0x7f2e0007e6a0 2024-05-09T05:23:50.657 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:50.657 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:23:50.657 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7f2e0007e6a0 msgr2=0x7f2e00080b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.1:6824/1100742567,v1:172.21.15.1:6825/1100742567] conn(0x7f2e0007e6a0 0x7f2e00080b70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2e000762d0 msgr2=0x7f2e000787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f2e000762d0 0x7f2e000787e0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f2e10034000 tx=0x7f2e1002e000 comp rx=0 tx=0).stop 2024-05-09T05:23:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e1c0ca970 msgr2=0x7f2e1c1e27d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 --2- 172.21.15.1:0/661849668 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f2e1c0ca970 0x7f2e1c1e27d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2e0800c430 tx=0x7f2e0800c8f0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e2147d640 1 -- 172.21.15.1:0/661849668 reap_dead start 2024-05-09T05:23:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 shutdown_connections 2024-05-09T05:23:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 >> 172.21.15.1:0/661849668 conn(0x7f2e1c0ce910 msgr2=0x7f2e1c06c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 shutdown_connections 2024-05-09T05:23:50.659 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.660+0000 7f2e22ee0640 1 -- 172.21.15.1:0/661849668 wait complete. 2024-05-09T05:23:50.669 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2024-05-09T05:23:50.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.789+0000 7f6dcf2a7640 1 Processor -- start 2024-05-09T05:23:50.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.790+0000 7f6dcf2a7640 1 -- start start 2024-05-09T05:23:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcf2a7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c89a0 0x7f6dc80ca930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcf2a7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6dc80c4af0 0x7f6dc8153490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcf2a7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6dc80c3be0 con 0x7f6dc80c46e0 2024-05-09T05:23:50.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcf2a7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6dc80c9290 con 0x7f6dc80c4af0 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcf2a7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc806e890 con 0x7f6dc80c89a0 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcd844640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6dc80c4af0 0x7f6dc8153490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcc842640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6dc80c46e0 0x7f6dc8159650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55268/0 (socket says 172.21.15.1:55268) 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcc842640 1 -- 172.21.15.1:0/2202803274 learned_addr learned my addr 172.21.15.1:0/2202803274 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcd844640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6dc80c4af0 0x7f6dc8153490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37270/0 (socket says 172.21.15.1:37270) 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcd043640 1 --2- 172.21.15.1:0/2202803274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c89a0 0x7f6dc80ca930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcd844640 1 -- 172.21.15.1:0/2202803274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c89a0 msgr2=0x7f6dc80ca930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcd844640 1 --2- 172.21.15.1:0/2202803274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c89a0 0x7f6dc80ca930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcd844640 1 -- 172.21.15.1:0/2202803274 >> v1:172.21.15.1:6789/0 conn(0x7f6dc80c46e0 legacy=0x7f6dc8159650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dcd844640 1 -- 172.21.15.1:0/2202803274 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc80c3e60 con 0x7f6dc80c4af0 2024-05-09T05:23:50.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.791+0000 7f6dc7fff640 1 -- 172.21.15.1:0/2202803274 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2580867986 0 0) 0x7f6dbc001110 con 0x7f6dc80c46e0 2024-05-09T05:23:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.792+0000 7f6dcd844640 1 --2- 172.21.15.1:0/2202803274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc8153490 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000bfa0 tx=0x7f6dc002c7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18f3cdd578ce1b76 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:50.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.793+0000 7f6dc7fff640 1 -- 172.21.15.1:0/2202803274 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0038070 con 0x7f6dc80c4af0 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.793+0000 7f6dc7fff640 1 -- 172.21.15.1:0/2202803274 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dc0009540 con 0x7f6dc80c4af0 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.793+0000 7f6dc7fff640 1 -- 172.21.15.1:0/2202803274 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dc0009910 con 0x7f6dc80c4af0 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.793+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/2202803274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 msgr2=0x7f6dc8153490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.793+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/2202803274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc8153490 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f6dc000bfa0 tx=0x7f6dc002c7e0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.793+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/2202803274 shutdown_connections 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.794+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/2202803274 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc8153490 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.794+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/2202803274 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c89a0 0x7f6dc80ca930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.794+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/2202803274 >> 172.21.15.1:0/2202803274 conn(0x7f6dc80747b0 msgr2=0x7f6dc80ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.794+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/2202803274 shutdown_connections 2024-05-09T05:23:50.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.794+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/2202803274 wait complete. 2024-05-09T05:23:50.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.795+0000 7f6dcf2a7640 1 Processor -- start 2024-05-09T05:23:50.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.795+0000 7f6dcf2a7640 1 -- start start 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcf2a7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c46e0 0x7f6dc80b3f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcf2a7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc81f6450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcf2a7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc80c89a0 0x7f6dc81e3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcf2a7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc80c97c0 con 0x7f6dc80c4af0 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcf2a7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6dc80c94e0 con 0x7f6dc80c89a0 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcf2a7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6dc80cae70 con 0x7f6dc80c46e0 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc81f6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc81f6450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37272/0 (socket says 172.21.15.1:37272) 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 -- 172.21.15.1:0/154261090 learned_addr learned my addr 172.21.15.1:0/154261090 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcd043640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c46e0 0x7f6dc80b3f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcd844640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc80c89a0 0x7f6dc81e3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 -- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c46e0 msgr2=0x7f6dc80b3f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c46e0 0x7f6dc80b3f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 -- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc80c89a0 msgr2=0x7f6dc81e3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc80c89a0 0x7f6dc81e3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 -- 172.21.15.1:0/154261090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6dc80ab740 con 0x7f6dc80c4af0 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.797+0000 7f6dcc842640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc81f6450 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00c3c0 tx=0x7f6dbc00c880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.798+0000 7f6dc5ffb640 1 -- 172.21.15.1:0/154261090 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc0074e0 con 0x7f6dc80c4af0 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.798+0000 7f6dc5ffb640 1 -- 172.21.15.1:0/154261090 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6dbc0047e0 con 0x7f6dc80c4af0 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.798+0000 7f6dc5ffb640 1 -- 172.21.15.1:0/154261090 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6dbc004c10 con 0x7f6dc80c4af0 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.798+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6dc81f6990 con 0x7f6dc80c4af0 2024-05-09T05:23:50.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.798+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6dc80ce1b0 con 0x7f6dc80c4af0 2024-05-09T05:23:50.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.799+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6dc8050870 con 0x7f6dc80c4af0 2024-05-09T05:23:50.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.800+0000 7f6dc5ffb640 1 -- 172.21.15.1:0/154261090 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6dbc011070 con 0x7f6dc80c4af0 2024-05-09T05:23:50.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.800+0000 7f6dc5ffb640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6db40762d0 0x7f6db40787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:50.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.800+0000 7f6dc5ffb640 1 -- 172.21.15.1:0/154261090 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6dbc097040 con 0x7f6dc80c4af0 2024-05-09T05:23:50.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.800+0000 7f6dcd043640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6db40762d0 0x7f6db40787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:50.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.801+0000 7f6dcd043640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6db40762d0 0x7f6db40787e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6db80079a0 tx=0x7f6db8008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:50.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.809+0000 7f6dc5ffb640 1 -- 172.21.15.1:0/154261090 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6dc80ab740 con 0x7f6dc80c4af0 2024-05-09T05:23:50.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:50.999+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) -- 0x7f6dc80ca360 con 0x7f6db40762d0 2024-05-09T05:23:50.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.000+0000 7f6dc5ffb640 1 -- 172.21.15.1:0/154261090 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f6dc80ca360 con 0x7f6db40762d0 2024-05-09T05:23:50.996 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 0 to deep-scrub 2024-05-09T05:23:50.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6db40762d0 msgr2=0x7f6db40787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:50.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6db40762d0 0x7f6db40787e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6db80079a0 tx=0x7f6db8008040 comp rx=0 tx=0).stop 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 msgr2=0x7f6dc81f6450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc81f6450 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6dbc00c3c0 tx=0x7f6dbc00c880 comp rx=0 tx=0).stop 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 shutdown_connections 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6db40762d0 0x7f6db40787e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6dc80c89a0 0x7f6dc81e3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6dc80c4af0 0x7f6dc81f6450 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 --2- 172.21.15.1:0/154261090 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6dc80c46e0 0x7f6dc80b3f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 >> 172.21.15.1:0/154261090 conn(0x7f6dc80747b0 msgr2=0x7f6dc8151830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 shutdown_connections 2024-05-09T05:23:51.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.002+0000 7f6dcf2a7640 1 -- 172.21.15.1:0/154261090 wait complete. 2024-05-09T05:23:51.011 INFO:tasks.ceph:Scrubbing osd.1 2024-05-09T05:23:51.011 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:23:51.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.134+0000 7f91a4e0c640 1 Processor -- start 2024-05-09T05:23:51.131 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.134+0000 7f91a4e0c640 1 -- start start 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f91a4e0c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f91a00c8b70 0x7f91a0156b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f91a4e0c640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f91a00c9520 0x7f91a00717f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f91a4e0c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 0x7f91a0159700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f91a4e0c640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f91a00cb450 con 0x7f91a00c9520 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f91a4e0c640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f91a00c9e50 con 0x7f91a00c8b70 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f91a4e0c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f91a00caed0 con 0x7f91a0071d30 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f9197fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f91a00c9520 0x7f91a00717f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f9197fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f91a00c9520 0x7f91a00717f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37278/0 (socket says 172.21.15.1:37278) 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.135+0000 7f9197fff640 1 -- 172.21.15.1:0/2654789756 learned_addr learned my addr 172.21.15.1:0/2654789756 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:51.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.136+0000 7f919ed9d640 1 --2- 172.21.15.1:0/2654789756 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 0x7f91a0159700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.136+0000 7f919e59c640 1 --2- 172.21.15.1:0/2654789756 >> v2:172.21.15.196:3300/0 conn(0x7f91a00c8b70 0x7f91a0156b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.136+0000 7f9197fff640 1 -- 172.21.15.1:0/2654789756 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 msgr2=0x7f91a0159700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:51.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.136+0000 7f9197fff640 1 --2- 172.21.15.1:0/2654789756 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 0x7f91a0159700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.136+0000 7f9197fff640 1 -- 172.21.15.1:0/2654789756 >> v2:172.21.15.196:3300/0 conn(0x7f91a00c8b70 msgr2=0x7f91a0156b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:51.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.136+0000 7f9197fff640 1 --2- 172.21.15.1:0/2654789756 >> v2:172.21.15.196:3300/0 conn(0x7f91a00c8b70 0x7f91a0156b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.136+0000 7f9197fff640 1 -- 172.21.15.1:0/2654789756 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a00cb170 con 0x7f91a00c9520 2024-05-09T05:23:51.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.136+0000 7f9197fff640 1 --2- 172.21.15.1:0/2654789756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c9520 0x7f91a00717f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91a0073740 tx=0x7f918802c890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f9dd4f6e79b5d13 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.137+0000 7f919dd9b640 1 -- 172.21.15.1:0/2654789756 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188032790 con 0x7f91a00c9520 2024-05-09T05:23:51.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.137+0000 7f919dd9b640 1 -- 172.21.15.1:0/2654789756 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f9188038070 con 0x7f91a00c9520 2024-05-09T05:23:51.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.137+0000 7f919dd9b640 1 -- 172.21.15.1:0/2654789756 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188032b50 con 0x7f91a00c9520 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.137+0000 7f91a4e0c640 1 -- 172.21.15.1:0/2654789756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c9520 msgr2=0x7f91a00717f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.137+0000 7f91a4e0c640 1 --2- 172.21.15.1:0/2654789756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c9520 0x7f91a00717f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f91a0073740 tx=0x7f918802c890 comp rx=0 tx=0).stop 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.138+0000 7f91a4e0c640 1 -- 172.21.15.1:0/2654789756 shutdown_connections 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.138+0000 7f91a4e0c640 1 --2- 172.21.15.1:0/2654789756 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 0x7f91a0159700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.138+0000 7f91a4e0c640 1 --2- 172.21.15.1:0/2654789756 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c9520 0x7f91a00717f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.138+0000 7f91a4e0c640 1 --2- 172.21.15.1:0/2654789756 >> v2:172.21.15.196:3300/0 conn(0x7f91a00c8b70 0x7f91a0156b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.138+0000 7f91a4e0c640 1 -- 172.21.15.1:0/2654789756 >> 172.21.15.1:0/2654789756 conn(0x7f91a006d800 msgr2=0x7f91a006dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.138+0000 7f91a4e0c640 1 -- 172.21.15.1:0/2654789756 shutdown_connections 2024-05-09T05:23:51.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.138+0000 7f91a4e0c640 1 -- 172.21.15.1:0/2654789756 wait complete. 2024-05-09T05:23:51.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.139+0000 7f91a4e0c640 1 Processor -- start 2024-05-09T05:23:51.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.140+0000 7f91a4e0c640 1 -- start start 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f91a4e0c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 0x7f91a0154970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f91a4e0c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c8b70 0x7f91a01e1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f91a4e0c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91a00c9520 0x7f91a014e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f91a4e0c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f91a0071210 con 0x7f91a00c8b70 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f91a4e0c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f91a00c3f30 con 0x7f91a00c9520 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f91a4e0c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f91a01570d0 con 0x7f91a0071d30 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f9197fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c8b70 0x7f91a01e1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f9197fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c8b70 0x7f91a01e1af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37294/0 (socket says 172.21.15.1:37294) 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f9197fff640 1 -- 172.21.15.1:0/3598404521 learned_addr learned my addr 172.21.15.1:0/3598404521 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f919e59c640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 0x7f91a0154970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f919ed9d640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91a00c9520 0x7f91a014e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f9197fff640 1 -- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 msgr2=0x7f91a0154970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f9197fff640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f91a0071d30 0x7f91a0154970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f9197fff640 1 -- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91a00c9520 msgr2=0x7f91a014e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f9197fff640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f91a00c9520 0x7f91a014e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.141+0000 7f9197fff640 1 -- 172.21.15.1:0/3598404521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91a01ee4f0 con 0x7f91a00c8b70 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.142+0000 7f9197fff640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c8b70 0x7f91a01e1af0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f918802c8d0 tx=0x7f91880046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.142+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188033620 con 0x7f91a00c8b70 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.142+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f91880337c0 con 0x7f91a00c8b70 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.142+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9188033bb0 con 0x7f91a00c8b70 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.142+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91a01ee7d0 con 0x7f91a00c8b70 2024-05-09T05:23:51.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.142+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91a01eed90 con 0x7f91a00c8b70 2024-05-09T05:23:51.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.143+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f91a0154eb0 con 0x7f91a00c8b70 2024-05-09T05:23:51.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.144+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f91880463f0 con 0x7f91a00c8b70 2024-05-09T05:23:51.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.144+0000 7f9196ffd640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9180076300 0x7f9180078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.144+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f91880bad80 con 0x7f91a00c8b70 2024-05-09T05:23:51.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.144+0000 7f9196ffd640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f918007e6d0 0x7f9180080ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.144+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 --> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f9188001d30 con 0x7f918007e6d0 2024-05-09T05:23:51.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.144+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f91880bb210 con 0x7f91a00c8b70 2024-05-09T05:23:51.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.144+0000 7f919e59c640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9180076300 0x7f9180078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.145+0000 7f919ed9d640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f918007e6d0 0x7f9180080ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.145+0000 7f919ed9d640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f918007e6d0 0x7f9180080ba0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.145+0000 7f919e59c640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9180076300 0x7f9180078810 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f91900079a0 tx=0x7f9190008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.146+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 <== osd.1 v2:172.21.15.1:6811/2455090880 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f9188001d30 con 0x7f918007e6d0 2024-05-09T05:23:51.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.172+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 --> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f91a01e23a0 con 0x7f918007e6d0 2024-05-09T05:23:51.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f9196ffd640 1 -- 172.21.15.1:0/3598404521 <== osd.1 v2:172.21.15.1:6811/2455090880 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f91a01e23a0 con 0x7f918007e6d0 2024-05-09T05:23:51.170 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:51.170 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:23:51.170 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f918007e6d0 msgr2=0x7f9180080ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.1:6811/2455090880,v1:172.21.15.1:6813/2455090880] conn(0x7f918007e6d0 0x7f9180080ba0 crc :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9180076300 msgr2=0x7f9180078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f9180076300 0x7f9180078810 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f91900079a0 tx=0x7f9190008040 comp rx=0 tx=0).stop 2024-05-09T05:23:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c8b70 msgr2=0x7f91a01e1af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 --2- 172.21.15.1:0/3598404521 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f91a00c8b70 0x7f91a01e1af0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f918802c8d0 tx=0x7f91880046c0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.171 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f919ed9d640 1 -- 172.21.15.1:0/3598404521 reap_dead start 2024-05-09T05:23:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 shutdown_connections 2024-05-09T05:23:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 >> 172.21.15.1:0/3598404521 conn(0x7f91a006d800 msgr2=0x7f91a0153a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 shutdown_connections 2024-05-09T05:23:51.172 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.173+0000 7f91a4e0c640 1 -- 172.21.15.1:0/3598404521 wait complete. 2024-05-09T05:23:51.185 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2024-05-09T05:23:51.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.304+0000 7ffb99c92640 1 Processor -- start 2024-05-09T05:23:51.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.304+0000 7ffb99c92640 1 -- start start 2024-05-09T05:23:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb99c92640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9415d130 0x7ffb941569c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb99c92640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ffb940c48c0 0x7ffb940c4cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb99c92640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffb940c3ce0 con 0x7ffb9406c6f0 2024-05-09T05:23:51.303 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb99c92640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ffb9415d540 con 0x7ffb940c48c0 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb99c92640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffb940c3fc0 con 0x7ffb9415d130 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb937fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9415d130 0x7ffb941569c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb937fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9415d130 0x7ffb941569c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59262/0 (socket says 172.21.15.1:59262) 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb937fe640 1 -- 172.21.15.1:0/1421737299 learned_addr learned my addr 172.21.15.1:0/1421737299 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb93fff640 1 --2- 172.21.15.1:0/1421737299 >> v2:172.21.15.196:3300/0 conn(0x7ffb940c48c0 0x7ffb940c4cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb937fe640 1 -- 172.21.15.1:0/1421737299 >> v2:172.21.15.196:3300/0 conn(0x7ffb940c48c0 msgr2=0x7ffb940c4cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb937fe640 1 --2- 172.21.15.1:0/1421737299 >> v2:172.21.15.196:3300/0 conn(0x7ffb940c48c0 0x7ffb940c4cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb937fe640 1 -- 172.21.15.1:0/1421737299 >> v1:172.21.15.196:6789/0 conn(0x7ffb9406c6f0 legacy=0x7ffb940716a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.306+0000 7ffb937fe640 1 -- 172.21.15.1:0/1421737299 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb9406e4a0 con 0x7ffb9415d130 2024-05-09T05:23:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.307+0000 7ffb937fe640 1 --2- 172.21.15.1:0/1421737299 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9415d130 0x7ffb941569c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009870 tx=0x7ffb8402a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=238e1a397e7fa8e4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.307+0000 7ffb927fc640 1 -- 172.21.15.1:0/1421737299 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb84037070 con 0x7ffb9415d130 2024-05-09T05:23:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.308+0000 7ffb927fc640 1 -- 172.21.15.1:0/1421737299 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb84032440 con 0x7ffb9415d130 2024-05-09T05:23:51.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.308+0000 7ffb927fc640 1 -- 172.21.15.1:0/1421737299 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb84032810 con 0x7ffb9415d130 2024-05-09T05:23:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.308+0000 7ffb99c92640 1 -- 172.21.15.1:0/1421737299 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9415d130 msgr2=0x7ffb941569c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.308+0000 7ffb99c92640 1 --2- 172.21.15.1:0/1421737299 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9415d130 0x7ffb941569c0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffb84009870 tx=0x7ffb8402a270 comp rx=0 tx=0).stop 2024-05-09T05:23:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.309+0000 7ffb99c92640 1 -- 172.21.15.1:0/1421737299 shutdown_connections 2024-05-09T05:23:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.309+0000 7ffb99c92640 1 --2- 172.21.15.1:0/1421737299 >> v2:172.21.15.196:3300/0 conn(0x7ffb940c48c0 0x7ffb940c4cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.309+0000 7ffb99c92640 1 --2- 172.21.15.1:0/1421737299 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9415d130 0x7ffb941569c0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.309+0000 7ffb99c92640 1 -- 172.21.15.1:0/1421737299 >> 172.21.15.1:0/1421737299 conn(0x7ffb940747b0 msgr2=0x7ffb940ce910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.309+0000 7ffb99c92640 1 -- 172.21.15.1:0/1421737299 shutdown_connections 2024-05-09T05:23:51.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.309+0000 7ffb99c92640 1 -- 172.21.15.1:0/1421737299 wait complete. 2024-05-09T05:23:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.310+0000 7ffb99c92640 1 Processor -- start 2024-05-09T05:23:51.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.311+0000 7ffb99c92640 1 -- start start 2024-05-09T05:23:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb99c92640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9406c6f0 0x7ffb941f33c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb99c92640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb940c48c0 0x7ffb94071650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb99c92640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb9415d130 0x7ffb941e5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb99c92640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffb9406e1a0 con 0x7ffb9415d130 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb99c92640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffb940c5210 con 0x7ffb940c48c0 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb99c92640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffb9406cb00 con 0x7ffb9406c6f0 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb93fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb9415d130 0x7ffb941e5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb93fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb9415d130 0x7ffb941e5ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37300/0 (socket says 172.21.15.1:37300) 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb93fff640 1 -- 172.21.15.1:0/3554789598 learned_addr learned my addr 172.21.15.1:0/3554789598 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb937fe640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9406c6f0 0x7ffb941f33c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb92ffd640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb940c48c0 0x7ffb94071650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb93fff640 1 -- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9406c6f0 msgr2=0x7ffb941f33c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb93fff640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9406c6f0 0x7ffb941f33c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb93fff640 1 -- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb940c48c0 msgr2=0x7ffb94071650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb93fff640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb940c48c0 0x7ffb94071650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb93fff640 1 -- 172.21.15.1:0/3554789598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb941f3900 con 0x7ffb9415d130 2024-05-09T05:23:51.311 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.312+0000 7ffb92ffd640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb940c48c0 0x7ffb94071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.313+0000 7ffb93fff640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb9415d130 0x7ffb941e5ff0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009d30 tx=0x7ffb8c00c490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.313+0000 7ffb7bfff640 1 -- 172.21.15.1:0/3554789598 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c018020 con 0x7ffb9415d130 2024-05-09T05:23:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.313+0000 7ffb7bfff640 1 -- 172.21.15.1:0/3554789598 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ffb8c014480 con 0x7ffb9415d130 2024-05-09T05:23:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.313+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb9415dc40 con 0x7ffb9415d130 2024-05-09T05:23:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.313+0000 7ffb7bfff640 1 -- 172.21.15.1:0/3554789598 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb8c014870 con 0x7ffb9415d130 2024-05-09T05:23:51.312 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.313+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb9415df20 con 0x7ffb9415d130 2024-05-09T05:23:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.314+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb94050870 con 0x7ffb9415d130 2024-05-09T05:23:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.315+0000 7ffb7bfff640 1 -- 172.21.15.1:0/3554789598 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffb8c005d10 con 0x7ffb9415d130 2024-05-09T05:23:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.315+0000 7ffb7bfff640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffb800762d0 0x7ffb800787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.315+0000 7ffb7bfff640 1 -- 172.21.15.1:0/3554789598 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ffb8c098090 con 0x7ffb9415d130 2024-05-09T05:23:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.316+0000 7ffb937fe640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffb800762d0 0x7ffb800787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.313 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.316+0000 7ffb937fe640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffb800762d0 0x7ffb800787e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffb84034000 tx=0x7ffb8402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.317 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.320+0000 7ffb7bfff640 1 -- 172.21.15.1:0/3554789598 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffb8c022070 con 0x7ffb9415d130 2024-05-09T05:23:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.512+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) -- 0x7ffb941f6210 con 0x7ffb800762d0 2024-05-09T05:23:51.509 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.513+0000 7ffb7bfff640 1 -- 172.21.15.1:0/3554789598 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ffb941f6210 con 0x7ffb800762d0 2024-05-09T05:23:51.510 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 1 to deep-scrub 2024-05-09T05:23:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffb800762d0 msgr2=0x7ffb800787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffb800762d0 0x7ffb800787e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7ffb84034000 tx=0x7ffb8402e000 comp rx=0 tx=0).stop 2024-05-09T05:23:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb9415d130 msgr2=0x7ffb941e5ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb9415d130 0x7ffb941e5ff0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7ffb8c009d30 tx=0x7ffb8c00c490 comp rx=0 tx=0).stop 2024-05-09T05:23:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 shutdown_connections 2024-05-09T05:23:51.512 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffb800762d0 0x7ffb800787e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffb940c48c0 0x7ffb94071650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffb9415d130 0x7ffb941e5ff0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 --2- 172.21.15.1:0/3554789598 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffb9406c6f0 0x7ffb941f33c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 >> 172.21.15.1:0/3554789598 conn(0x7ffb940747b0 msgr2=0x7ffb94154130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:51.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 shutdown_connections 2024-05-09T05:23:51.513 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.515+0000 7ffb99c92640 1 -- 172.21.15.1:0/3554789598 wait complete. 2024-05-09T05:23:51.524 INFO:tasks.ceph:Scrubbing osd.2 2024-05-09T05:23:51.524 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:23:51.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.646+0000 7f3e75470640 1 Processor -- start 2024-05-09T05:23:51.643 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.646+0000 7f3e75470640 1 -- start start 2024-05-09T05:23:51.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.647+0000 7f3e75470640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e700ce010 0x7f3e700c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.647+0000 7f3e75470640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e7015c7b0 con 0x7f3e700c77c0 2024-05-09T05:23:51.647 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.647+0000 7f3e75470640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f3e700cca50 con 0x7f3e700b4ec0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.647+0000 7f3e75470640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3e7015d0c0 con 0x7f3e700ce010 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6f7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f3e700c77c0 0x7f3e700c7bd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55272/0 (socket says 172.21.15.1:55272) 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/2666560127 learned_addr learned my addr 172.21.15.1:0/2666560127 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6e7fc640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e700ce010 0x7f3e700c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3345860981 0 0) 0x7f3e7015c7b0 con 0x7f3e700c77c0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e58003520 con 0x7f3e700c77c0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4136628389 0 0) 0x7f3e700cca50 con 0x7f3e700b4ec0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f3e7015c7b0 con 0x7f3e700b4ec0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 208556227 0 0) 0x7f3e58003520 con 0x7f3e700c77c0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e700ce010 msgr2=0x7f3e700c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e700ce010 0x7f3e700c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 >> v1:172.21.15.196:6789/0 conn(0x7f3e700b4ec0 legacy=0x7f3e700cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e700ccd30 con 0x7f3e700c77c0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f3e68001a70 con 0x7f3e700c77c0 2024-05-09T05:23:51.648 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 >> v1:172.21.15.1:6789/0 conn(0x7f3e700c77c0 legacy=0x7f3e700c7bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e58003c10 0x7f3e58004060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e58004800 0x7f3e58004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e58005210 0x7f3e5800b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7015d0c0 con 0x7f3e58005210 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3e5800c0f0 con 0x7f3e58003c10 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3e5800c2f0 con 0x7f3e58004800 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6e7fc640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e700ce010 0x7f3e700c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.648+0000 7f3e6f7fe640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e58005210 0x7f3e5800b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.649+0000 7f3e6effd640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e58003c10 0x7f3e58004060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.649+0000 7f3e6e7fc640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e58004800 0x7f3e58004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.649+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e58004800 msgr2=0x7f3e58004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.649+0000 7f3e6f7fe640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e58004800 0x7f3e58004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.649+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e58003c10 msgr2=0x7f3e58004060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.649+0000 7f3e6f7fe640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e58003c10 0x7f3e58004060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.649+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/2666560127 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e58003520 con 0x7f3e58005210 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.649+0000 7f3e6f7fe640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e58005210 0x7f3e5800b9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e7006f5b0 tx=0x7f3e58005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9764f3eaec0229c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.649 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6800a410 con 0x7f3e58005210 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e6802f070 con 0x7f3e58005210 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e6dffb640 1 -- 172.21.15.1:0/2666560127 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6802bb60 con 0x7f3e58005210 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e75470640 1 -- 172.21.15.1:0/2666560127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e58005210 msgr2=0x7f3e5800b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e75470640 1 --2- 172.21.15.1:0/2666560127 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e58005210 0x7f3e5800b9c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f3e7006f5b0 tx=0x7f3e58005830 comp rx=0 tx=0).stop 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/2666560127 reap_dead start 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e75470640 1 -- 172.21.15.1:0/2666560127 shutdown_connections 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e75470640 1 -- 172.21.15.1:0/2666560127 >> 172.21.15.1:0/2666560127 conn(0x7f3e700d2360 msgr2=0x7f3e700ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e75470640 1 -- 172.21.15.1:0/2666560127 shutdown_connections 2024-05-09T05:23:51.650 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.650+0000 7f3e75470640 1 -- 172.21.15.1:0/2666560127 wait complete. 2024-05-09T05:23:51.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.652+0000 7f3e75470640 1 Processor -- start 2024-05-09T05:23:51.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.652+0000 7f3e75470640 1 -- start start 2024-05-09T05:23:51.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e75470640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e58005210 0x7f3e7014f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e75470640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e701560e0 0x7f3e70151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e75470640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e701e99b0 0x7f3e7014c540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e75470640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f3e700cc750 con 0x7f3e701e99b0 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e75470640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f3e7015cd50 con 0x7f3e58005210 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e75470640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f3e70156ae0 con 0x7f3e701560e0 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e701e99b0 0x7f3e7014c540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e701e99b0 0x7f3e7014c540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37314/0 (socket says 172.21.15.1:37314) 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/847812589 learned_addr learned my addr 172.21.15.1:0/847812589 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6effd640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e58005210 0x7f3e7014f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6e7fc640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e701560e0 0x7f3e70151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e701560e0 msgr2=0x7f3e70151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f3e701560e0 0x7f3e70151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e58005210 msgr2=0x7f3e7014f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f3e58005210 0x7f3e7014f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/847812589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e7014d960 con 0x7f3e701e99b0 2024-05-09T05:23:51.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e6f7fe640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e701e99b0 0x7f3e7014c540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3e6800a880 tx=0x7f3e700cd010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.654+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6800a410 con 0x7f3e701e99b0 2024-05-09T05:23:51.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.655+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3e68001ac0 con 0x7f3e701e99b0 2024-05-09T05:23:51.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.655+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e7014f6e0 con 0x7f3e701e99b0 2024-05-09T05:23:51.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.655+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e6802cc90 con 0x7f3e701e99b0 2024-05-09T05:23:51.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.655+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e7014e2a0 con 0x7f3e701e99b0 2024-05-09T05:23:51.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.655+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f3e7014df40 con 0x7f3e701e99b0 2024-05-09T05:23:51.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.656+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f3e68034470 con 0x7f3e701e99b0 2024-05-09T05:23:51.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.657+0000 7f3e577fe640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.657+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f3e680b7fd0 con 0x7f3e701e99b0 2024-05-09T05:23:51.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.657+0000 7f3e577fe640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7f3e5c07e640 0x7f3e5c080b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.657+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 --> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f3e68035e90 con 0x7f3e5c07e640 2024-05-09T05:23:51.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.657+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f3e680b8460 con 0x7f3e701e99b0 2024-05-09T05:23:51.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.657+0000 7f3e6effd640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.655 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.657+0000 7f3e6e7fc640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7f3e5c07e640 0x7f3e5c080b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.658+0000 7f3e6e7fc640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7f3e5c07e640 0x7f3e5c080b10 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.658+0000 7f3e6effd640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3e60001940 tx=0x7f3e60009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.656 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.658+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 <== osd.2 v2:172.21.15.1:6800/239705464 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f3e68035e90 con 0x7f3e5c07e640 2024-05-09T05:23:51.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 --> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f3e70050870 con 0x7f3e5c07e640 2024-05-09T05:23:51.668 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e577fe640 1 -- 172.21.15.1:0/847812589 <== osd.2 v2:172.21.15.1:6800/239705464 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f3e70050870 con 0x7f3e5c07e640 2024-05-09T05:23:51.669 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:51.669 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:23:51.669 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7f3e5c07e640 msgr2=0x7f3e5c080b10 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.1:6800/239705464,v1:172.21.15.1:6801/239705464] conn(0x7f3e5c07e640 0x7f3e5c080b10 crc :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e5c0762d0 msgr2=0x7f3e5c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f3e5c0762d0 0x7f3e5c0787e0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f3e60001940 tx=0x7f3e60009040 comp rx=0 tx=0).stop 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e701e99b0 msgr2=0x7f3e7014c540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 --2- 172.21.15.1:0/847812589 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f3e701e99b0 0x7f3e7014c540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f3e6800a880 tx=0x7f3e700cd010 comp rx=0 tx=0).stop 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e6f7fe640 1 -- 172.21.15.1:0/847812589 reap_dead start 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 shutdown_connections 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 >> 172.21.15.1:0/847812589 conn(0x7f3e700d2360 msgr2=0x7f3e700cee50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 shutdown_connections 2024-05-09T05:23:51.670 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.672+0000 7f3e75470640 1 -- 172.21.15.1:0/847812589 wait complete. 2024-05-09T05:23:51.681 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2024-05-09T05:23:51.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.799+0000 7fe17d986640 1 Processor -- start 2024-05-09T05:23:51.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.799+0000 7fe17d986640 1 -- start start 2024-05-09T05:23:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.800+0000 7fe17d986640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe178156bf0 0x7fe17806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.800+0000 7fe17d986640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1780cc9d0 0x7fe17806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.800+0000 7fe17d986640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fe1780b5290 con 0x7fe1780cc5c0 2024-05-09T05:23:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.800+0000 7fe17d986640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fe17815ff50 con 0x7fe178156bf0 2024-05-09T05:23:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.800+0000 7fe17d986640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe1780c8020 con 0x7fe1780cc9d0 2024-05-09T05:23:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe1777fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1780cc9d0 0x7fe17806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe176ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe178156bf0 0x7fe17806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe1777fe640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1780cc9d0 0x7fe17806ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59298/0 (socket says 172.21.15.1:59298) 2024-05-09T05:23:51.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe176ffd640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fe178156bf0 0x7fe17806d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3300/0 says I am v2:172.21.15.1:58972/0 (socket says 172.21.15.1:58972) 2024-05-09T05:23:51.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe1777fe640 1 -- 172.21.15.1:0/2224308512 learned_addr learned my addr 172.21.15.1:0/2224308512 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:51.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe1777fe640 1 -- 172.21.15.1:0/2224308512 >> v2:172.21.15.196:3300/0 conn(0x7fe178156bf0 msgr2=0x7fe17806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:51.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe1777fe640 1 --2- 172.21.15.1:0/2224308512 >> v2:172.21.15.196:3300/0 conn(0x7fe178156bf0 0x7fe17806d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe1777fe640 1 -- 172.21.15.1:0/2224308512 >> v1:172.21.15.196:6789/0 conn(0x7fe1780cc5c0 legacy=0x7fe17806e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe1777fe640 1 -- 172.21.15.1:0/2224308512 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1780c82e0 con 0x7fe1780cc9d0 2024-05-09T05:23:51.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.801+0000 7fe176ffd640 1 --2- 172.21.15.1:0/2224308512 >> v2:172.21.15.196:3300/0 conn(0x7fe178156bf0 0x7fe17806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:23:51.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.802+0000 7fe1777fe640 1 --2- 172.21.15.1:0/2224308512 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1780cc9d0 0x7fe17806ef50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe170009e50 tx=0x7fe17002a710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df483fee039cfbf7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.802+0000 7fe1767fc640 1 -- 172.21.15.1:0/2224308512 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe170036070 con 0x7fe1780cc9d0 2024-05-09T05:23:51.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.802+0000 7fe1767fc640 1 -- 172.21.15.1:0/2224308512 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe170031600 con 0x7fe1780cc9d0 2024-05-09T05:23:51.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.802+0000 7fe1767fc640 1 -- 172.21.15.1:0/2224308512 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1700319d0 con 0x7fe1780cc9d0 2024-05-09T05:23:51.800 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.803+0000 7fe17d986640 1 -- 172.21.15.1:0/2224308512 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1780cc9d0 msgr2=0x7fe17806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.803+0000 7fe17d986640 1 --2- 172.21.15.1:0/2224308512 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1780cc9d0 0x7fe17806ef50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe170009e50 tx=0x7fe17002a710 comp rx=0 tx=0).stop 2024-05-09T05:23:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.803+0000 7fe17d986640 1 -- 172.21.15.1:0/2224308512 shutdown_connections 2024-05-09T05:23:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.803+0000 7fe17d986640 1 --2- 172.21.15.1:0/2224308512 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe1780cc9d0 0x7fe17806ef50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.803+0000 7fe17d986640 1 --2- 172.21.15.1:0/2224308512 >> v2:172.21.15.196:3300/0 conn(0x7fe178156bf0 0x7fe17806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.803+0000 7fe17d986640 1 -- 172.21.15.1:0/2224308512 >> 172.21.15.1:0/2224308512 conn(0x7fe178068db0 msgr2=0x7fe1780d1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.803+0000 7fe17d986640 1 -- 172.21.15.1:0/2224308512 shutdown_connections 2024-05-09T05:23:51.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.803+0000 7fe17d986640 1 -- 172.21.15.1:0/2224308512 wait complete. 2024-05-09T05:23:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.804+0000 7fe17d986640 1 Processor -- start 2024-05-09T05:23:51.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.805+0000 7fe17d986640 1 -- start start 2024-05-09T05:23:51.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe17d986640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1780cc5c0 0x7fe178157e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe17d986640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1780cc9d0 0x7fe178151f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe17d986640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe178156bf0 0x7fe17814f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe17d986640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fe1781604d0 con 0x7fe1780cc5c0 2024-05-09T05:23:51.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe17d986640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fe1781601f0 con 0x7fe1780cc9d0 2024-05-09T05:23:51.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe17d986640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fe1781607b0 con 0x7fe178156bf0 2024-05-09T05:23:51.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe176ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1780cc5c0 0x7fe178157e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe176ffd640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1780cc5c0 0x7fe178157e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37328/0 (socket says 172.21.15.1:37328) 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe176ffd640 1 -- 172.21.15.1:0/3921460525 learned_addr learned my addr 172.21.15.1:0/3921460525 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe16e7fc640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1780cc9d0 0x7fe178151f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe1777fe640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe178156bf0 0x7fe17814f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe176ffd640 1 -- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe178156bf0 msgr2=0x7fe17814f590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe176ffd640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe178156bf0 0x7fe17814f590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe176ffd640 1 -- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1780cc9d0 msgr2=0x7fe178151f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe176ffd640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1780cc9d0 0x7fe178151f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.806+0000 7fe176ffd640 1 -- 172.21.15.1:0/3921460525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1781583c0 con 0x7fe1780cc5c0 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.807+0000 7fe176ffd640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1780cc5c0 0x7fe178157e80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe16800e990 tx=0x7fe16800ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.807+0000 7fe174ff9640 1 -- 172.21.15.1:0/3921460525 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe168017070 con 0x7fe1780cc5c0 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.807+0000 7fe174ff9640 1 -- 172.21.15.1:0/3921460525 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fe16800c9e0 con 0x7fe1780cc5c0 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.807+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe178151b40 con 0x7fe1780cc5c0 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.807+0000 7fe174ff9640 1 -- 172.21.15.1:0/3921460525 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe16800cdb0 con 0x7fe1780cc5c0 2024-05-09T05:23:51.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.807+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1781573a0 con 0x7fe1780cc5c0 2024-05-09T05:23:51.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.808+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1780b7f20 con 0x7fe1780cc5c0 2024-05-09T05:23:51.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.808+0000 7fe174ff9640 1 -- 172.21.15.1:0/3921460525 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fe16801a070 con 0x7fe1780cc5c0 2024-05-09T05:23:51.806 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.809+0000 7fe174ff9640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1580762d0 0x7fe1580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.809+0000 7fe174ff9640 1 -- 172.21.15.1:0/3921460525 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fe168097400 con 0x7fe1780cc5c0 2024-05-09T05:23:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.809+0000 7fe16e7fc640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1580762d0 0x7fe1580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:51.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.810+0000 7fe16e7fc640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1580762d0 0x7fe1580787e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe1780cddb0 tx=0x7fe160009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:51.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:51.814+0000 7fe174ff9640 1 -- 172.21.15.1:0/3921460525 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fe1780b7f20 con 0x7fe1780cc5c0 2024-05-09T05:23:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.009+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) -- 0x7fe1780cc020 con 0x7fe1580762d0 2024-05-09T05:23:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.009+0000 7fe174ff9640 1 -- 172.21.15.1:0/3921460525 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fe1780cc020 con 0x7fe1580762d0 2024-05-09T05:23:52.006 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 2 to deep-scrub 2024-05-09T05:23:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1580762d0 msgr2=0x7fe1580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1580762d0 0x7fe1580787e0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe1780cddb0 tx=0x7fe160009040 comp rx=0 tx=0).stop 2024-05-09T05:23:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1780cc5c0 msgr2=0x7fe178157e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1780cc5c0 0x7fe178157e80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe16800e990 tx=0x7fe16800ee50 comp rx=0 tx=0).stop 2024-05-09T05:23:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 shutdown_connections 2024-05-09T05:23:52.009 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fe1580762d0 0x7fe1580787e0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fe1780cc9d0 0x7fe178151f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fe178156bf0 0x7fe17814f590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 --2- 172.21.15.1:0/3921460525 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fe1780cc5c0 0x7fe178157e80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 >> 172.21.15.1:0/3921460525 conn(0x7fe178068db0 msgr2=0x7fe178155730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 shutdown_connections 2024-05-09T05:23:52.010 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.012+0000 7fe17d986640 1 -- 172.21.15.1:0/3921460525 wait complete. 2024-05-09T05:23:52.021 INFO:tasks.ceph:Scrubbing osd.3 2024-05-09T05:23:52.021 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:23:52.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.145+0000 7f1a5516c640 1 Processor -- start 2024-05-09T05:23:52.143 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.146+0000 7f1a5516c640 1 -- start start 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a5516c640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f1a50153730 0x7f1a50164d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a5516c640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a500c8a50 con 0x7f1a500cae10 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a5516c640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f1a500c8d10 con 0x7f1a500ca970 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a5516c640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f1a500c9d00 con 0x7f1a50153730 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a4e59c640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f1a500cae10 0x7f1a500c67b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55274/0 (socket says 172.21.15.1:55274) 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a4e59c640 1 -- 172.21.15.1:0/1885027673 learned_addr learned my addr 172.21.15.1:0/1885027673 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a4f59e640 1 --2- 172.21.15.1:0/1885027673 >> v2:172.21.15.196:3300/0 conn(0x7f1a50153730 0x7f1a50164d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1915826559 0 0) 0x7f1a500c8a50 con 0x7f1a500cae10 2024-05-09T05:23:52.147 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.147+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f1a3c003520 con 0x7f1a500cae10 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 838021753 0 0) 0x7f1a3c003520 con 0x7f1a500cae10 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 >> v2:172.21.15.196:3300/0 conn(0x7f1a50153730 msgr2=0x7f1a50164d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 --2- 172.21.15.1:0/1885027673 >> v2:172.21.15.196:3300/0 conn(0x7f1a50153730 0x7f1a50164d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 >> v1:172.21.15.196:6789/0 conn(0x7f1a500ca970 legacy=0x7f1a5015c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a500c5e10 con 0x7f1a500cae10 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f1a380016f0 con 0x7f1a500cae10 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 >> v1:172.21.15.1:6789/0 conn(0x7f1a500cae10 legacy=0x7f1a500c67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1a3c003a50 0x7f1a3c003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c004800 0x7f1a3c004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a3c005250 0x7f1a3c00b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1a500c9d00 con 0x7f1a3c005250 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1a3c00c1e0 con 0x7f1a3c004800 2024-05-09T05:23:52.148 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1a3c00c3e0 con 0x7f1a3c003a50 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4f59e640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a3c005250 0x7f1a3c00b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4e59c640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c004800 0x7f1a3c004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4ed9d640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1a3c003a50 0x7f1a3c003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1885027673 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1a3c003a50 msgr2=0x7f1a3c003ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4f59e640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1a3c003a50 0x7f1a3c003ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1885027673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c004800 msgr2=0x7f1a3c004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4f59e640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a3c004800 0x7f1a3c004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.148+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1885027673 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a3c003520 con 0x7f1a3c005250 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.149+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1885027673 reap_dead start 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.149+0000 7f1a4f59e640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a3c005250 0x7f1a3c00b9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1a50072e10 tx=0x7f1a3c005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6e9afe1d48d846ba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.149+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40009c10 con 0x7f1a3c005250 2024-05-09T05:23:52.149 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.149+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a40031410 con 0x7f1a3c005250 2024-05-09T05:23:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.149+0000 7f1a4dd9b640 1 -- 172.21.15.1:0/1885027673 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a400317c0 con 0x7f1a3c005250 2024-05-09T05:23:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.150+0000 7f1a5516c640 1 -- 172.21.15.1:0/1885027673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a3c005250 msgr2=0x7f1a3c00b9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.150+0000 7f1a5516c640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a3c005250 0x7f1a3c00b9c0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f1a50072e10 tx=0x7f1a3c005830 comp rx=0 tx=0).stop 2024-05-09T05:23:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.150+0000 7f1a5516c640 1 -- 172.21.15.1:0/1885027673 shutdown_connections 2024-05-09T05:23:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.150+0000 7f1a5516c640 1 --2- 172.21.15.1:0/1885027673 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a3c005250 0x7f1a3c00b9c0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.150+0000 7f1a5516c640 1 -- 172.21.15.1:0/1885027673 >> 172.21.15.1:0/1885027673 conn(0x7f1a500ce910 msgr2=0x7f1a500ced20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.150+0000 7f1a5516c640 1 -- 172.21.15.1:0/1885027673 shutdown_connections 2024-05-09T05:23:52.150 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.150+0000 7f1a5516c640 1 -- 172.21.15.1:0/1885027673 wait complete. 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.152+0000 7f1a5516c640 1 Processor -- start 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.152+0000 7f1a5516c640 1 -- start start 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a5516c640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1a3c005250 0x7f1a501f31a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a5516c640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a501e3b10 0x7f1a5013c140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a5516c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a5013c680 0x7f1a501576c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a5516c640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f1a500c63d0 con 0x7f1a5013c680 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a5516c640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f1a500c9870 con 0x7f1a501e3b10 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a5516c640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f1a500c9500 con 0x7f1a3c005250 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a4f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a5013c680 0x7f1a501576c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a4f59e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a5013c680 0x7f1a501576c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37344/0 (socket says 172.21.15.1:37344) 2024-05-09T05:23:52.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1487149813 learned_addr learned my addr 172.21.15.1:0/1487149813 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a4ed9d640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1a3c005250 0x7f1a501f31a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.153+0000 7f1a4e59c640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a501e3b10 0x7f1a5013c140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1a3c005250 msgr2=0x7f1a501f31a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a4f59e640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f1a3c005250 0x7f1a501f31a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a501e3b10 msgr2=0x7f1a5013c140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a4f59e640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f1a501e3b10 0x7f1a5013c140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1487149813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a501f36e0 con 0x7f1a5013c680 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a4f59e640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a5013c680 0x7f1a501576c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a40033040 tx=0x7f1a40009750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a40009c10 con 0x7f1a5013c680 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f1a40043070 con 0x7f1a5013c680 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a4003c850 con 0x7f1a5013c680 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a501f6de0 con 0x7f1a5013c680 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.154+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a501414c0 con 0x7f1a5013c680 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.155+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f1a50050870 con 0x7f1a5013c680 2024-05-09T05:23:52.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.156+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f1a40005d10 con 0x7f1a5013c680 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.156+0000 7f1a477fe640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1a300762d0 0x7f1a300787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.156+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f1a400b8b00 con 0x7f1a5013c680 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.156+0000 7f1a477fe640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f1a3007e6a0 0x7f1a30080b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.156+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 --> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f1a40007e80 con 0x7f1a3007e6a0 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.156+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f1a400b8f90 con 0x7f1a5013c680 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.156+0000 7f1a4ed9d640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1a300762d0 0x7f1a300787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.157+0000 7f1a4e59c640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f1a3007e6a0 0x7f1a30080b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.157+0000 7f1a4e59c640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f1a3007e6a0 0x7f1a30080b70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.157+0000 7f1a4ed9d640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1a300762d0 0x7f1a300787e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1a480055d0 tx=0x7f1a48009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.157 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.158+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 <== osd.3 v2:172.21.15.1:6808/3991537322 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f1a40007e80 con 0x7f1a3007e6a0 2024-05-09T05:23:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.171+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 --> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f1a501488f0 con 0x7f1a3007e6a0 2024-05-09T05:23:52.168 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.171+0000 7f1a477fe640 1 -- 172.21.15.1:0/1487149813 <== osd.3 v2:172.21.15.1:6808/3991537322 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f1a501488f0 con 0x7f1a3007e6a0 2024-05-09T05:23:52.168 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:52.168 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:23:52.168 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.171+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f1a3007e6a0 msgr2=0x7f1a30080b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.171+0000 7f1a5516c640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.1:6808/3991537322,v1:172.21.15.1:6809/3991537322] conn(0x7f1a3007e6a0 0x7f1a30080b70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1a300762d0 msgr2=0x7f1a300787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a5516c640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f1a300762d0 0x7f1a300787e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f1a480055d0 tx=0x7f1a48009040 comp rx=0 tx=0).stop 2024-05-09T05:23:52.169 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a5013c680 msgr2=0x7f1a501576c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a5516c640 1 --2- 172.21.15.1:0/1487149813 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f1a5013c680 0x7f1a501576c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f1a40033040 tx=0x7f1a40009750 comp rx=0 tx=0).stop 2024-05-09T05:23:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a4f59e640 1 -- 172.21.15.1:0/1487149813 reap_dead start 2024-05-09T05:23:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 shutdown_connections 2024-05-09T05:23:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 >> 172.21.15.1:0/1487149813 conn(0x7f1a500ce910 msgr2=0x7f1a50145000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 shutdown_connections 2024-05-09T05:23:52.170 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.172+0000 7f1a5516c640 1 -- 172.21.15.1:0/1487149813 wait complete. 2024-05-09T05:23:52.180 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2024-05-09T05:23:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.299+0000 7fbe07061640 1 Processor -- start 2024-05-09T05:23:52.296 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.299+0000 7fbe07061640 1 -- start start 2024-05-09T05:23:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbe07061640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe00070df0 0x7fbe00154bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.297 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbe07061640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe000704b0 con 0x7fbe000c9ff0 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbe07061640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fbe00070790 con 0x7fbe000c6100 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbe07061640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fbe000c7a30 con 0x7fbe00070df0 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbdfffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe00070df0 0x7fbe00154bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbe055fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fbe000c9ff0 0x7fbe000c9520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55286/0 (socket says 172.21.15.1:55286) 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbe04dfd640 1 --1- >> v1:172.21.15.196:6789/0 conn(0x7fbe000c6100 0x7fbe0006c7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.196:6789/0 says I am v1:172.21.15.1:45608/0 (socket says 172.21.15.1:45608) 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbdfffff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fbe00070df0 0x7fbe00154bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37350/0 (socket says 172.21.15.1:37350) 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.300+0000 7fbe055fe640 1 -- 172.21.15.1:0/1087257242 learned_addr learned my addr 172.21.15.1:0/1087257242 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.301+0000 7fbdfffff640 1 -- 172.21.15.1:0/1087257242 >> v1:172.21.15.196:6789/0 conn(0x7fbe000c6100 legacy=0x7fbe0006c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.301+0000 7fbdfffff640 1 -- 172.21.15.1:0/1087257242 >> v1:172.21.15.1:6789/0 conn(0x7fbe000c9ff0 legacy=0x7fbe000c9520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.301+0000 7fbdfffff640 1 -- 172.21.15.1:0/1087257242 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe000c7d10 con 0x7fbe00070df0 2024-05-09T05:23:52.299 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.301+0000 7fbdfffff640 1 --2- 172.21.15.1:0/1087257242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00070df0 0x7fbe00154bc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009d90 tx=0x7fbdf002a6f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=387408af31fd0e70 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.301+0000 7fbdff7fe640 1 -- 172.21.15.1:0/1087257242 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0007740 con 0x7fbe00070df0 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbdff7fe640 1 -- 172.21.15.1:0/1087257242 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdf00078e0 con 0x7fbe00070df0 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbdff7fe640 1 -- 172.21.15.1:0/1087257242 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf0007cb0 con 0x7fbe00070df0 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbe07061640 1 -- 172.21.15.1:0/1087257242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00070df0 msgr2=0x7fbe00154bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbe07061640 1 --2- 172.21.15.1:0/1087257242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00070df0 0x7fbe00154bc0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fbdf0009d90 tx=0x7fbdf002a6f0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbe07061640 1 -- 172.21.15.1:0/1087257242 shutdown_connections 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbe07061640 1 --2- 172.21.15.1:0/1087257242 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe00070df0 0x7fbe00154bc0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbe07061640 1 -- 172.21.15.1:0/1087257242 >> 172.21.15.1:0/1087257242 conn(0x7fbe0006d800 msgr2=0x7fbe0006dc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbe07061640 1 -- 172.21.15.1:0/1087257242 shutdown_connections 2024-05-09T05:23:52.300 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.302+0000 7fbe07061640 1 -- 172.21.15.1:0/1087257242 wait complete. 2024-05-09T05:23:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.304+0000 7fbe07061640 1 Processor -- start 2024-05-09T05:23:52.301 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.304+0000 7fbe07061640 1 -- start start 2024-05-09T05:23:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.305+0000 7fbe07061640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe00070df0 0x7fbe000ca6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.305+0000 7fbe07061640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbe000c6100 0x7fbe000c4970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.305+0000 7fbe07061640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe000c9ff0 0x7fbe000c4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.305+0000 7fbe07061640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0006d080 con 0x7fbe000c9ff0 2024-05-09T05:23:52.304 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.305+0000 7fbe07061640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fbe0006cf00 con 0x7fbe00070df0 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.305+0000 7fbe07061640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fbe000c9c30 con 0x7fbe000c6100 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe000c9ff0 0x7fbe000c4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe000c9ff0 0x7fbe000c4eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37358/0 (socket says 172.21.15.1:37358) 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 -- 172.21.15.1:0/2301713149 learned_addr learned my addr 172.21.15.1:0/2301713149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe04dfd640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe00070df0 0x7fbe000ca6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbdfffff640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbe000c6100 0x7fbe000c4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 -- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbe000c6100 msgr2=0x7fbe000c4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbe000c6100 0x7fbe000c4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 -- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe00070df0 msgr2=0x7fbe000ca6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe00070df0 0x7fbe000ca6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.305 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 -- 172.21.15.1:0/2301713149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe00152e70 con 0x7fbe000c9ff0 2024-05-09T05:23:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe055fe640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe000c9ff0 0x7fbe000c4eb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800c3a0 tx=0x7fbdf800c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbdfd7fa640 1 -- 172.21.15.1:0/2301713149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf8017070 con 0x7fbe000c9ff0 2024-05-09T05:23:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbdfd7fa640 1 -- 172.21.15.1:0/2301713149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fbdf8005ce0 con 0x7fbe000c9ff0 2024-05-09T05:23:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe001e33a0 con 0x7fbe000c9ff0 2024-05-09T05:23:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbdfd7fa640 1 -- 172.21.15.1:0/2301713149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdf80072f0 con 0x7fbe000c9ff0 2024-05-09T05:23:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.306+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe001e2700 con 0x7fbe000c9ff0 2024-05-09T05:23:52.306 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.307+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbe001e2210 con 0x7fbe000c9ff0 2024-05-09T05:23:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.308+0000 7fbdfd7fa640 1 -- 172.21.15.1:0/2301713149 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fbdf801a070 con 0x7fbe000c9ff0 2024-05-09T05:23:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.309+0000 7fbdfd7fa640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbde0076300 0x7fbde0078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.309+0000 7fbdfd7fa640 1 -- 172.21.15.1:0/2301713149 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fbdf8096940 con 0x7fbe000c9ff0 2024-05-09T05:23:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.309+0000 7fbe04dfd640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbde0076300 0x7fbde0078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.307 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.309+0000 7fbe04dfd640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbde0076300 0x7fbde0078810 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbde8000f60 tx=0x7fbde8000bc0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.310 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.313+0000 7fbdfd7fa640 1 -- 172.21.15.1:0/2301713149 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fbdf805fb40 con 0x7fbe000c9ff0 2024-05-09T05:23:52.499 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.502+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) -- 0x7fbe00151e10 con 0x7fbde0076300 2024-05-09T05:23:52.500 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.503+0000 7fbdfd7fa640 1 -- 172.21.15.1:0/2301713149 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7fbe00151e10 con 0x7fbde0076300 2024-05-09T05:23:52.500 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 3 to deep-scrub 2024-05-09T05:23:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbde0076300 msgr2=0x7fbde0078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbde0076300 0x7fbde0078810 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fbde8000f60 tx=0x7fbde8000bc0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe000c9ff0 msgr2=0x7fbe000c4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe000c9ff0 0x7fbe000c4eb0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fbdf800c3a0 tx=0x7fbdf800c860 comp rx=0 tx=0).stop 2024-05-09T05:23:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 shutdown_connections 2024-05-09T05:23:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fbde0076300 0x7fbde0078810 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.503 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fbe000c9ff0 0x7fbe000c4eb0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fbe000c6100 0x7fbe000c4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 --2- 172.21.15.1:0/2301713149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fbe00070df0 0x7fbe000ca6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 >> 172.21.15.1:0/2301713149 conn(0x7fbe0006d800 msgr2=0x7fbe00071580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 shutdown_connections 2024-05-09T05:23:52.504 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.506+0000 7fbe07061640 1 -- 172.21.15.1:0/2301713149 wait complete. 2024-05-09T05:23:52.515 INFO:tasks.ceph:Scrubbing osd.4 2024-05-09T05:23:52.515 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:23:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.636+0000 7fc36f625640 1 Processor -- start 2024-05-09T05:23:52.633 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.636+0000 7fc36f625640 1 -- start start 2024-05-09T05:23:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36f625640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3680c69a0 0x7fc368160a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36f625640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3680cb610 con 0x7fc36806e1d0 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36f625640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc368156ef0 con 0x7fc368156ae0 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36f625640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc3680cb050 con 0x7fc3680c69a0 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36d3c1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3680c69a0 0x7fc368160a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36d3c1640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3680c69a0 0x7fc368160a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37374/0 (socket says 172.21.15.1:37374) 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36cbc0640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7fc36806e1d0 0x7fc3680cc800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55294/0 (socket says 172.21.15.1:55294) 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36d3c1640 1 -- 172.21.15.1:0/3551865167 learned_addr learned my addr 172.21.15.1:0/3551865167 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36d3c1640 1 -- 172.21.15.1:0/3551865167 >> v1:172.21.15.196:6789/0 conn(0x7fc368156ae0 legacy=0x7fc368167190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36d3c1640 1 -- 172.21.15.1:0/3551865167 >> v1:172.21.15.1:6789/0 conn(0x7fc36806e1d0 legacy=0x7fc3680cc800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.638+0000 7fc36d3c1640 1 -- 172.21.15.1:0/3551865167 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3680cb330 con 0x7fc3680c69a0 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.639+0000 7fc36d3c1640 1 --2- 172.21.15.1:0/3551865167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3680c69a0 0x7fc368160a10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc360009870 tx=0x7fc36002a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a785d3d1e827d9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.639+0000 7fc367fff640 1 -- 172.21.15.1:0/3551865167 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360037070 con 0x7fc3680c69a0 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.639+0000 7fc367fff640 1 -- 172.21.15.1:0/3551865167 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc36002ae50 con 0x7fc3680c69a0 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.639+0000 7fc367fff640 1 -- 172.21.15.1:0/3551865167 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360032860 con 0x7fc3680c69a0 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.640+0000 7fc36f625640 1 -- 172.21.15.1:0/3551865167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3680c69a0 msgr2=0x7fc368160a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.640+0000 7fc36f625640 1 --2- 172.21.15.1:0/3551865167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3680c69a0 0x7fc368160a10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc360009870 tx=0x7fc36002a300 comp rx=0 tx=0).stop 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.640+0000 7fc36f625640 1 -- 172.21.15.1:0/3551865167 shutdown_connections 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.640+0000 7fc36f625640 1 --2- 172.21.15.1:0/3551865167 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3680c69a0 0x7fc368160a10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.640+0000 7fc36f625640 1 -- 172.21.15.1:0/3551865167 >> 172.21.15.1:0/3551865167 conn(0x7fc368068830 msgr2=0x7fc368068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.640+0000 7fc36f625640 1 -- 172.21.15.1:0/3551865167 shutdown_connections 2024-05-09T05:23:52.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.640+0000 7fc36f625640 1 -- 172.21.15.1:0/3551865167 wait complete. 2024-05-09T05:23:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.642+0000 7fc36f625640 1 Processor -- start 2024-05-09T05:23:52.639 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.642+0000 7fc36f625640 1 -- start start 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36f625640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36806e1d0 0x7fc368155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36f625640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc368156ae0 0x7fc36815b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36f625640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc368154390 0x7fc3681e8c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36f625640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3681604f0 con 0x7fc36806e1d0 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36f625640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc368160210 con 0x7fc368156ae0 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36f625640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc3680ccf10 con 0x7fc368154390 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36d3c1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36806e1d0 0x7fc368155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36d3c1640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36806e1d0 0x7fc368155ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37390/0 (socket says 172.21.15.1:37390) 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36d3c1640 1 -- 172.21.15.1:0/3516578635 learned_addr learned my addr 172.21.15.1:0/3516578635 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:52.641 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36cbc0640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc368156ae0 0x7fc36815b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36dbc2640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc368154390 0x7fc3681e8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36d3c1640 1 -- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc368154390 msgr2=0x7fc3681e8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36d3c1640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc368154390 0x7fc3681e8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36d3c1640 1 -- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc368156ae0 msgr2=0x7fc36815b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36d3c1640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc368156ae0 0x7fc36815b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36d3c1640 1 -- 172.21.15.1:0/3516578635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc36815fbc0 con 0x7fc36806e1d0 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.643+0000 7fc36cbc0640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc368156ae0 0x7fc36815b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.644+0000 7fc36d3c1640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36806e1d0 0x7fc368155ed0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc360000c00 tx=0x7fc3600046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.644+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360041020 con 0x7fc36806e1d0 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.644+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fc36003c890 con 0x7fc36806e1d0 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.644+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc36815b920 con 0x7fc36806e1d0 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.644+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc360037040 con 0x7fc36806e1d0 2024-05-09T05:23:52.642 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.644+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc36815bb30 con 0x7fc36806e1d0 2024-05-09T05:23:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.645+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3680cb8a0 con 0x7fc36806e1d0 2024-05-09T05:23:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.646+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc360044020 con 0x7fc36806e1d0 2024-05-09T05:23:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.646+0000 7fc365ffb640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3540762d0 0x7fc3540787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.646+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc3600b8560 con 0x7fc36806e1d0 2024-05-09T05:23:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.647+0000 7fc36cbc0640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3540762d0 0x7fc3540787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.647+0000 7fc365ffb640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7fc35407e6a0 0x7fc354080b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.647+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 --> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc360008f00 con 0x7fc35407e6a0 2024-05-09T05:23:52.644 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.647+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc3600b89f0 con 0x7fc36806e1d0 2024-05-09T05:23:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.647+0000 7fc36dbc2640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7fc35407e6a0 0x7fc354080b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.647+0000 7fc36cbc0640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3540762d0 0x7fc3540787e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc368152a50 tx=0x7fc35c007d20 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.647+0000 7fc36dbc2640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7fc35407e6a0 0x7fc354080b70 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.645 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.648+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 <== osd.4 v2:172.21.15.196:6808/4274960119 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc360008f00 con 0x7fc35407e6a0 2024-05-09T05:23:52.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.661+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 --> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fc3680ca0b0 con 0x7fc35407e6a0 2024-05-09T05:23:52.658 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc365ffb640 1 -- 172.21.15.1:0/3516578635 <== osd.4 v2:172.21.15.196:6808/4274960119 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fc3680ca0b0 con 0x7fc35407e6a0 2024-05-09T05:23:52.659 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:52.659 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:23:52.659 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7fc35407e6a0 msgr2=0x7fc354080b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6808/4274960119,v1:172.21.15.196:6809/4274960119] conn(0x7fc35407e6a0 0x7fc354080b70 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3540762d0 msgr2=0x7fc3540787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3540762d0 0x7fc3540787e0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7fc368152a50 tx=0x7fc35c007d20 comp rx=0 tx=0).stop 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36806e1d0 msgr2=0x7fc368155ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 --2- 172.21.15.1:0/3516578635 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc36806e1d0 0x7fc368155ed0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc360000c00 tx=0x7fc3600046c0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36dbc2640 1 -- 172.21.15.1:0/3516578635 reap_dead start 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 shutdown_connections 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 >> 172.21.15.1:0/3516578635 conn(0x7fc368068830 msgr2=0x7fc368075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 shutdown_connections 2024-05-09T05:23:52.660 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.662+0000 7fc36f625640 1 -- 172.21.15.1:0/3516578635 wait complete. 2024-05-09T05:23:52.671 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2024-05-09T05:23:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.790+0000 7f28dd1a8640 1 Processor -- start 2024-05-09T05:23:52.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.790+0000 7f28dd1a8640 1 -- start start 2024-05-09T05:23:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.791+0000 7f28dd1a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d80cbde0 0x7f28d81549c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.791+0000 7f28dd1a8640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f28d80c49e0 0x7f28d80c5190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28dd1a8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28d80cb060 0x7f28d80cb470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28dd1a8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f28d80c6970 con 0x7f28d80cb060 2024-05-09T05:23:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28dd1a8640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f28d80b2860 con 0x7f28d80c49e0 2024-05-09T05:23:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28dd1a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f28d80c57b0 con 0x7f28d80cbde0 2024-05-09T05:23:52.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d759e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28d80cb060 0x7f28d80cb470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d759e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f28d80cb060 0x7f28d80cb470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37392/0 (socket says 172.21.15.1:37392) 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d759e640 1 -- 172.21.15.1:0/1159381539 learned_addr learned my addr 172.21.15.1:0/1159381539 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d6d9d640 1 --2- 172.21.15.1:0/1159381539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d80cbde0 0x7f28d81549c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d659c640 1 --2- 172.21.15.1:0/1159381539 >> v2:172.21.15.196:3300/0 conn(0x7f28d80c49e0 0x7f28d80c5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d759e640 1 -- 172.21.15.1:0/1159381539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d80cbde0 msgr2=0x7f28d81549c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d759e640 1 --2- 172.21.15.1:0/1159381539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d80cbde0 0x7f28d81549c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d759e640 1 -- 172.21.15.1:0/1159381539 >> v2:172.21.15.196:3300/0 conn(0x7f28d80c49e0 msgr2=0x7f28d80c5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d759e640 1 --2- 172.21.15.1:0/1159381539 >> v2:172.21.15.196:3300/0 conn(0x7f28d80c49e0 0x7f28d80c5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.792+0000 7f28d759e640 1 -- 172.21.15.1:0/1159381539 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d80c5d60 con 0x7f28d80cb060 2024-05-09T05:23:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.793+0000 7f28d759e640 1 --2- 172.21.15.1:0/1159381539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80cb470 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f28d80d0ab0 tx=0x7f28c802a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=955c2e09d836d1a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.791 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.793+0000 7f28d5d9b640 1 -- 172.21.15.1:0/1159381539 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c8036070 con 0x7f28d80cb060 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.793+0000 7f28d5d9b640 1 -- 172.21.15.1:0/1159381539 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c8031600 con 0x7f28d80cb060 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.793+0000 7f28d5d9b640 1 -- 172.21.15.1:0/1159381539 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c80319d0 con 0x7f28d80cb060 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 -- 172.21.15.1:0/1159381539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 msgr2=0x7f28d80cb470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/1159381539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80cb470 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f28d80d0ab0 tx=0x7f28c802a6a0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 -- 172.21.15.1:0/1159381539 shutdown_connections 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/1159381539 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80cb470 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/1159381539 >> v2:172.21.15.196:3300/0 conn(0x7f28d80c49e0 0x7f28d80c5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/1159381539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d80cbde0 0x7f28d81549c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 -- 172.21.15.1:0/1159381539 >> 172.21.15.1:0/1159381539 conn(0x7f28d8068830 msgr2=0x7f28d8068c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 -- 172.21.15.1:0/1159381539 shutdown_connections 2024-05-09T05:23:52.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.794+0000 7f28dd1a8640 1 -- 172.21.15.1:0/1159381539 wait complete. 2024-05-09T05:23:52.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.796+0000 7f28dd1a8640 1 Processor -- start 2024-05-09T05:23:52.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.796+0000 7f28dd1a8640 1 -- start start 2024-05-09T05:23:52.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28dd1a8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d80c49e0 0x7f28d8152d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28dd1a8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80c9380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28dd1a8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d81f3af0 0x7f28d8158480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28dd1a8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f28d80c6600 con 0x7f28d80cb060 2024-05-09T05:23:52.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28dd1a8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f28d80c6320 con 0x7f28d80c49e0 2024-05-09T05:23:52.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28dd1a8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f28d815dc00 con 0x7f28d81f3af0 2024-05-09T05:23:52.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80c9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d659c640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80c9380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37394/0 (socket says 172.21.15.1:37394) 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d659c640 1 -- 172.21.15.1:0/3269802814 learned_addr learned my addr 172.21.15.1:0/3269802814 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d6d9d640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d80c49e0 0x7f28d8152d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d759e640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d81f3af0 0x7f28d8158480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d659c640 1 -- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d81f3af0 msgr2=0x7f28d8158480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d659c640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d81f3af0 0x7f28d8158480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d659c640 1 -- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d80c49e0 msgr2=0x7f28d8152d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d659c640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d80c49e0 0x7f28d8152d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.797+0000 7f28d659c640 1 -- 172.21.15.1:0/3269802814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28d81532d0 con 0x7f28d80cb060 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.798+0000 7f28d659c640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80c9380 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f28c000e870 tx=0x7f28c000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.796 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.798+0000 7f28cf7fe640 1 -- 172.21.15.1:0/3269802814 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c0017070 con 0x7f28d80cb060 2024-05-09T05:23:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.798+0000 7f28cf7fe640 1 -- 172.21.15.1:0/3269802814 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f28c000ee60 con 0x7f28d80cb060 2024-05-09T05:23:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.798+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28d81f0770 con 0x7f28d80cb060 2024-05-09T05:23:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.798+0000 7f28cf7fe640 1 -- 172.21.15.1:0/3269802814 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c000cb20 con 0x7f28d80cb060 2024-05-09T05:23:52.797 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.798+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28d815d9b0 con 0x7f28d80cb060 2024-05-09T05:23:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.799+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f28d80768e0 con 0x7f28d80cb060 2024-05-09T05:23:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.800+0000 7f28cf7fe640 1 -- 172.21.15.1:0/3269802814 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f28c001a070 con 0x7f28d80cb060 2024-05-09T05:23:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.800+0000 7f28cf7fe640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28b80762d0 0x7f28b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.800+0000 7f28cf7fe640 1 -- 172.21.15.1:0/3269802814 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f28c00974c0 con 0x7f28d80cb060 2024-05-09T05:23:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.800+0000 7f28d6d9d640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28b80762d0 0x7f28b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:52.798 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.801+0000 7f28d6d9d640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28b80762d0 0x7f28b80787e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009aa0 tx=0x7f28d0009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:52.802 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.805+0000 7f28cf7fe640 1 -- 172.21.15.1:0/3269802814 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f28d80768e0 con 0x7f28d80cb060 2024-05-09T05:23:52.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.997+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) -- 0x7f28d8050870 con 0x7f28b80762d0 2024-05-09T05:23:52.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.997+0000 7f28cf7fe640 1 -- 172.21.15.1:0/3269802814 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f28d8050870 con 0x7f28b80762d0 2024-05-09T05:23:52.995 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 4 to deep-scrub 2024-05-09T05:23:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28b80762d0 msgr2=0x7f28b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28b80762d0 0x7f28b80787e0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f28d0009aa0 tx=0x7f28d0009190 comp rx=0 tx=0).stop 2024-05-09T05:23:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 msgr2=0x7f28d80c9380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80c9380 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f28c000e870 tx=0x7f28c000ed30 comp rx=0 tx=0).stop 2024-05-09T05:23:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 shutdown_connections 2024-05-09T05:23:52.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f28b80762d0 0x7f28b80787e0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f28d81f3af0 0x7f28d8158480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f28d80cb060 0x7f28d80c9380 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:52.999+0000 7f28dd1a8640 1 --2- 172.21.15.1:0/3269802814 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f28d80c49e0 0x7f28d8152d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:52.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.000+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 >> 172.21.15.1:0/3269802814 conn(0x7f28d8068830 msgr2=0x7f28d8072a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:52.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.000+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 shutdown_connections 2024-05-09T05:23:52.998 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.000+0000 7f28dd1a8640 1 -- 172.21.15.1:0/3269802814 wait complete. 2024-05-09T05:23:53.008 INFO:tasks.ceph:Scrubbing osd.5 2024-05-09T05:23:53.009 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:23:53.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.131+0000 7f44d3f86640 1 Processor -- start 2024-05-09T05:23:53.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.131+0000 7f44d3f86640 1 -- start start 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d3f86640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f44cc0c7990 0x7f44cc0c7da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d3f86640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44cc15c7b0 con 0x7f44cc0ce010 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d3f86640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f44cc0cca50 con 0x7f44cc0b4ec0 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d3f86640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f44cc15d0c0 con 0x7f44cc0c7990 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d1521640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f44cc0ce010 0x7f44cc0c7280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55298/0 (socket says 172.21.15.1:55298) 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d1521640 1 -- 172.21.15.1:0/3188142239 learned_addr learned my addr 172.21.15.1:0/3188142239 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d2523640 1 --2- 172.21.15.1:0/3188142239 >> v2:172.21.15.196:3300/0 conn(0x7f44cc0c7990 0x7f44cc0c7da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 2755852665 0 0) 0x7f44cc15c7b0 con 0x7f44cc0ce010 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f44b4003520 con 0x7f44cc0ce010 2024-05-09T05:23:53.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 1467339436 0 0) 0x7f44b4003520 con 0x7f44cc0ce010 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 >> v2:172.21.15.196:3300/0 conn(0x7f44cc0c7990 msgr2=0x7f44cc0c7da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 --2- 172.21.15.1:0/3188142239 >> v2:172.21.15.196:3300/0 conn(0x7f44cc0c7990 0x7f44cc0c7da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 >> v1:172.21.15.196:6789/0 conn(0x7f44cc0b4ec0 legacy=0x7f44cc0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 --> v1:172.21.15.1:6789/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44cc0ccd30 con 0x7f44cc0ce010 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 <== mon.0 v1:172.21.15.1:6789/0 3 ==== mon_map magic: 0 ==== 413+0+0 (unknown 4043683594 0 0) 0x7f44c00016f0 con 0x7f44cc0ce010 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 >> v1:172.21.15.1:6789/0 conn(0x7f44cc0ce010 legacy=0x7f44cc0c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b4003a50 0x7f44b4003ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.133+0000 7f44d0d20640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4004800 0x7f44b4004c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d0d20640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44b4005250 0x7f44b400b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc15d0c0 con 0x7f44b4003a50 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f44b400c1d0 con 0x7f44b4005250 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f44b400c3d0 con 0x7f44b4004800 2024-05-09T05:23:53.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d1d22640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b4003a50 0x7f44b4003ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d1521640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4004800 0x7f44b4004c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d2523640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44b4005250 0x7f44b400b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d1d22640 1 -- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4004800 msgr2=0x7f44b4004c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d1d22640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4004800 0x7f44b4004c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d1d22640 1 -- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44b4005250 msgr2=0x7f44b400b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d1d22640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44b4005250 0x7f44b400b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d2523640 1 -- 172.21.15.1:0/3188142239 reap_dead start 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d1d22640 1 -- 172.21.15.1:0/3188142239 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44b4003520 con 0x7f44b4003a50 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.134+0000 7f44d1521640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4004800 0x7f44b4004c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.135+0000 7f44d1d22640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b4003a50 0x7f44b4003ec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f44bc001e90 tx=0x7f44b4005830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ff02a884130d96e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.135+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc001ab0 con 0x7f44b4003a50 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.135+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44bc02ae10 con 0x7f44b4003a50 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.135+0000 7f44d0d20640 1 -- 172.21.15.1:0/3188142239 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44bc003b70 con 0x7f44b4003a50 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.135+0000 7f44d3f86640 1 -- 172.21.15.1:0/3188142239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b4003a50 msgr2=0x7f44b4003ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.135 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.135+0000 7f44d3f86640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b4003a50 0x7f44b4003ec0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f44bc001e90 tx=0x7f44b4005830 comp rx=0 tx=0).stop 2024-05-09T05:23:53.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.135+0000 7f44d3f86640 1 -- 172.21.15.1:0/3188142239 shutdown_connections 2024-05-09T05:23:53.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.136+0000 7f44d3f86640 1 --2- 172.21.15.1:0/3188142239 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44b4003a50 0x7f44b4003ec0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.136+0000 7f44d3f86640 1 -- 172.21.15.1:0/3188142239 >> 172.21.15.1:0/3188142239 conn(0x7f44cc0d2360 msgr2=0x7f44cc0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:53.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.136+0000 7f44d3f86640 1 -- 172.21.15.1:0/3188142239 shutdown_connections 2024-05-09T05:23:53.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.136+0000 7f44d3f86640 1 -- 172.21.15.1:0/3188142239 wait complete. 2024-05-09T05:23:53.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.137+0000 7f44d3f86640 1 Processor -- start 2024-05-09T05:23:53.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.137+0000 7f44d3f86640 1 -- start start 2024-05-09T05:23:53.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d3f86640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4003a50 0x7f44cc15bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d3f86640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44cc14cd90 0x7f44cc06e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d3f86640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44cc15b820 0x7f44cc14a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d3f86640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc0cc750 con 0x7f44cc15b820 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d3f86640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f44cc15cad0 con 0x7f44cc14cd90 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d3f86640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f44cc156ae0 con 0x7f44b4003a50 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44cc15b820 0x7f44cc14a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44cc15b820 0x7f44cc14a460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37400/0 (socket says 172.21.15.1:37400) 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 -- 172.21.15.1:0/3084588960 learned_addr learned my addr 172.21.15.1:0/3084588960 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d1d22640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4003a50 0x7f44cc15bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d1521640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44cc14cd90 0x7f44cc06e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 -- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4003a50 msgr2=0x7f44cc15bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f44b4003a50 0x7f44cc15bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 -- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44cc14cd90 msgr2=0x7f44cc06e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f44cc14cd90 0x7f44cc06e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 -- 172.21.15.1:0/3084588960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44cc15b610 con 0x7f44cc15b820 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44d2523640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44cc15b820 0x7f44cc14a460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f44cc15a3a0 tx=0x7f44c4011cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c4004240 con 0x7f44cc15b820 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f44c401b070 con 0x7f44cc15b820 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.139+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c400f770 con 0x7f44cc15b820 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.140+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44cc15c2b0 con 0x7f44cc15b820 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.140+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44cc14c7d0 con 0x7f44cc15b820 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.140+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f44cc156e60 con 0x7f44cc15b820 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.141+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f44c4012040 con 0x7f44cc15b820 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.142+0000 7f44ca7fc640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f44b80762d0 0x7f44b80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.142+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f44c4097f20 con 0x7f44cc15b820 2024-05-09T05:23:53.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.142+0000 7f44ca7fc640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f44b807e6a0 0x7f44b8080b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.142+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 --> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f44c4016e50 con 0x7f44b807e6a0 2024-05-09T05:23:53.141 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.142+0000 7f44d1d22640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f44b80762d0 0x7f44b80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.142+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f44c40982d0 con 0x7f44cc15b820 2024-05-09T05:23:53.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.142+0000 7f44d1521640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f44b807e6a0 0x7f44b8080b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.143+0000 7f44d1d22640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f44b80762d0 0x7f44b80787e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f44bc032000 tx=0x7f44bc02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.143+0000 7f44d1521640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f44b807e6a0 0x7f44b8080b70 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.142 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.144+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 <== osd.5 v2:172.21.15.196:6823/3481343234 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f44c4016e50 con 0x7f44b807e6a0 2024-05-09T05:23:53.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.157+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 --> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7f44cc050870 con 0x7f44b807e6a0 2024-05-09T05:23:53.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.157+0000 7f44ca7fc640 1 -- 172.21.15.1:0/3084588960 <== osd.5 v2:172.21.15.196:6823/3481343234 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7f44cc050870 con 0x7f44b807e6a0 2024-05-09T05:23:53.154 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:23:53.155 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:53.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f44b807e6a0 msgr2=0x7f44b8080b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6823/3481343234,v1:172.21.15.196:6825/3481343234] conn(0x7f44b807e6a0 0x7f44b8080b70 crc :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f44b80762d0 msgr2=0x7f44b80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f44b80762d0 0x7f44b80787e0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f44bc032000 tx=0x7f44bc02e000 comp rx=0 tx=0).stop 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44cc15b820 msgr2=0x7f44cc14a460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 --2- 172.21.15.1:0/3084588960 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f44cc15b820 0x7f44cc14a460 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f44cc15a3a0 tx=0x7f44c4011cf0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d2523640 1 -- 172.21.15.1:0/3084588960 reap_dead start 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 shutdown_connections 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 >> 172.21.15.1:0/3084588960 conn(0x7f44cc0d2360 msgr2=0x7f44cc155380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 shutdown_connections 2024-05-09T05:23:53.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.158+0000 7f44d3f86640 1 -- 172.21.15.1:0/3084588960 wait complete. 2024-05-09T05:23:53.166 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2024-05-09T05:23:53.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.286+0000 7f65cce2b640 1 Processor -- start 2024-05-09T05:23:53.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.286+0000 7f65cce2b640 1 -- start start 2024-05-09T05:23:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65cce2b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c8156bf0 0x7f65c806d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65cce2b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f65c80cc9d0 0x7f65c806ef50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65cce2b640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f65c80b5290 con 0x7f65c80cc5c0 2024-05-09T05:23:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65cce2b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f65c815ff50 con 0x7f65c80cc9d0 2024-05-09T05:23:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65cce2b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f65c80c8020 con 0x7f65c8156bf0 2024-05-09T05:23:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65bdd9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f65c80cc5c0 0x7f65c806e840 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:55312/0 (socket says 172.21.15.1:55312) 2024-05-09T05:23:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65bdd9b640 1 -- 172.21.15.1:0/2586552457 learned_addr learned my addr 172.21.15.1:0/2586552457 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:23:53.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65c6d9d640 1 --2- 172.21.15.1:0/2586552457 >> v2:172.21.15.196:3300/0 conn(0x7f65c80cc9d0 0x7f65c806ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.287+0000 7f65c659c640 1 --2- 172.21.15.1:0/2586552457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c8156bf0 0x7f65c806d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.288+0000 7f65c5d9b640 1 -- 172.21.15.1:0/2586552457 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3009435323 0 0) 0x7f65c80b5290 con 0x7f65c80cc5c0 2024-05-09T05:23:53.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.288+0000 7f65c5d9b640 1 -- 172.21.15.1:0/2586552457 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f65b4003520 con 0x7f65c80cc5c0 2024-05-09T05:23:53.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.288+0000 7f65c5d9b640 1 -- 172.21.15.1:0/2586552457 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 877488238 0 0) 0x7f65b4003520 con 0x7f65c80cc5c0 2024-05-09T05:23:53.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.288+0000 7f65c6d9d640 1 -- 172.21.15.1:0/2586552457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c8156bf0 msgr2=0x7f65c806d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:53.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.288+0000 7f65c6d9d640 1 --2- 172.21.15.1:0/2586552457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c8156bf0 0x7f65c806d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.288+0000 7f65c6d9d640 1 -- 172.21.15.1:0/2586552457 >> v1:172.21.15.1:6789/0 conn(0x7f65c80cc5c0 legacy=0x7f65c806e840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.288+0000 7f65c6d9d640 1 -- 172.21.15.1:0/2586552457 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c80c82e0 con 0x7f65c80cc9d0 2024-05-09T05:23:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.289+0000 7f65c6d9d640 1 --2- 172.21.15.1:0/2586552457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 0x7f65c806ef50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009d90 tx=0x7f65c002a5f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6208358189c268bc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.289+0000 7f65c5d9b640 1 -- 172.21.15.1:0/2586552457 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65b0001710 con 0x7f65c80cc9d0 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.289+0000 7f65c5d9b640 1 -- 172.21.15.1:0/2586552457 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65c0007540 con 0x7f65c80cc9d0 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.289+0000 7f65c5d9b640 1 -- 172.21.15.1:0/2586552457 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c00078f0 con 0x7f65c80cc9d0 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.290+0000 7f65cce2b640 1 -- 172.21.15.1:0/2586552457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 msgr2=0x7f65c806ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.290+0000 7f65cce2b640 1 --2- 172.21.15.1:0/2586552457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 0x7f65c806ef50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f65c0009d90 tx=0x7f65c002a5f0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.290+0000 7f65cce2b640 1 -- 172.21.15.1:0/2586552457 shutdown_connections 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.290+0000 7f65cce2b640 1 --2- 172.21.15.1:0/2586552457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 0x7f65c806ef50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.290+0000 7f65cce2b640 1 --2- 172.21.15.1:0/2586552457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c8156bf0 0x7f65c806d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.290+0000 7f65cce2b640 1 -- 172.21.15.1:0/2586552457 >> 172.21.15.1:0/2586552457 conn(0x7f65c8068db0 msgr2=0x7f65c80d1c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.290+0000 7f65cce2b640 1 -- 172.21.15.1:0/2586552457 shutdown_connections 2024-05-09T05:23:53.288 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.290+0000 7f65cce2b640 1 -- 172.21.15.1:0/2586552457 wait complete. 2024-05-09T05:23:53.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.292+0000 7f65cce2b640 1 Processor -- start 2024-05-09T05:23:53.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.292+0000 7f65cce2b640 1 -- start start 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65cce2b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c80cc5c0 0x7f65c8153a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65cce2b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 0x7f65c80c93b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65cce2b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8156bf0 0x7f65c81e9620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65cce2b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f65c81604d0 con 0x7f65c8156bf0 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65cce2b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f65c81601f0 con 0x7f65c80cc9d0 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65cce2b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f65c81607b0 con 0x7f65c80cc5c0 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8156bf0 0x7f65c81e9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c6d9d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8156bf0 0x7f65c81e9620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37414/0 (socket says 172.21.15.1:37414) 2024-05-09T05:23:53.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c6d9d640 1 -- 172.21.15.1:0/3738651343 learned_addr learned my addr 172.21.15.1:0/3738651343 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c659c640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c80cc5c0 0x7f65c8153a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65bdd9b640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 0x7f65c80c93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c6d9d640 1 -- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c80cc5c0 msgr2=0x7f65c8153a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c6d9d640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c80cc5c0 0x7f65c8153a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c6d9d640 1 -- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 msgr2=0x7f65c80c93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c6d9d640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 0x7f65c80c93b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.293+0000 7f65c6d9d640 1 -- 172.21.15.1:0/3738651343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65c8157d10 con 0x7f65c8156bf0 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.294+0000 7f65c6d9d640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8156bf0 0x7f65c81e9620 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f65c002abe0 tx=0x7f65c0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.294+0000 7f65bf7fe640 1 -- 172.21.15.1:0/3738651343 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c0009450 con 0x7f65c8156bf0 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.294+0000 7f65bf7fe640 1 -- 172.21.15.1:0/3738651343 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f65c0004240 con 0x7f65c8156bf0 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.294+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65c80c98f0 con 0x7f65c8156bf0 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.294+0000 7f65bf7fe640 1 -- 172.21.15.1:0/3738651343 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65c00045f0 con 0x7f65c8156bf0 2024-05-09T05:23:53.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.294+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65c80cb3b0 con 0x7f65c8156bf0 2024-05-09T05:23:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.295+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65c8157ee0 con 0x7f65c8156bf0 2024-05-09T05:23:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.296+0000 7f65bf7fe640 1 -- 172.21.15.1:0/3738651343 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f65c003b410 con 0x7f65c8156bf0 2024-05-09T05:23:53.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.296+0000 7f65bf7fe640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f65a80762d0 0x7f65a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.296+0000 7f65bf7fe640 1 -- 172.21.15.1:0/3738651343 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f65c00b8750 con 0x7f65c8156bf0 2024-05-09T05:23:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.296+0000 7f65c659c640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f65a80762d0 0x7f65a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.294 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.297+0000 7f65c659c640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f65a80762d0 0x7f65a80787e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009aa0 tx=0x7f65b8009190 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.298 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.301+0000 7f65bf7fe640 1 -- 172.21.15.1:0/3738651343 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f65c003f030 con 0x7f65c8156bf0 2024-05-09T05:23:53.492 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.495+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) -- 0x7f65c8050870 con 0x7f65a80762d0 2024-05-09T05:23:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.497+0000 7f65bf7fe640 1 -- 172.21.15.1:0/3738651343 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f65c8050870 con 0x7f65a80762d0 2024-05-09T05:23:53.493 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 5 to deep-scrub 2024-05-09T05:23:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f65a80762d0 msgr2=0x7f65a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.495 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f65a80762d0 0x7f65a80787e0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f65b8009aa0 tx=0x7f65b8009190 comp rx=0 tx=0).stop 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8156bf0 msgr2=0x7f65c81e9620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8156bf0 0x7f65c81e9620 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f65c002abe0 tx=0x7f65c0004a40 comp rx=0 tx=0).stop 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 shutdown_connections 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f65a80762d0 0x7f65a80787e0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f65c80cc9d0 0x7f65c80c93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f65c8156bf0 0x7f65c81e9620 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 --2- 172.21.15.1:0/3738651343 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f65c80cc5c0 0x7f65c8153a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 >> 172.21.15.1:0/3738651343 conn(0x7f65c8068db0 msgr2=0x7f65c81541b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 shutdown_connections 2024-05-09T05:23:53.496 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.498+0000 7f65cce2b640 1 -- 172.21.15.1:0/3738651343 wait complete. 2024-05-09T05:23:53.507 INFO:tasks.ceph:Scrubbing osd.6 2024-05-09T05:23:53.507 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:23:53.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.629+0000 7ff81642b640 1 Processor -- start 2024-05-09T05:23:53.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.629+0000 7ff81642b640 1 -- start start 2024-05-09T05:23:53.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.630+0000 7ff81642b640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff8100ce010 0x7ff8100c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.630+0000 7ff81642b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100c77c0 0x7ff8100c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.630+0000 7ff81642b640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff81015c7b0 con 0x7ff8100b4ec0 2024-05-09T05:23:53.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.630+0000 7ff81642b640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff81015d0c0 con 0x7ff8100ce010 2024-05-09T05:23:53.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.630+0000 7ff81642b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff8100cca50 con 0x7ff8100c77c0 2024-05-09T05:23:53.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff8149c8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100c77c0 0x7ff8100c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff80f7fe640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff8100ce010 0x7ff8100c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.628 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff8149c8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100c77c0 0x7ff8100c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:59394/0 (socket says 172.21.15.1:59394) 2024-05-09T05:23:53.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff8149c8640 1 -- 172.21.15.1:0/2178848614 learned_addr learned my addr 172.21.15.1:0/2178848614 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:53.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff8149c8640 1 -- 172.21.15.1:0/2178848614 >> v2:172.21.15.196:3300/0 conn(0x7ff8100ce010 msgr2=0x7ff8100c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:53.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff8149c8640 1 --2- 172.21.15.1:0/2178848614 >> v2:172.21.15.196:3300/0 conn(0x7ff8100ce010 0x7ff8100c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff8149c8640 1 -- 172.21.15.1:0/2178848614 >> v1:172.21.15.196:6789/0 conn(0x7ff8100b4ec0 legacy=0x7ff8100cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff8149c8640 1 -- 172.21.15.1:0/2178848614 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8100ccd30 con 0x7ff8100c77c0 2024-05-09T05:23:53.629 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.631+0000 7ff80f7fe640 1 --2- 172.21.15.1:0/2178848614 >> v2:172.21.15.196:3300/0 conn(0x7ff8100ce010 0x7ff8100c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:23:53.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.632+0000 7ff8149c8640 1 --2- 172.21.15.1:0/2178848614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100c77c0 0x7ff8100c7bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff808009e50 tx=0x7ff80802a7a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=648d807479a46e90 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.632+0000 7ff80effd640 1 -- 172.21.15.1:0/2178848614 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff808036070 con 0x7ff8100c77c0 2024-05-09T05:23:53.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.632+0000 7ff80effd640 1 -- 172.21.15.1:0/2178848614 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff808031600 con 0x7ff8100c77c0 2024-05-09T05:23:53.630 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.632+0000 7ff80effd640 1 -- 172.21.15.1:0/2178848614 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8080319d0 con 0x7ff8100c77c0 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.633+0000 7ff81642b640 1 -- 172.21.15.1:0/2178848614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100c77c0 msgr2=0x7ff8100c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.633+0000 7ff81642b640 1 --2- 172.21.15.1:0/2178848614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100c77c0 0x7ff8100c7bd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7ff808009e50 tx=0x7ff80802a7a0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.633+0000 7ff81642b640 1 -- 172.21.15.1:0/2178848614 shutdown_connections 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.633+0000 7ff81642b640 1 --2- 172.21.15.1:0/2178848614 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100c77c0 0x7ff8100c7bd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.633+0000 7ff81642b640 1 --2- 172.21.15.1:0/2178848614 >> v2:172.21.15.196:3300/0 conn(0x7ff8100ce010 0x7ff8100c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.633+0000 7ff81642b640 1 -- 172.21.15.1:0/2178848614 >> 172.21.15.1:0/2178848614 conn(0x7ff8100d2360 msgr2=0x7ff8100ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.633+0000 7ff81642b640 1 -- 172.21.15.1:0/2178848614 shutdown_connections 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.633+0000 7ff81642b640 1 -- 172.21.15.1:0/2178848614 wait complete. 2024-05-09T05:23:53.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.635+0000 7ff81642b640 1 Processor -- start 2024-05-09T05:23:53.632 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.635+0000 7ff81642b640 1 -- start start 2024-05-09T05:23:53.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff81642b640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100b4ec0 0x7ff8101f34d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff81642b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8100c77c0 0x7ff8101e83e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff81642b640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8100ce010 0x7ff8101e6c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff81642b640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff8100cc750 con 0x7ff8100c77c0 2024-05-09T05:23:53.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff81642b640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff81015cd50 con 0x7ff8100ce010 2024-05-09T05:23:53.634 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff81642b640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff8100c8110 con 0x7ff8100b4ec0 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8100c77c0 0x7ff8101e83e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80f7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8100c77c0 0x7ff8101e83e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37420/0 (socket says 172.21.15.1:37420) 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80f7fe640 1 -- 172.21.15.1:0/3187855601 learned_addr learned my addr 172.21.15.1:0/3187855601 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff8149c8640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8100ce010 0x7ff8101e6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80ffff640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100b4ec0 0x7ff8101f34d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80f7fe640 1 -- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100b4ec0 msgr2=0x7ff8101f34d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80f7fe640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff8100b4ec0 0x7ff8101f34d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80f7fe640 1 -- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8100ce010 msgr2=0x7ff8101e6c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80f7fe640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8100ce010 0x7ff8101e6c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff80f7fe640 1 -- 172.21.15.1:0/3187855601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8101e8920 con 0x7ff8100c77c0 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.636+0000 7ff8149c8640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff8100ce010 0x7ff8101e6c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.637+0000 7ff80f7fe640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8100c77c0 0x7ff8101e83e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff80400e990 tx=0x7ff80400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.637+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff804017070 con 0x7ff8100c77c0 2024-05-09T05:23:53.635 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.637+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff80400c9e0 con 0x7ff8100c77c0 2024-05-09T05:23:53.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.637+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff80400cdb0 con 0x7ff8100c77c0 2024-05-09T05:23:53.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.637+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8101e7150 con 0x7ff8100c77c0 2024-05-09T05:23:53.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.637+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8100c9250 con 0x7ff8100c77c0 2024-05-09T05:23:53.636 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.638+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff810050870 con 0x7ff8100c77c0 2024-05-09T05:23:53.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.639+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff80401a070 con 0x7ff8100c77c0 2024-05-09T05:23:53.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.639+0000 7ff80cff9640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7fc0762d0 0x7ff7fc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.639+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff804097ac0 con 0x7ff8100c77c0 2024-05-09T05:23:53.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.640+0000 7ff80cff9640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7ff7fc07e6a0 0x7ff7fc080b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.640+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 --> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff804016ec0 con 0x7ff7fc07e6a0 2024-05-09T05:23:53.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.640+0000 7ff80ffff640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7fc0762d0 0x7ff7fc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.640+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7ff804097e70 con 0x7ff8100c77c0 2024-05-09T05:23:53.637 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.640+0000 7ff8149c8640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7ff7fc07e6a0 0x7ff7fc080b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.640+0000 7ff80ffff640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7fc0762d0 0x7ff7fc0787e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff8000015e0 tx=0x7ff800009040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.640+0000 7ff8149c8640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7ff7fc07e6a0 0x7ff7fc080b70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.638 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.641+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 <== osd.6 v2:172.21.15.196:6800/3854610795 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff804016ec0 con 0x7ff7fc07e6a0 2024-05-09T05:23:53.651 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.654+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 --> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7ff8101e8c60 con 0x7ff7fc07e6a0 2024-05-09T05:23:53.652 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.655+0000 7ff80cff9640 1 -- 172.21.15.1:0/3187855601 <== osd.6 v2:172.21.15.196:6800/3854610795 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7ff8101e8c60 con 0x7ff7fc07e6a0 2024-05-09T05:23:53.652 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:53.652 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:23:53.652 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:53.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.655+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7ff7fc07e6a0 msgr2=0x7ff7fc080b70 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.655+0000 7ff81642b640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6800/3854610795,v1:172.21.15.196:6801/3854610795] conn(0x7ff7fc07e6a0 0x7ff7fc080b70 crc :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.653 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.655+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7fc0762d0 msgr2=0x7ff7fc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.655+0000 7ff81642b640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff7fc0762d0 0x7ff7fc0787e0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ff8000015e0 tx=0x7ff800009040 comp rx=0 tx=0).stop 2024-05-09T05:23:53.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.655+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8100c77c0 msgr2=0x7ff8101e83e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.655+0000 7ff81642b640 1 --2- 172.21.15.1:0/3187855601 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff8100c77c0 0x7ff8101e83e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7ff80400e990 tx=0x7ff80400ee50 comp rx=0 tx=0).stop 2024-05-09T05:23:53.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.655+0000 7ff8149c8640 1 -- 172.21.15.1:0/3187855601 reap_dead start 2024-05-09T05:23:53.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.656+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 shutdown_connections 2024-05-09T05:23:53.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.656+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 >> 172.21.15.1:0/3187855601 conn(0x7ff8100d2360 msgr2=0x7ff81015b580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:53.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.656+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 shutdown_connections 2024-05-09T05:23:53.654 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.656+0000 7ff81642b640 1 -- 172.21.15.1:0/3187855601 wait complete. 2024-05-09T05:23:53.664 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2024-05-09T05:23:53.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.787+0000 7ff6aa805640 1 Processor -- start 2024-05-09T05:23:53.785 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.787+0000 7ff6aa805640 1 -- start start 2024-05-09T05:23:53.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff6aa805640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7ff6a40cf360 0x7ff6a40b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff6aa805640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6a40b4570 0x7ff6a4156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.786 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff6aa805640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff6a40ce5c0 con 0x7ff6a415c9a0 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff6aa805640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ff6a415d4c0 con 0x7ff6a40b4570 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff6aa805640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7ff6a415d7a0 con 0x7ff6a40cf360 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff69bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6a40b4570 0x7ff6a4156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff69bfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ff6a40b4570 0x7ff6a4156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37434/0 (socket says 172.21.15.1:37434) 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff69bfff640 1 -- 172.21.15.1:0/480368987 learned_addr learned my addr 172.21.15.1:0/480368987 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff6a3fff640 1 --2- 172.21.15.1:0/480368987 >> v2:172.21.15.196:3300/0 conn(0x7ff6a40cf360 0x7ff6a40b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff69bfff640 1 -- 172.21.15.1:0/480368987 >> v2:172.21.15.196:3300/0 conn(0x7ff6a40cf360 msgr2=0x7ff6a40b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff69bfff640 1 --2- 172.21.15.1:0/480368987 >> v2:172.21.15.196:3300/0 conn(0x7ff6a40cf360 0x7ff6a40b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff69bfff640 1 -- 172.21.15.1:0/480368987 >> v1:172.21.15.196:6789/0 conn(0x7ff6a415c9a0 legacy=0x7ff6a40cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.789+0000 7ff69bfff640 1 -- 172.21.15.1:0/480368987 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a40ce880 con 0x7ff6a40b4570 2024-05-09T05:23:53.787 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.790+0000 7ff69bfff640 1 --2- 172.21.15.1:0/480368987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a4156b50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff6a408f300 tx=0x7ff68c02a300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c54859e98cefd4d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.788 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.790+0000 7ff6a37fe640 1 -- 172.21.15.1:0/480368987 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c037070 con 0x7ff6a40b4570 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.790+0000 7ff6a37fe640 1 -- 172.21.15.1:0/480368987 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff68c02ae50 con 0x7ff6a40b4570 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.790+0000 7ff6a37fe640 1 -- 172.21.15.1:0/480368987 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff68c032860 con 0x7ff6a40b4570 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.791+0000 7ff6aa805640 1 -- 172.21.15.1:0/480368987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 msgr2=0x7ff6a4156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.791+0000 7ff6aa805640 1 --2- 172.21.15.1:0/480368987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a4156b50 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7ff6a408f300 tx=0x7ff68c02a300 comp rx=0 tx=0).stop 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.791+0000 7ff6aa805640 1 -- 172.21.15.1:0/480368987 shutdown_connections 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.791+0000 7ff6aa805640 1 --2- 172.21.15.1:0/480368987 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a4156b50 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.791+0000 7ff6aa805640 1 --2- 172.21.15.1:0/480368987 >> v2:172.21.15.196:3300/0 conn(0x7ff6a40cf360 0x7ff6a40b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.791+0000 7ff6aa805640 1 -- 172.21.15.1:0/480368987 >> 172.21.15.1:0/480368987 conn(0x7ff6a406de10 msgr2=0x7ff6a406e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.791+0000 7ff6aa805640 1 -- 172.21.15.1:0/480368987 shutdown_connections 2024-05-09T05:23:53.789 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.791+0000 7ff6aa805640 1 -- 172.21.15.1:0/480368987 wait complete. 2024-05-09T05:23:53.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.793+0000 7ff6aa805640 1 Processor -- start 2024-05-09T05:23:53.790 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.793+0000 7ff6aa805640 1 -- start start 2024-05-09T05:23:53.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6aa805640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a41e6f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6aa805640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a415c9a0 0x7ff6a40d0a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.792 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6aa805640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a40c85d0 0x7ff6a4156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6aa805640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a40b5090 con 0x7ff6a40b4570 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6aa805640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a40cee40 con 0x7ff6a415c9a0 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6aa805640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff6a4157090 con 0x7ff6a40c85d0 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a41e6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a41e6f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37444/0 (socket says 172.21.15.1:37444) 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 -- 172.21.15.1:0/3308557717 learned_addr learned my addr 172.21.15.1:0/3308557717 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a8da2640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a40c85d0 0x7ff6a4156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff69bfff640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a415c9a0 0x7ff6a40d0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 -- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a40c85d0 msgr2=0x7ff6a4156110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a40c85d0 0x7ff6a4156110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 -- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a415c9a0 msgr2=0x7ff6a40d0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a415c9a0 0x7ff6a40d0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 -- 172.21.15.1:0/3308557717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a41e7470 con 0x7ff6a40b4570 2024-05-09T05:23:53.793 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a8da2640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a40c85d0 0x7ff6a4156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff69bfff640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a415c9a0 0x7ff6a40d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:23:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.794+0000 7ff6a3fff640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a41e6f30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff69400e870 tx=0x7ff69400ec70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.795+0000 7ff6a17fa640 1 -- 172.21.15.1:0/3308557717 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff694017070 con 0x7ff6a40b4570 2024-05-09T05:23:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.795+0000 7ff6a17fa640 1 -- 172.21.15.1:0/3308557717 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7ff69400c940 con 0x7ff6a40b4570 2024-05-09T05:23:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.795+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a41f5720 con 0x7ff6a40b4570 2024-05-09T05:23:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.795+0000 7ff6a17fa640 1 -- 172.21.15.1:0/3308557717 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69400cd30 con 0x7ff6a40b4570 2024-05-09T05:23:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.795+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a41e98c0 con 0x7ff6a40b4570 2024-05-09T05:23:53.794 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.796+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a4050870 con 0x7ff6a40b4570 2024-05-09T05:23:53.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.796+0000 7ff6a17fa640 1 -- 172.21.15.1:0/3308557717 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff69401a070 con 0x7ff6a40b4570 2024-05-09T05:23:53.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.797+0000 7ff6a17fa640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6840762d0 0x7ff6840787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:53.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.797+0000 7ff6a17fa640 1 -- 172.21.15.1:0/3308557717 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ff694097860 con 0x7ff6a40b4570 2024-05-09T05:23:53.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.797+0000 7ff69bfff640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6840762d0 0x7ff6840787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:53.795 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.798+0000 7ff69bfff640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6840762d0 0x7ff6840787e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff68c034000 tx=0x7ff68c02e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:53.799 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.802+0000 7ff6a17fa640 1 -- 172.21.15.1:0/3308557717 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff6940611d0 con 0x7ff6a40b4570 2024-05-09T05:23:53.990 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.993+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) -- 0x7ff6a40cdf30 con 0x7ff6840762d0 2024-05-09T05:23:53.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.995+0000 7ff6a17fa640 1 -- 172.21.15.1:0/3308557717 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7ff6a40cdf30 con 0x7ff6840762d0 2024-05-09T05:23:53.991 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 6 to deep-scrub 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6840762d0 msgr2=0x7ff6840787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6840762d0 0x7ff6840787e0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ff68c034000 tx=0x7ff68c02e000 comp rx=0 tx=0).stop 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 msgr2=0x7ff6a41e6f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a41e6f30 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff69400e870 tx=0x7ff69400ec70 comp rx=0 tx=0).stop 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 shutdown_connections 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff6840762d0 0x7ff6840787e0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff6a415c9a0 0x7ff6a40d0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff6a40c85d0 0x7ff6a4156110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 --2- 172.21.15.1:0/3308557717 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff6a40b4570 0x7ff6a41e6f30 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.996+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 >> 172.21.15.1:0/3308557717 conn(0x7ff6a406de10 msgr2=0x7ff6a40cfc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.997+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 shutdown_connections 2024-05-09T05:23:53.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:53.997+0000 7ff6aa805640 1 -- 172.21.15.1:0/3308557717 wait complete. 2024-05-09T05:23:54.005 INFO:tasks.ceph:Scrubbing osd.7 2024-05-09T05:23:54.005 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2024-05-09T05:23:54.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.131+0000 7fa7936f9640 1 Processor -- start 2024-05-09T05:23:54.128 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.131+0000 7fa7936f9640 1 -- start start 2024-05-09T05:23:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.132+0000 7fa7936f9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa78c0cf360 0x7fa78c0b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.132+0000 7fa7936f9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fa78c0b4570 0x7fa78c156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa7936f9640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fa78c0ce5c0 con 0x7fa78c15c9a0 2024-05-09T05:23:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa7936f9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fa78c15d4c0 con 0x7fa78c0cf360 2024-05-09T05:23:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa7936f9640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fa78c15d7a0 con 0x7fa78c0b4570 2024-05-09T05:23:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa791495640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa78c0cf360 0x7fa78c0b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa791495640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fa78c0cf360 0x7fa78c0b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37452/0 (socket says 172.21.15.1:37452) 2024-05-09T05:23:54.132 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa791495640 1 -- 172.21.15.1:0/1552926919 learned_addr learned my addr 172.21.15.1:0/1552926919 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa790c94640 1 --2- 172.21.15.1:0/1552926919 >> v2:172.21.15.196:3300/0 conn(0x7fa78c0b4570 0x7fa78c156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa791495640 1 -- 172.21.15.1:0/1552926919 >> v2:172.21.15.196:3300/0 conn(0x7fa78c0b4570 msgr2=0x7fa78c156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa791495640 1 --2- 172.21.15.1:0/1552926919 >> v2:172.21.15.196:3300/0 conn(0x7fa78c0b4570 0x7fa78c156b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa791495640 1 -- 172.21.15.1:0/1552926919 >> v1:172.21.15.196:6789/0 conn(0x7fa78c15c9a0 legacy=0x7fa78c0cd3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.133+0000 7fa791495640 1 -- 172.21.15.1:0/1552926919 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c0ce880 con 0x7fa78c0cf360 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.134+0000 7fa791495640 1 --2- 172.21.15.1:0/1552926919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c0cf360 0x7fa78c0b4b50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0cf8c0 tx=0x7fa78802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eae620b3a9451178 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.134+0000 7fa783fff640 1 -- 172.21.15.1:0/1552926919 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa788037070 con 0x7fa78c0cf360 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.134+0000 7fa783fff640 1 -- 172.21.15.1:0/1552926919 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa788032440 con 0x7fa78c0cf360 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.134+0000 7fa783fff640 1 -- 172.21.15.1:0/1552926919 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa788032810 con 0x7fa78c0cf360 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.135+0000 7fa7936f9640 1 -- 172.21.15.1:0/1552926919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c0cf360 msgr2=0x7fa78c0b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.135+0000 7fa7936f9640 1 --2- 172.21.15.1:0/1552926919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c0cf360 0x7fa78c0b4b50 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fa78c0cf8c0 tx=0x7fa78802a270 comp rx=0 tx=0).stop 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.135+0000 7fa7936f9640 1 -- 172.21.15.1:0/1552926919 shutdown_connections 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.135+0000 7fa7936f9640 1 --2- 172.21.15.1:0/1552926919 >> v2:172.21.15.196:3300/0 conn(0x7fa78c0b4570 0x7fa78c156b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.135+0000 7fa7936f9640 1 --2- 172.21.15.1:0/1552926919 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c0cf360 0x7fa78c0b4b50 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.135+0000 7fa7936f9640 1 -- 172.21.15.1:0/1552926919 >> 172.21.15.1:0/1552926919 conn(0x7fa78c06de10 msgr2=0x7fa78c06e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:54.133 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.135+0000 7fa7936f9640 1 -- 172.21.15.1:0/1552926919 shutdown_connections 2024-05-09T05:23:54.134 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.135+0000 7fa7936f9640 1 -- 172.21.15.1:0/1552926919 wait complete. 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.136+0000 7fa7936f9640 1 Processor -- start 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.137+0000 7fa7936f9640 1 -- start start 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa7936f9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa78c0b4570 0x7fa78c157550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa7936f9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c15c9a0 0x7fa78c159b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa7936f9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa78c0c6e20 0x7fa78c0cad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa7936f9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c0b5090 con 0x7fa78c15c9a0 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa7936f9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fa78c0cee40 con 0x7fa78c0c6e20 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa7936f9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fa78c157090 con 0x7fa78c0b4570 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c15c9a0 0x7fa78c159b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c15c9a0 0x7fa78c159b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37462/0 (socket says 172.21.15.1:37462) 2024-05-09T05:23:54.136 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 -- 172.21.15.1:0/329565902 learned_addr learned my addr 172.21.15.1:0/329565902 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa791495640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa78c0b4570 0x7fa78c157550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa791c96640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa78c0c6e20 0x7fa78c0cad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 -- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa78c0b4570 msgr2=0x7fa78c157550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fa78c0b4570 0x7fa78c157550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 -- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa78c0c6e20 msgr2=0x7fa78c0cad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fa78c0c6e20 0x7fa78c0cad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 -- 172.21.15.1:0/329565902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa78c157a90 con 0x7fa78c15c9a0 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.138+0000 7fa790c94640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c15c9a0 0x7fa78c159b60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00d5f0 tx=0x7fa77c00dab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.139+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c0125f0 con 0x7fa78c15c9a0 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.139+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fa77c019070 con 0x7fa78c15c9a0 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.139+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa78c0d1090 con 0x7fa78c15c9a0 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.139+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa77c0129d0 con 0x7fa78c15c9a0 2024-05-09T05:23:54.137 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.139+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa78c1f63e0 con 0x7fa78c15c9a0 2024-05-09T05:23:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.140+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa78c050870 con 0x7fa78c15c9a0 2024-05-09T05:23:54.138 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.141+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fa77c012b70 con 0x7fa78c15c9a0 2024-05-09T05:23:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.141+0000 7fa781ffb640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa770076300 0x7fa770078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.141+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fa77c098500 con 0x7fa78c15c9a0 2024-05-09T05:23:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.141+0000 7fa781ffb640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7fa77007e6d0 0x7fa770080ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.141+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 --> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa770081270 con 0x7fa77007e6d0 2024-05-09T05:23:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.141+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fa77c098990 con 0x7fa78c15c9a0 2024-05-09T05:23:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.141+0000 7fa791495640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa770076300 0x7fa770078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.142+0000 7fa791c96640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7fa77007e6d0 0x7fa770080ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.139 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.142+0000 7fa791c96640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7fa77007e6d0 0x7fa770080ba0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.142+0000 7fa791495640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa770076300 0x7fa770078810 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa788034000 tx=0x7fa78802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.140 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.143+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 <== osd.7 v2:172.21.15.196:6816/747967558 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa770081270 con 0x7fa77007e6d0 2024-05-09T05:23:54.153 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.156+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 --> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) -- 0x7fa78c0cdf30 con 0x7fa77007e6d0 2024-05-09T05:23:54.154 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa781ffb640 1 -- 172.21.15.1:0/329565902 <== osd.7 v2:172.21.15.196:6816/747967558 2 ==== command_reply(tid 2: 0 ) ==== 8+0+97 (crc 0 0 0) 0x7fa78c0cdf30 con 0x7fa77007e6d0 2024-05-09T05:23:54.154 INFO:teuthology.orchestra.run.smithi001.stdout:{ 2024-05-09T05:23:54.154 INFO:teuthology.orchestra.run.smithi001.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-05-09T05:23:54.154 INFO:teuthology.orchestra.run.smithi001.stdout:} 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7fa77007e6d0 msgr2=0x7fa770080ba0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6816/747967558,v1:172.21.15.196:6817/747967558] conn(0x7fa77007e6d0 0x7fa770080ba0 crc :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa770076300 msgr2=0x7fa770078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fa770076300 0x7fa770078810 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa788034000 tx=0x7fa78802e000 comp rx=0 tx=0).stop 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c15c9a0 msgr2=0x7fa78c159b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 --2- 172.21.15.1:0/329565902 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fa78c15c9a0 0x7fa78c159b60 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa77c00d5f0 tx=0x7fa77c00dab0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa791c96640 1 -- 172.21.15.1:0/329565902 reap_dead start 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 shutdown_connections 2024-05-09T05:23:54.155 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 >> 172.21.15.1:0/329565902 conn(0x7fa78c06de10 msgr2=0x7fa78c15ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:54.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 shutdown_connections 2024-05-09T05:23:54.156 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.158+0000 7fa7936f9640 1 -- 172.21.15.1:0/329565902 wait complete. 2024-05-09T05:23:54.165 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2024-05-09T05:23:54.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.284+0000 7f829ee70640 1 Processor -- start 2024-05-09T05:23:54.281 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.284+0000 7f829ee70640 1 -- start start 2024-05-09T05:23:54.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829ee70640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f82980b4ec0 0x7f82980cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829ee70640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8298160460 0x7f8298160870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829ee70640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f829815c7b0 con 0x7f82980ce010 2024-05-09T05:23:54.283 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829ee70640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f829815d0c0 con 0x7f8298160460 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829ee70640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f82980cca50 con 0x7f82980b4ec0 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829d40d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8298160460 0x7f8298160870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829d40d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f8298160460 0x7f8298160870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37464/0 (socket says 172.21.15.1:37464) 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829d40d640 1 -- 172.21.15.1:0/3023679504 learned_addr learned my addr 172.21.15.1:0/3023679504 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829cc0c640 1 --2- 172.21.15.1:0/3023679504 >> v2:172.21.15.196:3300/0 conn(0x7f82980b4ec0 0x7f82980cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829d40d640 1 -- 172.21.15.1:0/3023679504 >> v2:172.21.15.196:3300/0 conn(0x7f82980b4ec0 msgr2=0x7f82980cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829d40d640 1 --2- 172.21.15.1:0/3023679504 >> v2:172.21.15.196:3300/0 conn(0x7f82980b4ec0 0x7f82980cf9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829d40d640 1 -- 172.21.15.1:0/3023679504 >> v1:172.21.15.196:6789/0 conn(0x7f82980ce010 legacy=0x7f829815fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:54.284 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.286+0000 7f829d40d640 1 -- 172.21.15.1:0/3023679504 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f82980ccd30 con 0x7f8298160460 2024-05-09T05:23:54.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.287+0000 7f829d40d640 1 --2- 172.21.15.1:0/3023679504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f8298160870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f829815d240 tx=0x7f829002a7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=19d3dcf9e1522a69 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.287+0000 7f82977fe640 1 -- 172.21.15.1:0/3023679504 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290036070 con 0x7f8298160460 2024-05-09T05:23:54.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.287+0000 7f82977fe640 1 -- 172.21.15.1:0/3023679504 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8290031600 con 0x7f8298160460 2024-05-09T05:23:54.285 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.287+0000 7f82977fe640 1 -- 172.21.15.1:0/3023679504 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f82900319f0 con 0x7f8298160460 2024-05-09T05:23:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.288+0000 7f829ee70640 1 -- 172.21.15.1:0/3023679504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 msgr2=0x7f8298160870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.288+0000 7f829ee70640 1 --2- 172.21.15.1:0/3023679504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f8298160870 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f829815d240 tx=0x7f829002a7c0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.288+0000 7f829ee70640 1 -- 172.21.15.1:0/3023679504 shutdown_connections 2024-05-09T05:23:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.288+0000 7f829ee70640 1 --2- 172.21.15.1:0/3023679504 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f8298160870 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.288+0000 7f829ee70640 1 --2- 172.21.15.1:0/3023679504 >> v2:172.21.15.196:3300/0 conn(0x7f82980b4ec0 0x7f82980cf9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.288+0000 7f829ee70640 1 -- 172.21.15.1:0/3023679504 >> 172.21.15.1:0/3023679504 conn(0x7f82980d2360 msgr2=0x7f82980ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.288+0000 7f829ee70640 1 -- 172.21.15.1:0/3023679504 shutdown_connections 2024-05-09T05:23:54.286 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.288+0000 7f829ee70640 1 -- 172.21.15.1:0/3023679504 wait complete. 2024-05-09T05:23:54.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.290+0000 7f829ee70640 1 Processor -- start 2024-05-09T05:23:54.287 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.290+0000 7f829ee70640 1 -- start start 2024-05-09T05:23:54.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829ee70640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82980b4ec0 0x7f82981553f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.289 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829ee70640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f82980ce010 0x7f829815ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829ee70640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f82981e8380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829ee70640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f82980cc750 con 0x7f8298160460 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829ee70640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f829815cd50 con 0x7f82980b4ec0 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829ee70640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f82980cff10 con 0x7f82980ce010 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829d40d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f82981e8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829d40d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f82981e8380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37466/0 (socket says 172.21.15.1:37466) 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.291+0000 7f829d40d640 1 -- 172.21.15.1:0/1647494466 learned_addr learned my addr 172.21.15.1:0/1647494466 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f8297fff640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f82980ce010 0x7f829815ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829cc0c640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82980b4ec0 0x7f82981553f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829d40d640 1 -- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f82980ce010 msgr2=0x7f829815ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829d40d640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f82980ce010 0x7f829815ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829d40d640 1 -- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82980b4ec0 msgr2=0x7f82981553f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.290 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829d40d640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82980b4ec0 0x7f82981553f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829d40d640 1 -- 172.21.15.1:0/1647494466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f829815ef40 con 0x7f8298160460 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829d40d640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f82981e8380 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8290009f10 tx=0x7f8290004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f82957fa640 1 -- 172.21.15.1:0/1647494466 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290036050 con 0x7f8298160460 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f82957fa640 1 -- 172.21.15.1:0/1647494466 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f8290042070 con 0x7f8298160460 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f82981f70f0 con 0x7f8298160460 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f82957fa640 1 -- 172.21.15.1:0/1647494466 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8290009510 con 0x7f8298160460 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.292+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8298155110 con 0x7f8298160460 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.293+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f82980b5f80 con 0x7f8298160460 2024-05-09T05:23:54.291 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.294+0000 7f82957fa640 1 -- 172.21.15.1:0/1647494466 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f8290004470 con 0x7f8298160460 2024-05-09T05:23:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.295+0000 7f82957fa640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f827c0762d0 0x7f827c0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.295+0000 7f82957fa640 1 -- 172.21.15.1:0/1647494466 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f82900b8d00 con 0x7f8298160460 2024-05-09T05:23:54.292 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.295+0000 7f829cc0c640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f827c0762d0 0x7f827c0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.293 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.296+0000 7f829cc0c640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f827c0762d0 0x7f827c0787e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f828c0079a0 tx=0x7f828c008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.295 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.299+0000 7f82957fa640 1 -- 172.21.15.1:0/1647494466 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f82980b5f80 con 0x7f8298160460 2024-05-09T05:23:54.485 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.488+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) -- 0x7f8298050870 con 0x7f827c0762d0 2024-05-09T05:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.489+0000 7f82957fa640 1 -- 172.21.15.1:0/1647494466 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) ==== 41+0+0 (secure 0 0 0) 0x7f8298050870 con 0x7f827c0762d0 2024-05-09T05:23:54.486 INFO:teuthology.orchestra.run.smithi001.stderr:instructed osd(s) 7 to deep-scrub 2024-05-09T05:23:54.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f827c0762d0 msgr2=0x7f827c0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.488 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f827c0762d0 0x7f827c0787e0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f828c0079a0 tx=0x7f828c008040 comp rx=0 tx=0).stop 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 msgr2=0x7f82981e8380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f82981e8380 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8290009f10 tx=0x7f8290004a40 comp rx=0 tx=0).stop 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 shutdown_connections 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f827c0762d0 0x7f827c0787e0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f8298160460 0x7f82981e8380 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f82980ce010 0x7f829815ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 --2- 172.21.15.1:0/1647494466 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f82980b4ec0 0x7f82981553f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 >> 172.21.15.1:0/1647494466 conn(0x7f82980d2360 msgr2=0x7f8298153420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 shutdown_connections 2024-05-09T05:23:54.489 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.491+0000 7f829ee70640 1 -- 172.21.15.1:0/1647494466 wait complete. 2024-05-09T05:23:54.500 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:23:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.619+0000 7fcb00c96640 1 Processor -- start 2024-05-09T05:23:54.616 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.619+0000 7fcb00c96640 1 -- start start 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcb00c96640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 0x7fcafc156c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcb00c96640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcafc06e600 0x7fcafc070bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcb00c96640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fcafc0c6bc0 con 0x7fcafc06e1f0 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcb00c96640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fcafc15c8e0 con 0x7fcafc06e600 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcb00c96640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcafc06eaf0 con 0x7fcafc0c64b0 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcafad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcafc06e600 0x7fcafc070bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcafad9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fcafc06e600 0x7fcafc070bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37478/0 (socket says 172.21.15.1:37478) 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcafad9d640 1 -- 172.21.15.1:0/445689222 learned_addr learned my addr 172.21.15.1:0/445689222 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcafa59c640 1 --2- 172.21.15.1:0/445689222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 0x7fcafc156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.619 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcafad9d640 1 -- 172.21.15.1:0/445689222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 msgr2=0x7fcafc156c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:23:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcafad9d640 1 --2- 172.21.15.1:0/445689222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 0x7fcafc156c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcafad9d640 1 -- 172.21.15.1:0/445689222 >> v1:172.21.15.196:6789/0 conn(0x7fcafc06e1f0 legacy=0x7fcafc068650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.621+0000 7fcafad9d640 1 -- 172.21.15.1:0/445689222 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcafc15cba0 con 0x7fcafc06e600 2024-05-09T05:23:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.622+0000 7fcafad9d640 1 --2- 172.21.15.1:0/445689222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc070bd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009870 tx=0x7fcaf402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ac2fa692280cc83 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.622+0000 7fcaf959a640 1 -- 172.21.15.1:0/445689222 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4037070 con 0x7fcafc06e600 2024-05-09T05:23:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.622+0000 7fcaf959a640 1 -- 172.21.15.1:0/445689222 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcaf4032440 con 0x7fcafc06e600 2024-05-09T05:23:54.620 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.622+0000 7fcaf959a640 1 -- 172.21.15.1:0/445689222 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4032810 con 0x7fcafc06e600 2024-05-09T05:23:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.623+0000 7fcb00c96640 1 -- 172.21.15.1:0/445689222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 msgr2=0x7fcafc070bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.623+0000 7fcb00c96640 1 --2- 172.21.15.1:0/445689222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc070bd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4009870 tx=0x7fcaf402a270 comp rx=0 tx=0).stop 2024-05-09T05:23:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.623+0000 7fcb00c96640 1 -- 172.21.15.1:0/445689222 shutdown_connections 2024-05-09T05:23:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.623+0000 7fcb00c96640 1 --2- 172.21.15.1:0/445689222 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc070bd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.623+0000 7fcb00c96640 1 --2- 172.21.15.1:0/445689222 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 0x7fcafc156c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.623+0000 7fcb00c96640 1 -- 172.21.15.1:0/445689222 >> 172.21.15.1:0/445689222 conn(0x7fcafc0cc6a0 msgr2=0x7fcafc0ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.623+0000 7fcb00c96640 1 -- 172.21.15.1:0/445689222 shutdown_connections 2024-05-09T05:23:54.621 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.623+0000 7fcb00c96640 1 -- 172.21.15.1:0/445689222 wait complete. 2024-05-09T05:23:54.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.625+0000 7fcb00c96640 1 Processor -- start 2024-05-09T05:23:54.622 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.625+0000 7fcb00c96640 1 -- start start 2024-05-09T05:23:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcb00c96640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc06e1f0 0x7fcafc1f36f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcb00c96640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc0b4b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.624 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcb00c96640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 0x7fcafc0b3550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcb00c96640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc15d440 con 0x7fcafc06e600 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcb00c96640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fcafc15d160 con 0x7fcafc06e1f0 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcb00c96640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fcafc068d60 con 0x7fcafc0c64b0 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcaf9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc0b4b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcaf9d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc0b4b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:37488/0 (socket says 172.21.15.1:37488) 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.626+0000 7fcaf9d9b640 1 -- 172.21.15.1:0/2623466149 learned_addr learned my addr 172.21.15.1:0/2623466149 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcafad9d640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 0x7fcafc0b3550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcafa59c640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc06e1f0 0x7fcafc1f36f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf9d9b640 1 -- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 msgr2=0x7fcafc0b3550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf9d9b640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 0x7fcafc0b3550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf9d9b640 1 -- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc06e1f0 msgr2=0x7fcafc1f36f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf9d9b640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc06e1f0 0x7fcafc1f36f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf9d9b640 1 -- 172.21.15.1:0/2623466149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcafc0b3a90 con 0x7fcafc06e600 2024-05-09T05:23:54.625 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf9d9b640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc0b4b00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcae400c3a0 tx=0x7fcae400c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf2ffd640 1 -- 172.21.15.1:0/2623466149 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae40052d0 con 0x7fcafc06e600 2024-05-09T05:23:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf2ffd640 1 -- 172.21.15.1:0/2623466149 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7fcae4007020 con 0x7fcafc06e600 2024-05-09T05:23:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcafc0b4820 con 0x7fcafc06e600 2024-05-09T05:23:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcaf2ffd640 1 -- 172.21.15.1:0/2623466149 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcae40054b0 con 0x7fcafc06e600 2024-05-09T05:23:54.626 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.627+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcafc0b2d50 con 0x7fcafc06e600 2024-05-09T05:23:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.628+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcafc06f7b0 con 0x7fcafc06e600 2024-05-09T05:23:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.629+0000 7fcaf2ffd640 1 -- 172.21.15.1:0/2623466149 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fcae4007410 con 0x7fcafc06e600 2024-05-09T05:23:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.629+0000 7fcaf2ffd640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcadc0762d0 0x7fcadc0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:23:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.630+0000 7fcaf2ffd640 1 -- 172.21.15.1:0/2623466149 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fcae4096940 con 0x7fcafc06e600 2024-05-09T05:23:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.630+0000 7fcafa59c640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcadc0762d0 0x7fcadc0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:23:54.627 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.630+0000 7fcafa59c640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcadc0762d0 0x7fcadc0787e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcaec003aa0 tx=0x7fcaec009830 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:23:54.631 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.634+0000 7fcaf2ffd640 1 -- 172.21.15.1:0/2623466149 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fcafc06f7b0 con 0x7fcafc06e600 2024-05-09T05:23:54.821 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.824+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fcafc0b45d0 con 0x7fcadc0762d0 2024-05-09T05:23:54.827 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.831+0000 7fcaf2ffd640 1 -- 172.21.15.1:0/2623466149 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+252607 (secure 0 0 0) 0x7fcafc0b45d0 con 0x7fcadc0762d0 2024-05-09T05:23:54.830 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:23:54.831 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcadc0762d0 msgr2=0x7fcadc0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcadc0762d0 0x7fcadc0787e0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fcaec003aa0 tx=0x7fcaec009830 comp rx=0 tx=0).stop 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 msgr2=0x7fcafc0b4b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc0b4b00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fcae400c3a0 tx=0x7fcae400c860 comp rx=0 tx=0).stop 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 shutdown_connections 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fcadc0762d0 0x7fcadc0787e0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fcafc0c64b0 0x7fcafc0b3550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fcafc06e600 0x7fcafc0b4b00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.833 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 --2- 172.21.15.1:0/2623466149 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fcafc06e1f0 0x7fcafc1f36f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:23:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 >> 172.21.15.1:0/2623466149 conn(0x7fcafc0cc6a0 msgr2=0x7fcafc0ccab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:23:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 shutdown_connections 2024-05-09T05:23:54.834 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:23:54.834+0000 7fcb00c96640 1 -- 172.21.15.1:0/2623466149 wait complete. 2024-05-09T05:23:54.843 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2024-05-09T05:23:53.834991+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19180393,"num_objects":105,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":102,"num_read_kb":167,"num_write":1813207,"num_write_kb":20380719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366735,"ondisk_log_size":366735,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4716776,"kb_used_data":39528,"kb_used_omap":12,"kb_used_meta":4677171,"kb_avail":750257944,"statfs":{"total":773094113280,"available":768264134656,"internally_reserved":0,"allocated":40476672,"data_stored":39046538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4789423665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003645"},"pg_stats":[{"pgid":"3.3f","version":"23'5187","reported_seq":5211,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.889349+0000","last_change":"2024-05-09T05:06:05.069529+0000","last_active":"2024-05-09T05:23:41.889349+0000","last_peered":"2024-05-09T05:23:41.889349+0000","last_clean":"2024-05-09T05:23:41.889349+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:23:41.889349+0000","last_undegraded":"2024-05-09T05:23:41.889349+0000","last_fullsized":"2024-05-09T05:23:41.889349+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26135,"num_write_kb":280897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'6912","reported_seq":6937,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.798946+0000","last_change":"2024-05-09T05:06:05.058787+0000","last_active":"2024-05-09T05:23:41.798946+0000","last_peered":"2024-05-09T05:23:41.798946+0000","last_clean":"2024-05-09T05:23:41.798946+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:23:41.798946+0000","last_undegraded":"2024-05-09T05:23:41.798946+0000","last_fullsized":"2024-05-09T05:23:41.798946+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35479,"num_write_kb":400336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'5571","reported_seq":5596,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.786040+0000","last_change":"2024-05-09T05:06:05.054822+0000","last_active":"2024-05-09T05:23:41.786040+0000","last_peered":"2024-05-09T05:23:41.786040+0000","last_clean":"2024-05-09T05:23:41.786040+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:23:41.786040+0000","last_undegraded":"2024-05-09T05:23:41.786040+0000","last_fullsized":"2024-05-09T05:23:41.786040+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27199,"num_write_kb":317974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'5494","reported_seq":5518,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.804206+0000","last_change":"2024-05-09T05:06:05.066220+0000","last_active":"2024-05-09T05:23:41.804206+0000","last_peered":"2024-05-09T05:23:41.804206+0000","last_clean":"2024-05-09T05:23:41.804206+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:23:41.804206+0000","last_undegraded":"2024-05-09T05:23:41.804206+0000","last_fullsized":"2024-05-09T05:23:41.804206+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28054,"num_write_kb":296869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'7058","reported_seq":7081,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790501+0000","last_change":"2024-05-09T05:06:05.063485+0000","last_active":"2024-05-09T05:23:41.790501+0000","last_peered":"2024-05-09T05:23:41.790501+0000","last_clean":"2024-05-09T05:23:41.790501+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:23:41.790501+0000","last_undegraded":"2024-05-09T05:23:41.790501+0000","last_fullsized":"2024-05-09T05:23:41.790501+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:46:02.051230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34281,"num_write_kb":379027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'4948","reported_seq":4971,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.889531+0000","last_change":"2024-05-09T05:06:05.058244+0000","last_active":"2024-05-09T05:23:41.889531+0000","last_peered":"2024-05-09T05:23:41.889531+0000","last_clean":"2024-05-09T05:23:41.889531+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:23:41.889531+0000","last_undegraded":"2024-05-09T05:23:41.889531+0000","last_fullsized":"2024-05-09T05:23:41.889531+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:59:40.145705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24780,"num_write_kb":275405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'5454","reported_seq":5477,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.811426+0000","last_change":"2024-05-09T05:06:05.069292+0000","last_active":"2024-05-09T05:23:41.811426+0000","last_peered":"2024-05-09T05:23:41.811426+0000","last_clean":"2024-05-09T05:23:41.811426+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:23:41.811426+0000","last_undegraded":"2024-05-09T05:23:41.811426+0000","last_fullsized":"2024-05-09T05:23:41.811426+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27432,"num_write_kb":306787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'6053","reported_seq":6076,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.840232+0000","last_change":"2024-05-09T05:06:05.067625+0000","last_active":"2024-05-09T05:23:41.840232+0000","last_peered":"2024-05-09T05:23:41.840232+0000","last_clean":"2024-05-09T05:23:41.840232+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:23:41.840232+0000","last_undegraded":"2024-05-09T05:23:41.840232+0000","last_fullsized":"2024-05-09T05:23:41.840232+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6053,"log_dups_size":0,"ondisk_log_size":6053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29445,"num_write_kb":339212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"23'7129","reported_seq":7152,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.913522+0000","last_change":"2024-05-09T05:06:05.069263+0000","last_active":"2024-05-09T05:23:41.913522+0000","last_peered":"2024-05-09T05:23:41.913522+0000","last_clean":"2024-05-09T05:23:41.913522+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:23:41.913522+0000","last_undegraded":"2024-05-09T05:23:41.913522+0000","last_fullsized":"2024-05-09T05:23:41.913522+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:22:36.648496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36611,"num_write_kb":392539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'6358","reported_seq":6381,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.896558+0000","last_change":"2024-05-09T05:06:05.063164+0000","last_active":"2024-05-09T05:23:41.896558+0000","last_peered":"2024-05-09T05:23:41.896558+0000","last_clean":"2024-05-09T05:23:41.896558+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:23:41.896558+0000","last_undegraded":"2024-05-09T05:23:41.896558+0000","last_fullsized":"2024-05-09T05:23:41.896558+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32647,"num_write_kb":360281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'4747","reported_seq":4770,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.882769+0000","last_change":"2024-05-09T05:06:05.066065+0000","last_active":"2024-05-09T05:23:41.882769+0000","last_peered":"2024-05-09T05:23:41.882769+0000","last_clean":"2024-05-09T05:23:41.882769+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:23:41.882769+0000","last_undegraded":"2024-05-09T05:23:41.882769+0000","last_fullsized":"2024-05-09T05:23:41.882769+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22850,"num_write_kb":288147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'4814","reported_seq":4837,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.916891+0000","last_change":"2024-05-09T05:06:05.061403+0000","last_active":"2024-05-09T05:23:41.916891+0000","last_peered":"2024-05-09T05:23:41.916891+0000","last_clean":"2024-05-09T05:23:41.916891+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:23:41.916891+0000","last_undegraded":"2024-05-09T05:23:41.916891+0000","last_fullsized":"2024-05-09T05:23:41.916891+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":267591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'5105","reported_seq":5128,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.762396+0000","last_change":"2024-05-09T05:06:05.069386+0000","last_active":"2024-05-09T05:23:41.762396+0000","last_peered":"2024-05-09T05:23:41.762396+0000","last_clean":"2024-05-09T05:23:41.762396+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:23:41.762396+0000","last_undegraded":"2024-05-09T05:23:41.762396+0000","last_fullsized":"2024-05-09T05:23:41.762396+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25963,"num_write_kb":288363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'5957","reported_seq":5981,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.820132+0000","last_change":"2024-05-09T05:06:05.058131+0000","last_active":"2024-05-09T05:23:41.820132+0000","last_peered":"2024-05-09T05:23:41.820132+0000","last_clean":"2024-05-09T05:23:41.820132+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:23:41.820132+0000","last_undegraded":"2024-05-09T05:23:41.820132+0000","last_fullsized":"2024-05-09T05:23:41.820132+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29976,"num_write_kb":329157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'5739","reported_seq":5763,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.843588+0000","last_change":"2024-05-09T05:06:05.063208+0000","last_active":"2024-05-09T05:23:41.843588+0000","last_peered":"2024-05-09T05:23:41.843588+0000","last_clean":"2024-05-09T05:23:41.843588+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:23:41.843588+0000","last_undegraded":"2024-05-09T05:23:41.843588+0000","last_fullsized":"2024-05-09T05:23:41.843588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28226,"num_write_kb":322058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'5341","reported_seq":5364,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.852751+0000","last_change":"2024-05-09T05:06:05.053019+0000","last_active":"2024-05-09T05:23:41.852751+0000","last_peered":"2024-05-09T05:23:41.852751+0000","last_clean":"2024-05-09T05:23:41.852751+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:23:41.852751+0000","last_undegraded":"2024-05-09T05:23:41.852751+0000","last_fullsized":"2024-05-09T05:23:41.852751+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27307,"num_write_kb":289793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"23'6965","reported_seq":6982,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803206+0000","last_change":"2024-05-09T05:07:01.750820+0000","last_active":"2024-05-09T05:23:41.803206+0000","last_peered":"2024-05-09T05:23:41.803206+0000","last_clean":"2024-05-09T05:23:41.803206+0000","last_became_active":"2024-05-09T05:07:01.750492+0000","last_became_peered":"2024-05-09T05:07:01.750492+0000","last_unstale":"2024-05-09T05:23:41.803206+0000","last_undegraded":"2024-05-09T05:23:41.803206+0000","last_fullsized":"2024-05-09T05:23:41.803206+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6965,"log_dups_size":0,"ondisk_log_size":6965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:29:23.141520+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34877,"num_write_kb":385595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"23'4549","reported_seq":4572,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.791997+0000","last_change":"2024-05-09T05:06:05.065034+0000","last_active":"2024-05-09T05:23:41.791997+0000","last_peered":"2024-05-09T05:23:41.791997+0000","last_clean":"2024-05-09T05:23:41.791997+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:23:41.791997+0000","last_undegraded":"2024-05-09T05:23:41.791997+0000","last_fullsized":"2024-05-09T05:23:41.791997+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22029,"num_write_kb":252128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'5335","reported_seq":5358,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.781003+0000","last_change":"2024-05-09T05:06:05.065589+0000","last_active":"2024-05-09T05:23:41.781003+0000","last_peered":"2024-05-09T05:23:41.781003+0000","last_clean":"2024-05-09T05:23:41.781003+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:23:41.781003+0000","last_undegraded":"2024-05-09T05:23:41.781003+0000","last_fullsized":"2024-05-09T05:23:41.781003+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:35.608204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26106,"num_write_kb":309468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'5370","reported_seq":5393,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.861348+0000","last_change":"2024-05-09T05:06:05.061513+0000","last_active":"2024-05-09T05:23:41.861348+0000","last_peered":"2024-05-09T05:23:41.861348+0000","last_clean":"2024-05-09T05:23:41.861348+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:23:41.861348+0000","last_undegraded":"2024-05-09T05:23:41.861348+0000","last_fullsized":"2024-05-09T05:23:41.861348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:46:47.323095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26990,"num_write_kb":308937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'5403","reported_seq":5427,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.875126+0000","last_change":"2024-05-09T05:06:05.064506+0000","last_active":"2024-05-09T05:23:41.875126+0000","last_peered":"2024-05-09T05:23:41.875126+0000","last_clean":"2024-05-09T05:23:41.875126+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:23:41.875126+0000","last_undegraded":"2024-05-09T05:23:41.875126+0000","last_fullsized":"2024-05-09T05:23:41.875126+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26698,"num_write_kb":315887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'6082","reported_seq":6105,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.904062+0000","last_change":"2024-05-09T05:06:05.071748+0000","last_active":"2024-05-09T05:23:41.904062+0000","last_peered":"2024-05-09T05:23:41.904062+0000","last_clean":"2024-05-09T05:23:41.904062+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:23:41.904062+0000","last_undegraded":"2024-05-09T05:23:41.904062+0000","last_fullsized":"2024-05-09T05:23:41.904062+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:01:04.927783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30462,"num_write_kb":343332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"23'5512","reported_seq":5535,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.774669+0000","last_change":"2024-05-09T05:06:05.063694+0000","last_active":"2024-05-09T05:23:41.774669+0000","last_peered":"2024-05-09T05:23:41.774669+0000","last_clean":"2024-05-09T05:23:41.774669+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:23:41.774669+0000","last_undegraded":"2024-05-09T05:23:41.774669+0000","last_fullsized":"2024-05-09T05:23:41.774669+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:35:37.923954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27471,"num_write_kb":310767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'5065","reported_seq":5089,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.839542+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:23:41.839542+0000","last_peered":"2024-05-09T05:23:41.839542+0000","last_clean":"2024-05-09T05:23:41.839542+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:23:41.839542+0000","last_undegraded":"2024-05-09T05:23:41.839542+0000","last_fullsized":"2024-05-09T05:23:41.839542+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:26:25.084297+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24983,"num_write_kb":280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'6632","reported_seq":6656,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.735021+0000","last_change":"2024-05-09T05:06:05.063479+0000","last_active":"2024-05-09T05:23:41.735021+0000","last_peered":"2024-05-09T05:23:41.735021+0000","last_clean":"2024-05-09T05:23:41.735021+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:23:41.735021+0000","last_undegraded":"2024-05-09T05:23:41.735021+0000","last_fullsized":"2024-05-09T05:23:41.735021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:25:28.166958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33114,"num_write_kb":370371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'5121","reported_seq":5150,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.872969+0000","last_change":"2024-05-09T05:07:01.662425+0000","last_active":"2024-05-09T05:23:41.872969+0000","last_peered":"2024-05-09T05:23:41.872969+0000","last_clean":"2024-05-09T05:23:41.872969+0000","last_became_active":"2024-05-09T05:07:01.662096+0000","last_became_peered":"2024-05-09T05:07:01.662096+0000","last_unstale":"2024-05-09T05:23:41.872969+0000","last_undegraded":"2024-05-09T05:23:41.872969+0000","last_fullsized":"2024-05-09T05:23:41.872969+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:29:50.949207+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25567,"num_write_kb":287181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'6014","reported_seq":6039,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.784763+0000","last_change":"2024-05-09T05:06:05.061538+0000","last_active":"2024-05-09T05:23:41.784763+0000","last_peered":"2024-05-09T05:23:41.784763+0000","last_clean":"2024-05-09T05:23:41.784763+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:23:41.784763+0000","last_undegraded":"2024-05-09T05:23:41.784763+0000","last_fullsized":"2024-05-09T05:23:41.784763+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30657,"num_write_kb":340145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'5132","reported_seq":5150,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.884549+0000","last_change":"2024-05-09T05:07:01.723062+0000","last_active":"2024-05-09T05:23:41.884549+0000","last_peered":"2024-05-09T05:23:41.884549+0000","last_clean":"2024-05-09T05:23:41.884549+0000","last_became_active":"2024-05-09T05:07:01.722736+0000","last_became_peered":"2024-05-09T05:07:01.722736+0000","last_unstale":"2024-05-09T05:23:41.884549+0000","last_undegraded":"2024-05-09T05:23:41.884549+0000","last_fullsized":"2024-05-09T05:23:41.884549+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25502,"num_write_kb":286500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"23'4664","reported_seq":4689,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.846783+0000","last_change":"2024-05-09T05:06:05.066437+0000","last_active":"2024-05-09T05:23:41.846783+0000","last_peered":"2024-05-09T05:23:41.846783+0000","last_clean":"2024-05-09T05:23:41.846783+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:23:41.846783+0000","last_undegraded":"2024-05-09T05:23:41.846783+0000","last_fullsized":"2024-05-09T05:23:41.846783+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23461,"num_write_kb":270138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'6545","reported_seq":6569,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.857347+0000","last_change":"2024-05-09T05:06:05.066822+0000","last_active":"2024-05-09T05:23:41.857347+0000","last_peered":"2024-05-09T05:23:41.857347+0000","last_clean":"2024-05-09T05:23:41.857347+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:23:41.857347+0000","last_undegraded":"2024-05-09T05:23:41.857347+0000","last_fullsized":"2024-05-09T05:23:41.857347+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6545,"log_dups_size":0,"ondisk_log_size":6545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32981,"num_write_kb":377918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'6387","reported_seq":6411,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.809273+0000","last_change":"2024-05-09T05:06:05.063094+0000","last_active":"2024-05-09T05:23:41.809273+0000","last_peered":"2024-05-09T05:23:41.809273+0000","last_clean":"2024-05-09T05:23:41.809273+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:23:41.809273+0000","last_undegraded":"2024-05-09T05:23:41.809273+0000","last_fullsized":"2024-05-09T05:23:41.809273+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:10:09.511767+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32133,"num_write_kb":346002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'7361","reported_seq":7385,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.851736+0000","last_change":"2024-05-09T05:06:05.062264+0000","last_active":"2024-05-09T05:23:41.851736+0000","last_peered":"2024-05-09T05:23:41.851736+0000","last_clean":"2024-05-09T05:23:41.851736+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:23:41.851736+0000","last_undegraded":"2024-05-09T05:23:41.851736+0000","last_fullsized":"2024-05-09T05:23:41.851736+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36142,"num_write_kb":398870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'5228","reported_seq":5251,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803020+0000","last_change":"2024-05-09T05:06:05.060715+0000","last_active":"2024-05-09T05:23:41.803020+0000","last_peered":"2024-05-09T05:23:41.803020+0000","last_clean":"2024-05-09T05:23:41.803020+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:23:41.803020+0000","last_undegraded":"2024-05-09T05:23:41.803020+0000","last_fullsized":"2024-05-09T05:23:41.803020+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:45.171073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25537,"num_write_kb":278419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'577","reported_seq":602,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:36.679365+0000","last_change":"2024-05-09T05:06:04.069645+0000","last_active":"2024-05-09T05:23:36.679365+0000","last_peered":"2024-05-09T05:23:36.679365+0000","last_clean":"2024-05-09T05:23:36.679365+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:23:36.679365+0000","last_undegraded":"2024-05-09T05:23:36.679365+0000","last_fullsized":"2024-05-09T05:23:36.679365+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":577,"log_dups_size":0,"ondisk_log_size":577,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":573,"num_write_kb":4218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'4521","reported_seq":4545,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.926254+0000","last_change":"2024-05-09T05:06:05.061157+0000","last_active":"2024-05-09T05:23:41.926254+0000","last_peered":"2024-05-09T05:23:41.926254+0000","last_clean":"2024-05-09T05:23:41.926254+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:23:41.926254+0000","last_undegraded":"2024-05-09T05:23:41.926254+0000","last_fullsized":"2024-05-09T05:23:41.926254+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22336,"num_write_kb":249223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'111","reported_seq":136,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.634306+0000","last_change":"2024-05-09T05:06:04.059337+0000","last_active":"2024-05-09T05:23:41.634306+0000","last_peered":"2024-05-09T05:23:41.634306+0000","last_clean":"2024-05-09T05:23:41.634306+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:23:41.634306+0000","last_undegraded":"2024-05-09T05:23:41.634306+0000","last_fullsized":"2024-05-09T05:23:41.634306+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":111,"log_dups_size":0,"ondisk_log_size":111,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1715788,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":1741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.829290+0000","last_change":"2024-05-09T05:06:04.058497+0000","last_active":"2024-05-09T05:22:54.829290+0000","last_peered":"2024-05-09T05:22:54.829290+0000","last_clean":"2024-05-09T05:22:54.829290+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:22:54.829290+0000","last_undegraded":"2024-05-09T05:22:54.829290+0000","last_fullsized":"2024-05-09T05:22:54.829290+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:31:18.314155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'5503","reported_seq":5526,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.842314+0000","last_change":"2024-05-09T05:06:05.071008+0000","last_active":"2024-05-09T05:23:41.842314+0000","last_peered":"2024-05-09T05:23:41.842314+0000","last_clean":"2024-05-09T05:23:41.842314+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:23:41.842314+0000","last_undegraded":"2024-05-09T05:23:41.842314+0000","last_fullsized":"2024-05-09T05:23:41.842314+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27748,"num_write_kb":311965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.156887+0000","last_change":"2024-05-09T05:06:04.056540+0000","last_active":"2024-05-09T05:22:59.156887+0000","last_peered":"2024-05-09T05:22:59.156887+0000","last_clean":"2024-05-09T05:22:59.156887+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:22:59.156887+0000","last_undegraded":"2024-05-09T05:22:59.156887+0000","last_fullsized":"2024-05-09T05:22:59.156887+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:13:24.572375+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'6912","reported_seq":6936,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.824242+0000","last_change":"2024-05-09T05:06:05.066100+0000","last_active":"2024-05-09T05:23:41.824242+0000","last_peered":"2024-05-09T05:23:41.824242+0000","last_clean":"2024-05-09T05:23:41.824242+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:23:41.824242+0000","last_undegraded":"2024-05-09T05:23:41.824242+0000","last_fullsized":"2024-05-09T05:23:41.824242+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34348,"num_write_kb":375438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'9","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.705733+0000","last_change":"2024-05-09T05:06:04.061495+0000","last_active":"2024-05-09T05:23:41.705733+0000","last_peered":"2024-05-09T05:23:41.705733+0000","last_clean":"2024-05-09T05:23:41.705733+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:23:41.705733+0000","last_undegraded":"2024-05-09T05:23:41.705733+0000","last_fullsized":"2024-05-09T05:23:41.705733+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'5546","reported_seq":5569,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.742360+0000","last_change":"2024-05-09T05:06:05.071334+0000","last_active":"2024-05-09T05:23:41.742360+0000","last_peered":"2024-05-09T05:23:41.742360+0000","last_clean":"2024-05-09T05:23:41.742360+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:23:41.742360+0000","last_undegraded":"2024-05-09T05:23:41.742360+0000","last_fullsized":"2024-05-09T05:23:41.742360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5546,"log_dups_size":0,"ondisk_log_size":5546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:41:01.468195+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27252,"num_write_kb":314086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.148513+0000","last_change":"2024-05-09T05:06:04.055964+0000","last_active":"2024-05-09T05:22:59.148513+0000","last_peered":"2024-05-09T05:22:59.148513+0000","last_clean":"2024-05-09T05:22:59.148513+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:22:59.148513+0000","last_undegraded":"2024-05-09T05:22:59.148513+0000","last_fullsized":"2024-05-09T05:22:59.148513+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:37:38.978681+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'5049","reported_seq":5073,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.871941+0000","last_change":"2024-05-09T05:06:05.059170+0000","last_active":"2024-05-09T05:23:41.871941+0000","last_peered":"2024-05-09T05:23:41.871941+0000","last_clean":"2024-05-09T05:23:41.871941+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:23:41.871941+0000","last_undegraded":"2024-05-09T05:23:41.871941+0000","last_fullsized":"2024-05-09T05:23:41.871941+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25089,"num_write_kb":276787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'25","reported_seq":51,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:01.656821+0000","last_change":"2024-05-09T05:06:04.060131+0000","last_active":"2024-05-09T05:23:01.656821+0000","last_peered":"2024-05-09T05:23:01.656821+0000","last_clean":"2024-05-09T05:23:01.656821+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:23:01.656821+0000","last_undegraded":"2024-05-09T05:23:01.656821+0000","last_fullsized":"2024-05-09T05:23:01.656821+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":25,"log_dups_size":0,"ondisk_log_size":25,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'7381","reported_seq":7404,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.917166+0000","last_change":"2024-05-09T05:06:05.067723+0000","last_active":"2024-05-09T05:23:41.917166+0000","last_peered":"2024-05-09T05:23:41.917166+0000","last_clean":"2024-05-09T05:23:41.917166+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:23:41.917166+0000","last_undegraded":"2024-05-09T05:23:41.917166+0000","last_fullsized":"2024-05-09T05:23:41.917166+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":7381,"log_dups_size":0,"ondisk_log_size":7381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:10.665385+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36744,"num_write_kb":418024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'333","reported_seq":358,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.155427+0000","last_change":"2024-05-09T05:06:04.059075+0000","last_active":"2024-05-09T05:22:59.155427+0000","last_peered":"2024-05-09T05:22:59.155427+0000","last_clean":"2024-05-09T05:22:59.155427+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:22:59.155427+0000","last_undegraded":"2024-05-09T05:22:59.155427+0000","last_fullsized":"2024-05-09T05:22:59.155427+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":333,"log_dups_size":0,"ondisk_log_size":333,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":331,"num_write_kb":4250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'5538","reported_seq":5561,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.787753+0000","last_change":"2024-05-09T05:06:05.065165+0000","last_active":"2024-05-09T05:23:41.787753+0000","last_peered":"2024-05-09T05:23:41.787753+0000","last_clean":"2024-05-09T05:23:41.787753+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:23:41.787753+0000","last_undegraded":"2024-05-09T05:23:41.787753+0000","last_fullsized":"2024-05-09T05:23:41.787753+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:25.402099+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27557,"num_write_kb":302974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"23'15","reported_seq":41,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.143384+0000","last_change":"2024-05-09T05:06:04.057612+0000","last_active":"2024-05-09T05:22:59.143384+0000","last_peered":"2024-05-09T05:22:59.143384+0000","last_clean":"2024-05-09T05:22:59.143384+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:22:59.143384+0000","last_undegraded":"2024-05-09T05:22:59.143384+0000","last_fullsized":"2024-05-09T05:22:59.143384+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'5579","reported_seq":5603,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.842718+0000","last_change":"2024-05-09T05:06:05.069199+0000","last_active":"2024-05-09T05:23:41.842718+0000","last_peered":"2024-05-09T05:23:41.842718+0000","last_clean":"2024-05-09T05:23:41.842718+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:23:41.842718+0000","last_undegraded":"2024-05-09T05:23:41.842718+0000","last_fullsized":"2024-05-09T05:23:41.842718+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27823,"num_write_kb":303973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'10","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.117627+0000","last_change":"2024-05-09T05:06:04.062200+0000","last_active":"2024-05-09T05:22:59.117627+0000","last_peered":"2024-05-09T05:22:59.117627+0000","last_clean":"2024-05-09T05:22:59.117627+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:22:59.117627+0000","last_undegraded":"2024-05-09T05:22:59.117627+0000","last_fullsized":"2024-05-09T05:22:59.117627+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:54.707519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'6303","reported_seq":6326,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.912348+0000","last_change":"2024-05-09T05:06:05.066931+0000","last_active":"2024-05-09T05:23:41.912348+0000","last_peered":"2024-05-09T05:23:41.912348+0000","last_clean":"2024-05-09T05:23:41.912348+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:23:41.912348+0000","last_undegraded":"2024-05-09T05:23:41.912348+0000","last_fullsized":"2024-05-09T05:23:41.912348+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30554,"num_write_kb":351862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'13","reported_seq":38,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.160623+0000","last_change":"2024-05-09T05:06:04.061374+0000","last_active":"2024-05-09T05:22:59.160623+0000","last_peered":"2024-05-09T05:22:59.160623+0000","last_clean":"2024-05-09T05:22:59.160623+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:22:59.160623+0000","last_undegraded":"2024-05-09T05:22:59.160623+0000","last_fullsized":"2024-05-09T05:22:59.160623+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'5877","reported_seq":5900,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.800435+0000","last_change":"2024-05-09T05:06:05.061968+0000","last_active":"2024-05-09T05:23:41.800435+0000","last_peered":"2024-05-09T05:23:41.800435+0000","last_clean":"2024-05-09T05:23:41.800435+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:23:41.800435+0000","last_undegraded":"2024-05-09T05:23:41.800435+0000","last_fullsized":"2024-05-09T05:23:41.800435+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:09:16.808904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29217,"num_write_kb":334206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"23'21","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.141090+0000","last_change":"2024-05-09T05:06:04.050997+0000","last_active":"2024-05-09T05:22:59.141090+0000","last_peered":"2024-05-09T05:22:59.141090+0000","last_clean":"2024-05-09T05:22:59.141090+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:22:59.141090+0000","last_undegraded":"2024-05-09T05:22:59.141090+0000","last_fullsized":"2024-05-09T05:22:59.141090+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:49:33.974616+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'5443","reported_seq":5466,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.843005+0000","last_change":"2024-05-09T05:06:05.068997+0000","last_active":"2024-05-09T05:23:41.843005+0000","last_peered":"2024-05-09T05:23:41.843005+0000","last_clean":"2024-05-09T05:23:41.843005+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:23:41.843005+0000","last_undegraded":"2024-05-09T05:23:41.843005+0000","last_fullsized":"2024-05-09T05:23:41.843005+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:57:38.257962+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27739,"num_write_kb":311936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.151151+0000","last_change":"2024-05-09T05:06:04.060896+0000","last_active":"2024-05-09T05:22:59.151151+0000","last_peered":"2024-05-09T05:22:59.151151+0000","last_clean":"2024-05-09T05:22:59.151151+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:22:59.151151+0000","last_undegraded":"2024-05-09T05:22:59.151151+0000","last_fullsized":"2024-05-09T05:22:59.151151+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:03:25.116011+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'5295","reported_seq":5318,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.754503+0000","last_change":"2024-05-09T05:06:05.064182+0000","last_active":"2024-05-09T05:23:41.754503+0000","last_peered":"2024-05-09T05:23:41.754503+0000","last_clean":"2024-05-09T05:23:41.754503+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:23:41.754503+0000","last_undegraded":"2024-05-09T05:23:41.754503+0000","last_fullsized":"2024-05-09T05:23:41.754503+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25732,"num_write_kb":299773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'9","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.132085+0000","last_change":"2024-05-09T05:06:04.060274+0000","last_active":"2024-05-09T05:22:59.132085+0000","last_peered":"2024-05-09T05:22:59.132085+0000","last_clean":"2024-05-09T05:22:59.132085+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:22:59.132085+0000","last_undegraded":"2024-05-09T05:22:59.132085+0000","last_fullsized":"2024-05-09T05:22:59.132085+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:23:44.165241+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'6402","reported_seq":6425,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.913813+0000","last_change":"2024-05-09T05:06:05.070324+0000","last_active":"2024-05-09T05:23:41.913813+0000","last_peered":"2024-05-09T05:23:41.913813+0000","last_clean":"2024-05-09T05:23:41.913813+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:23:41.913813+0000","last_undegraded":"2024-05-09T05:23:41.913813+0000","last_fullsized":"2024-05-09T05:23:41.913813+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:50:07.235982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31156,"num_write_kb":362485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"23'4","reported_seq":30,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.833604+0000","last_change":"2024-05-09T05:06:04.056972+0000","last_active":"2024-05-09T05:22:54.833604+0000","last_peered":"2024-05-09T05:22:54.833604+0000","last_clean":"2024-05-09T05:22:54.833604+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:22:54.833604+0000","last_undegraded":"2024-05-09T05:22:54.833604+0000","last_fullsized":"2024-05-09T05:22:54.833604+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'5559","reported_seq":5582,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.794865+0000","last_change":"2024-05-09T05:06:05.066101+0000","last_active":"2024-05-09T05:23:41.794865+0000","last_peered":"2024-05-09T05:23:41.794865+0000","last_clean":"2024-05-09T05:23:41.794865+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:23:41.794865+0000","last_undegraded":"2024-05-09T05:23:41.794865+0000","last_fullsized":"2024-05-09T05:23:41.794865+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:38:09.842221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28130,"num_write_kb":325652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'10","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.117825+0000","last_change":"2024-05-09T05:06:04.055325+0000","last_active":"2024-05-09T05:22:59.117825+0000","last_peered":"2024-05-09T05:22:59.117825+0000","last_clean":"2024-05-09T05:22:59.117825+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:22:59.117825+0000","last_undegraded":"2024-05-09T05:22:59.117825+0000","last_fullsized":"2024-05-09T05:22:59.117825+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:51:07.979398+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'5794","reported_seq":5817,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.896072+0000","last_change":"2024-05-09T05:06:05.070923+0000","last_active":"2024-05-09T05:23:41.896072+0000","last_peered":"2024-05-09T05:23:41.896072+0000","last_clean":"2024-05-09T05:23:41.896072+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:23:41.896072+0000","last_undegraded":"2024-05-09T05:23:41.896072+0000","last_fullsized":"2024-05-09T05:23:41.896072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:48:22.061237+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28847,"num_write_kb":320571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"17'72","reported_seq":122,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:19:52.665112+0000","last_change":"2024-05-09T05:05:53.330649+0000","last_active":"2024-05-09T05:19:52.665112+0000","last_peered":"2024-05-09T05:19:52.665112+0000","last_clean":"2024-05-09T05:19:52.665112+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:19:52.665112+0000","last_undegraded":"2024-05-09T05:19:52.665112+0000","last_fullsized":"2024-05-09T05:19:52.665112+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:05:51.871680+0000","last_clean_scrub_stamp":"2024-05-09T05:05:51.871680+0000","objects_scrubbed":0,"log_size":72,"log_dups_size":0,"ondisk_log_size":72,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:04:18.435742+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'7","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:20:42.700037+0000","last_change":"2024-05-09T05:06:04.054056+0000","last_active":"2024-05-09T05:20:42.700037+0000","last_peered":"2024-05-09T05:20:42.700037+0000","last_clean":"2024-05-09T05:20:42.700037+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:20:42.700037+0000","last_undegraded":"2024-05-09T05:20:42.700037+0000","last_fullsized":"2024-05-09T05:20:42.700037+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:39:01.238801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'5510","reported_seq":5534,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.818585+0000","last_change":"2024-05-09T05:06:05.065962+0000","last_active":"2024-05-09T05:23:41.818585+0000","last_peered":"2024-05-09T05:23:41.818585+0000","last_clean":"2024-05-09T05:23:41.818585+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:23:41.818585+0000","last_undegraded":"2024-05-09T05:23:41.818585+0000","last_fullsized":"2024-05-09T05:23:41.818585+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5510,"log_dups_size":0,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:41:15.646283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27376,"num_write_kb":322728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'9","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.151152+0000","last_change":"2024-05-09T05:06:04.068541+0000","last_active":"2024-05-09T05:22:59.151152+0000","last_peered":"2024-05-09T05:22:59.151152+0000","last_clean":"2024-05-09T05:22:59.151152+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:22:59.151152+0000","last_undegraded":"2024-05-09T05:22:59.151152+0000","last_fullsized":"2024-05-09T05:22:59.151152+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:41:34.111095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'5383","reported_seq":5408,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.827268+0000","last_change":"2024-05-09T05:06:05.061570+0000","last_active":"2024-05-09T05:23:41.827268+0000","last_peered":"2024-05-09T05:23:41.827268+0000","last_clean":"2024-05-09T05:23:41.827268+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:23:41.827268+0000","last_undegraded":"2024-05-09T05:23:41.827268+0000","last_fullsized":"2024-05-09T05:23:41.827268+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27075,"num_write_kb":305200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'45","reported_seq":107,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.161683+0000","last_change":"2024-05-09T05:06:04.059028+0000","last_active":"2024-05-09T05:22:59.161683+0000","last_peered":"2024-05-09T05:22:59.161683+0000","last_clean":"2024-05-09T05:22:59.161683+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:22:59.161683+0000","last_undegraded":"2024-05-09T05:22:59.161683+0000","last_fullsized":"2024-05-09T05:22:59.161683+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:08:29.298393+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88293,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":43,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'5441","reported_seq":5466,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.792420+0000","last_change":"2024-05-09T05:06:05.056055+0000","last_active":"2024-05-09T05:23:41.792420+0000","last_peered":"2024-05-09T05:23:41.792420+0000","last_clean":"2024-05-09T05:23:41.792420+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:23:41.792420+0000","last_undegraded":"2024-05-09T05:23:41.792420+0000","last_fullsized":"2024-05-09T05:23:41.792420+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27385,"num_write_kb":302253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.133458+0000","last_change":"2024-05-09T05:06:04.057195+0000","last_active":"2024-05-09T05:22:59.133458+0000","last_peered":"2024-05-09T05:22:59.133458+0000","last_clean":"2024-05-09T05:22:59.133458+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:22:59.133458+0000","last_undegraded":"2024-05-09T05:22:59.133458+0000","last_fullsized":"2024-05-09T05:22:59.133458+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'5472","reported_seq":5496,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.928867+0000","last_change":"2024-05-09T05:06:05.059490+0000","last_active":"2024-05-09T05:23:41.928867+0000","last_peered":"2024-05-09T05:23:41.928867+0000","last_clean":"2024-05-09T05:23:41.928867+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:23:41.928867+0000","last_undegraded":"2024-05-09T05:23:41.928867+0000","last_fullsized":"2024-05-09T05:23:41.928867+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:53.822814+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28049,"num_write_kb":304974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'6","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.144747+0000","last_change":"2024-05-09T05:06:04.052387+0000","last_active":"2024-05-09T05:22:59.144747+0000","last_peered":"2024-05-09T05:22:59.144747+0000","last_clean":"2024-05-09T05:22:59.144747+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:22:59.144747+0000","last_undegraded":"2024-05-09T05:22:59.144747+0000","last_fullsized":"2024-05-09T05:22:59.144747+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'4678","reported_seq":4702,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.919111+0000","last_change":"2024-05-09T05:06:05.064758+0000","last_active":"2024-05-09T05:23:41.919111+0000","last_peered":"2024-05-09T05:23:41.919111+0000","last_clean":"2024-05-09T05:23:41.919111+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:23:41.919111+0000","last_undegraded":"2024-05-09T05:23:41.919111+0000","last_fullsized":"2024-05-09T05:23:41.919111+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:17:09.025623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23621,"num_write_kb":274908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"23'8","reported_seq":34,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:54.841082+0000","last_change":"2024-05-09T05:06:04.068523+0000","last_active":"2024-05-09T05:22:54.841082+0000","last_peered":"2024-05-09T05:22:54.841082+0000","last_clean":"2024-05-09T05:22:54.841082+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:22:54.841082+0000","last_undegraded":"2024-05-09T05:22:54.841082+0000","last_fullsized":"2024-05-09T05:22:54.841082+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:31:51.898283+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'5941","reported_seq":5966,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.801021+0000","last_change":"2024-05-09T05:06:05.055320+0000","last_active":"2024-05-09T05:23:41.801021+0000","last_peered":"2024-05-09T05:23:41.801021+0000","last_clean":"2024-05-09T05:23:41.801021+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:23:41.801021+0000","last_undegraded":"2024-05-09T05:23:41.801021+0000","last_fullsized":"2024-05-09T05:23:41.801021+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":328170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'609","reported_seq":636,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:26.659629+0000","last_change":"2024-05-09T05:06:04.059720+0000","last_active":"2024-05-09T05:23:26.659629+0000","last_peered":"2024-05-09T05:23:26.659629+0000","last_clean":"2024-05-09T05:23:26.659629+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:23:26.659629+0000","last_undegraded":"2024-05-09T05:23:26.659629+0000","last_fullsized":"2024-05-09T05:23:26.659629+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":609,"log_dups_size":0,"ondisk_log_size":609,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'5503","reported_seq":5527,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.773405+0000","last_change":"2024-05-09T05:06:05.072079+0000","last_active":"2024-05-09T05:23:41.773405+0000","last_peered":"2024-05-09T05:23:41.773405+0000","last_clean":"2024-05-09T05:23:41.773405+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:23:41.773405+0000","last_undegraded":"2024-05-09T05:23:41.773405+0000","last_fullsized":"2024-05-09T05:23:41.773405+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:24:33.840344+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26876,"num_write_kb":308483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'10","reported_seq":36,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.130335+0000","last_change":"2024-05-09T05:06:04.066061+0000","last_active":"2024-05-09T05:22:59.130335+0000","last_peered":"2024-05-09T05:22:59.130335+0000","last_clean":"2024-05-09T05:22:59.130335+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:22:59.130335+0000","last_undegraded":"2024-05-09T05:22:59.130335+0000","last_fullsized":"2024-05-09T05:22:59.130335+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:48.534280+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'4642","reported_seq":4667,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.788049+0000","last_change":"2024-05-09T05:06:05.056442+0000","last_active":"2024-05-09T05:23:41.788049+0000","last_peered":"2024-05-09T05:23:41.788049+0000","last_clean":"2024-05-09T05:23:41.788049+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:23:41.788049+0000","last_undegraded":"2024-05-09T05:23:41.788049+0000","last_fullsized":"2024-05-09T05:23:41.788049+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22705,"num_write_kb":262889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'5","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.155406+0000","last_change":"2024-05-09T05:06:04.069188+0000","last_active":"2024-05-09T05:22:59.155406+0000","last_peered":"2024-05-09T05:22:59.155406+0000","last_clean":"2024-05-09T05:22:59.155406+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:22:59.155406+0000","last_undegraded":"2024-05-09T05:22:59.155406+0000","last_fullsized":"2024-05-09T05:22:59.155406+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:49:26.301743+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'5215","reported_seq":5239,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790504+0000","last_change":"2024-05-09T05:06:05.060880+0000","last_active":"2024-05-09T05:23:41.790504+0000","last_peered":"2024-05-09T05:23:41.790504+0000","last_clean":"2024-05-09T05:23:41.790504+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:23:41.790504+0000","last_undegraded":"2024-05-09T05:23:41.790504+0000","last_fullsized":"2024-05-09T05:23:41.790504+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:23:49.540111+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25185,"num_write_kb":293431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"23'10","reported_seq":37,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:26.659577+0000","last_change":"2024-05-09T05:06:04.065739+0000","last_active":"2024-05-09T05:23:26.659577+0000","last_peered":"2024-05-09T05:23:26.659577+0000","last_clean":"2024-05-09T05:23:26.659577+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:23:26.659577+0000","last_undegraded":"2024-05-09T05:23:26.659577+0000","last_fullsized":"2024-05-09T05:23:26.659577+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'5538","reported_seq":5562,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.803051+0000","last_change":"2024-05-09T05:06:05.065970+0000","last_active":"2024-05-09T05:23:41.803051+0000","last_peered":"2024-05-09T05:23:41.803051+0000","last_clean":"2024-05-09T05:23:41.803051+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:23:41.803051+0000","last_undegraded":"2024-05-09T05:23:41.803051+0000","last_fullsized":"2024-05-09T05:23:41.803051+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:11:59.685768+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27899,"num_write_kb":316727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"23'6","reported_seq":33,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:31.660184+0000","last_change":"2024-05-09T05:06:04.060028+0000","last_active":"2024-05-09T05:23:31.660184+0000","last_peered":"2024-05-09T05:23:31.660184+0000","last_clean":"2024-05-09T05:23:31.660184+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:23:31.660184+0000","last_undegraded":"2024-05-09T05:23:31.660184+0000","last_fullsized":"2024-05-09T05:23:31.660184+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'5078","reported_seq":5101,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.813122+0000","last_change":"2024-05-09T05:06:05.072332+0000","last_active":"2024-05-09T05:23:41.813122+0000","last_peered":"2024-05-09T05:23:41.813122+0000","last_clean":"2024-05-09T05:23:41.813122+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:23:41.813122+0000","last_undegraded":"2024-05-09T05:23:41.813122+0000","last_fullsized":"2024-05-09T05:23:41.813122+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:58:35.651549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24494,"num_write_kb":270229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1097","reported_seq":1122,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.133596+0000","last_change":"2024-05-09T05:06:04.068565+0000","last_active":"2024-05-09T05:22:59.133596+0000","last_peered":"2024-05-09T05:22:59.133596+0000","last_clean":"2024-05-09T05:22:59.133596+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:22:59.133596+0000","last_undegraded":"2024-05-09T05:22:59.133596+0000","last_fullsized":"2024-05-09T05:22:59.133596+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":1097,"log_dups_size":0,"ondisk_log_size":1097,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:26:56.758571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1089,"num_write_kb":4419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'4956","reported_seq":4980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.764163+0000","last_change":"2024-05-09T05:06:05.063184+0000","last_active":"2024-05-09T05:23:41.764163+0000","last_peered":"2024-05-09T05:23:41.764163+0000","last_clean":"2024-05-09T05:23:41.764163+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:23:41.764163+0000","last_undegraded":"2024-05-09T05:23:41.764163+0000","last_fullsized":"2024-05-09T05:23:41.764163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:13:03.053692+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24725,"num_write_kb":283470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'6745","reported_seq":6768,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.926010+0000","last_change":"2024-05-09T05:06:05.066127+0000","last_active":"2024-05-09T05:23:41.926010+0000","last_peered":"2024-05-09T05:23:41.926010+0000","last_clean":"2024-05-09T05:23:41.926010+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:23:41.926010+0000","last_undegraded":"2024-05-09T05:23:41.926010+0000","last_fullsized":"2024-05-09T05:23:41.926010+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:47:51.966141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33228,"num_write_kb":379009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"23'6","reported_seq":31,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.163507+0000","last_change":"2024-05-09T05:06:04.070200+0000","last_active":"2024-05-09T05:22:59.163507+0000","last_peered":"2024-05-09T05:22:59.163507+0000","last_clean":"2024-05-09T05:22:59.163507+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:22:59.163507+0000","last_undegraded":"2024-05-09T05:22:59.163507+0000","last_fullsized":"2024-05-09T05:22:59.163507+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:10.565220+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'4185","reported_seq":4201,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.772240+0000","last_change":"2024-05-09T05:07:01.723135+0000","last_active":"2024-05-09T05:23:41.772240+0000","last_peered":"2024-05-09T05:23:41.772240+0000","last_clean":"2024-05-09T05:23:41.772240+0000","last_became_active":"2024-05-09T05:07:01.722785+0000","last_became_peered":"2024-05-09T05:07:01.722785+0000","last_unstale":"2024-05-09T05:23:41.772240+0000","last_undegraded":"2024-05-09T05:23:41.772240+0000","last_fullsized":"2024-05-09T05:23:41.772240+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:57:53.799866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20471,"num_write_kb":229445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'21","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.166888+0000","last_change":"2024-05-09T05:06:04.070043+0000","last_active":"2024-05-09T05:22:59.166888+0000","last_peered":"2024-05-09T05:22:59.166888+0000","last_clean":"2024-05-09T05:22:59.166888+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:22:59.166888+0000","last_undegraded":"2024-05-09T05:22:59.166888+0000","last_fullsized":"2024-05-09T05:22:59.166888+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:18:39.807845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'6872","reported_seq":6897,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.883360+0000","last_change":"2024-05-09T05:06:05.059365+0000","last_active":"2024-05-09T05:23:41.883360+0000","last_peered":"2024-05-09T05:23:41.883360+0000","last_clean":"2024-05-09T05:23:41.883360+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:23:41.883360+0000","last_undegraded":"2024-05-09T05:23:41.883360+0000","last_fullsized":"2024-05-09T05:23:41.883360+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34844,"num_write_kb":385600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'9","reported_seq":35,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:59.127515+0000","last_change":"2024-05-09T05:06:04.061224+0000","last_active":"2024-05-09T05:22:59.127515+0000","last_peered":"2024-05-09T05:22:59.127515+0000","last_clean":"2024-05-09T05:22:59.127515+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:22:59.127515+0000","last_undegraded":"2024-05-09T05:22:59.127515+0000","last_fullsized":"2024-05-09T05:22:59.127515+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:30:05.997451+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'5811","reported_seq":5835,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:41.790466+0000","last_change":"2024-05-09T05:06:05.069572+0000","last_active":"2024-05-09T05:23:41.790466+0000","last_peered":"2024-05-09T05:23:41.790466+0000","last_clean":"2024-05-09T05:23:41.790466+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:23:41.790466+0000","last_undegraded":"2024-05-09T05:23:41.790466+0000","last_fullsized":"2024-05-09T05:23:41.790466+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:04.034274+0000","last_clean_scrub_stamp":"2024-05-09T05:06:04.034274+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:32:29.331565+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29692,"num_write_kb":336226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'740","reported_seq":767,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:22:46.774682+0000","last_change":"2024-05-09T05:06:04.060954+0000","last_active":"2024-05-09T05:22:46.774682+0000","last_peered":"2024-05-09T05:22:46.774682+0000","last_clean":"2024-05-09T05:22:46.774682+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:22:46.774682+0000","last_undegraded":"2024-05-09T05:22:46.774682+0000","last_fullsized":"2024-05-09T05:22:46.774682+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T05:06:03.028443+0000","last_clean_scrub_stamp":"2024-05-09T05:06:03.028443+0000","objects_scrubbed":0,"log_size":740,"log_dups_size":0,"ondisk_log_size":740,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"queued for deep scrub","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":739,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1808567,"num_write_kb":20359177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":362904,"ondisk_log_size":362904,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18590025,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":4539,"num_write_kb":19914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37421056,"data_stored":37180050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3759,"ondisk_log_size":3759,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":72,"ondisk_log_size":72,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509658,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729492,"kb_used_data":13132,"kb_used_omap":1,"kb_used_meta":716350,"kb_avail":93642348,"statfs":{"total":96636764160,"available":95889764352,"internally_reserved":0,"allocated":13447168,"data_stored":13261681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":733542858},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509659,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702748,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":702462,"kb_avail":93669092,"statfs":{"total":96636764160,"available":95917150208,"internally_reserved":0,"allocated":282624,"data_stored":90233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719321541},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542363,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384208,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":378174,"kb_avail":93987632,"statfs":{"total":96636764160,"available":96243335168,"internally_reserved":0,"allocated":6168576,"data_stored":5997855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387250630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542364,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698768,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":692734,"kb_avail":93673072,"statfs":{"total":96636764160,"available":95921225728,"internally_reserved":0,"allocated":6168576,"data_stored":5997723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":709360069},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542364,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679136,"kb_used_data":8536,"kb_used_omap":1,"kb_used_meta":670590,"kb_avail":93692704,"statfs":{"total":96636764160,"available":95941328896,"internally_reserved":0,"allocated":8740864,"data_stored":8564586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":686684616},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542365,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400676,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":400318,"kb_avail":93971164,"statfs":{"total":96636764160,"available":96226471936,"internally_reserved":0,"allocated":356352,"data_stored":175944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":409926085},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542364,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":446014,"kb_avail":93925560,"statfs":{"total":96636764160,"available":96179773440,"internally_reserved":0,"allocated":262144,"data_stored":87251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456718789},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542363,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675468,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":670526,"kb_avail":93696372,"statfs":{"total":96636764160,"available":95945084928,"internally_reserved":0,"allocated":5050368,"data_stored":4871265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":686619077},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4222976,"data_stored":4195178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":90225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8507392,"data_stored":8478867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12619776,"data_stored":12585594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:23:54.850 INFO:tasks.ceph:pgid 3.3f last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.850 INFO:tasks.ceph:pgid 3.3e last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.850 INFO:tasks.ceph:pgid 3.3d last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.850 INFO:tasks.ceph:pgid 3.3c last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.850 INFO:tasks.ceph:pgid 3.3b last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.850 INFO:tasks.ceph:pgid 3.3a last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.39 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.38 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.37 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.36 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.35 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.34 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.33 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.32 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.851 INFO:tasks.ceph:pgid 3.31 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.30 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.2f last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.2e last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.2d last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.2c last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.2b last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.2a last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.29 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.852 INFO:tasks.ceph:pgid 3.28 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.27 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.26 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.25 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.24 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.23 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.22 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.21 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.20 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 3.1a last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.853 INFO:tasks.ceph:pgid 2.1b last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.854 INFO:tasks.ceph:pgid 3.1b last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.854 INFO:tasks.ceph:pgid 2.1a last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.854 INFO:tasks.ceph:pgid 2.19 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.854 INFO:tasks.ceph:pgid 3.18 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.854 INFO:tasks.ceph:pgid 2.18 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.854 INFO:tasks.ceph:pgid 3.19 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.854 INFO:tasks.ceph:pgid 2.17 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 3.16 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 2.16 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 3.17 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 2.15 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 3.14 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 2.14 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 3.15 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 2.13 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.855 INFO:tasks.ceph:pgid 3.12 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 2.12 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 3.13 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 2.11 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 3.10 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 2.10 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 3.11 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 2.f last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 3.e last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.856 INFO:tasks.ceph:pgid 2.e last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 3.f last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 2.d last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 3.c last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 2.c last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 3.d last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-05-09T05:05:51.871680+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=5, tm_sec=51, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 3.2 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.857 INFO:tasks.ceph:pgid 3.1 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.858 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.858 INFO:tasks.ceph:pgid 3.0 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.858 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.858 INFO:tasks.ceph:pgid 3.3 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.858 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.858 INFO:tasks.ceph:pgid 3.5 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.858 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.858 INFO:tasks.ceph:pgid 3.4 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 3.7 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 3.6 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 2.8 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 3.9 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 2.9 last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 3.8 last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 2.a last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.859 INFO:tasks.ceph:pgid 3.b last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 2.b last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 3.a last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 3.1d last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 2.1c last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 3.1c last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 2.1d last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 3.1f last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 2.1e last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.860 INFO:tasks.ceph:pgid 3.1e last_scrub_stamp 2024-05-09T05:06:04.034274+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=4, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.861 INFO:tasks.ceph:pgid 2.1f last_scrub_stamp 2024-05-09T05:06:03.028443+0000 time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=6, tm_sec=3, tm_wday=3, tm_yday=130, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=5, tm_mday=9, tm_hour=5, tm_min=23, tm_sec=49, tm_wday=3, tm_yday=130, tm_isdst=0) 2024-05-09T05:23:54.861 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-05-09T05:24:06.849 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:24:06.853 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:24:14.861 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T05:24:14.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.995+0000 7f6bfcc7e640 1 Processor -- start 2024-05-09T05:24:14.991 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.995+0000 7f6bfcc7e640 1 -- start start 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.996+0000 7f6bfcc7e640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f6bf80d16b0 0x7f6bf80d03d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.996+0000 7f6bfcc7e640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6bf815ca90 0x7f6bf815cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.996+0000 7f6bfcc7e640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f6bf806f6d0 con 0x7f6bf806f1b0 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.996+0000 7f6bfcc7e640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf80cee50 con 0x7f6bf815ca90 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.996+0000 7f6bfcc7e640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f6bf80cd0e0 con 0x7f6bf80d16b0 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.996+0000 7f6bf6d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f6bf815ca90 0x7f6bf815cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.996+0000 7f6bf5d9b640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f6bf806f1b0 0x7f6bf80ce740 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37448/0 (socket says 172.21.15.1:37448) 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.996+0000 7f6bf5d9b640 1 -- 172.21.15.1:0/665140264 learned_addr learned my addr 172.21.15.1:0/665140264 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.997+0000 7f6bf659c640 1 --2- 172.21.15.1:0/665140264 >> v2:172.21.15.196:3300/0 conn(0x7f6bf80d16b0 0x7f6bf80d03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:14.993 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.997+0000 7f6bf559a640 1 -- 172.21.15.1:0/665140264 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3991778634 0 0) 0x7f6bf806f6d0 con 0x7f6bf806f1b0 2024-05-09T05:24:14.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.997+0000 7f6bf559a640 1 -- 172.21.15.1:0/665140264 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7f6be4003520 con 0x7f6bf806f1b0 2024-05-09T05:24:14.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.997+0000 7f6bf6d9d640 1 -- 172.21.15.1:0/665140264 >> v2:172.21.15.196:3300/0 conn(0x7f6bf80d16b0 msgr2=0x7f6bf80d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:14.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.997+0000 7f6bf6d9d640 1 --2- 172.21.15.1:0/665140264 >> v2:172.21.15.196:3300/0 conn(0x7f6bf80d16b0 0x7f6bf80d03d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:14.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.997+0000 7f6bf6d9d640 1 -- 172.21.15.1:0/665140264 >> v1:172.21.15.1:6789/0 conn(0x7f6bf806f1b0 legacy=0x7f6bf80ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:14.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.997+0000 7f6bf6d9d640 1 -- 172.21.15.1:0/665140264 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf80cc4d0 con 0x7f6bf815ca90 2024-05-09T05:24:14.994 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.997+0000 7f6bf659c640 1 --2- 172.21.15.1:0/665140264 >> v2:172.21.15.196:3300/0 conn(0x7f6bf80d16b0 0x7f6bf80d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:24:14.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bf6d9d640 1 --2- 172.21.15.1:0/665140264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf815ca90 0x7f6bf815cea0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6bf806d120 tx=0x7f6be802a230 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd7863a1dfcb825e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:14.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bf559a640 1 -- 172.21.15.1:0/665140264 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8037070 con 0x7f6bf815ca90 2024-05-09T05:24:14.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bf559a640 1 -- 172.21.15.1:0/665140264 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be8032440 con 0x7f6bf815ca90 2024-05-09T05:24:14.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bf559a640 1 -- 172.21.15.1:0/665140264 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be8032810 con 0x7f6bf815ca90 2024-05-09T05:24:14.995 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/665140264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf815ca90 msgr2=0x7f6bf815cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/665140264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf815ca90 0x7f6bf815cea0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f6bf806d120 tx=0x7f6be802a230 comp rx=0 tx=0).stop 2024-05-09T05:24:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/665140264 shutdown_connections 2024-05-09T05:24:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/665140264 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf815ca90 0x7f6bf815cea0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/665140264 >> v2:172.21.15.196:3300/0 conn(0x7f6bf80d16b0 0x7f6bf80d03d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.998+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/665140264 >> 172.21.15.1:0/665140264 conn(0x7f6bf80b3660 msgr2=0x7f6bf80b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.999+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/665140264 shutdown_connections 2024-05-09T05:24:14.996 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:14.999+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/665140264 wait complete. 2024-05-09T05:24:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.001+0000 7f6bfcc7e640 1 Processor -- start 2024-05-09T05:24:14.997 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.001+0000 7f6bfcc7e640 1 -- start start 2024-05-09T05:24:14.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bfcc7e640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6bf806f1b0 0x7f6bf80b4a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:14.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bfcc7e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf80d16b0 0x7f6bf8152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:14.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bfcc7e640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bf815ca90 0x7f6bf80ca360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:14.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bfcc7e640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf80ccd70 con 0x7f6bf80d16b0 2024-05-09T05:24:14.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bfcc7e640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f6bf80cca90 con 0x7f6bf815ca90 2024-05-09T05:24:14.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bfcc7e640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f6bf80d0910 con 0x7f6bf806f1b0 2024-05-09T05:24:14.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf80d16b0 0x7f6bf8152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:14.999 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf5d9b640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf80d16b0 0x7f6bf8152170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49812/0 (socket says 172.21.15.1:49812) 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf5d9b640 1 -- 172.21.15.1:0/3468633925 learned_addr learned my addr 172.21.15.1:0/3468633925 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf659c640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6bf806f1b0 0x7f6bf80b4a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf6d9d640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bf815ca90 0x7f6bf80ca360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf5d9b640 1 -- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6bf806f1b0 msgr2=0x7f6bf80b4a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf5d9b640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6bf806f1b0 0x7f6bf80b4a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf5d9b640 1 -- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bf815ca90 msgr2=0x7f6bf80ca360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf5d9b640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bf815ca90 0x7f6bf80ca360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.002+0000 7f6bf5d9b640 1 -- 172.21.15.1:0/3468633925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6bf80c8780 con 0x7f6bf80d16b0 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.003+0000 7f6bf5d9b640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf80d16b0 0x7f6bf8152170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6be000c410 tx=0x7f6be000c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.003+0000 7f6beeffd640 1 -- 172.21.15.1:0/3468633925 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be0017070 con 0x7f6bf80d16b0 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.003+0000 7f6beeffd640 1 -- 172.21.15.1:0/3468633925 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f6be0005ce0 con 0x7f6bf80d16b0 2024-05-09T05:24:15.000 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.003+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6bf81f2fa0 con 0x7f6bf80d16b0 2024-05-09T05:24:15.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.003+0000 7f6beeffd640 1 -- 172.21.15.1:0/3468633925 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6be00072d0 con 0x7f6bf80d16b0 2024-05-09T05:24:15.001 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.003+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6bf80b7440 con 0x7f6bf80d16b0 2024-05-09T05:24:15.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.004+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6bf806f6d0 con 0x7f6bf80d16b0 2024-05-09T05:24:15.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.005+0000 7f6beeffd640 1 -- 172.21.15.1:0/3468633925 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f6be001a070 con 0x7f6bf80d16b0 2024-05-09T05:24:15.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.005+0000 7f6beeffd640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6bd80762d0 0x7f6bd80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.005+0000 7f6beeffd640 1 -- 172.21.15.1:0/3468633925 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f6be00968c0 con 0x7f6bf80d16b0 2024-05-09T05:24:15.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.005+0000 7f6bf659c640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6bd80762d0 0x7f6bd80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.002 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.006+0000 7f6bf659c640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6bd80762d0 0x7f6bd80787e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009fd0 tx=0x7f6bf0009870 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:15.005 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.010+0000 7f6beeffd640 1 -- 172.21.15.1:0/3468633925 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f6bf806f6d0 con 0x7f6bf80d16b0 2024-05-09T05:24:15.193 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.197+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 --> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f6bf80c98a0 con 0x7f6bd80762d0 2024-05-09T05:24:15.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.204+0000 7f6beeffd640 1 -- 172.21.15.1:0/3468633925 <== mgr.4099 v2:172.21.15.196:6832/1287021241 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+254249 (secure 0 0 0) 0x7f6bf80c98a0 con 0x7f6bd80762d0 2024-05-09T05:24:15.203 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:24:15.204 INFO:teuthology.orchestra.run.smithi001.stderr:dumped all 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6bd80762d0 msgr2=0x7f6bd80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6bd80762d0 0x7f6bd80787e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f6bf0009fd0 tx=0x7f6bf0009870 comp rx=0 tx=0).stop 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf80d16b0 msgr2=0x7f6bf8152170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf80d16b0 0x7f6bf8152170 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f6be000c410 tx=0x7f6be000c8d0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 shutdown_connections 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f6bd80762d0 0x7f6bd80787e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f6bf815ca90 0x7f6bf80ca360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f6bf80d16b0 0x7f6bf8152170 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.206 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 --2- 172.21.15.1:0/3468633925 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f6bf806f1b0 0x7f6bf80b4a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 >> 172.21.15.1:0/3468633925 conn(0x7f6bf80b3660 msgr2=0x7f6bf80cefd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:15.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 shutdown_connections 2024-05-09T05:24:15.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.208+0000 7f6bfcc7e640 1 -- 172.21.15.1:0/3468633925 wait complete. 2024-05-09T05:24:15.215 INFO:teuthology.orchestra.run.smithi001.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2024-05-09T05:24:13.840543+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":19180393,"num_objects":105,"num_object_clones":0,"num_object_copies":210,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":105,"num_whiteouts":0,"num_read":102,"num_read_kb":167,"num_write":1813207,"num_write_kb":20380719,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":366832,"ondisk_log_size":366832,"up":194,"acting":194,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":194,"num_osds":8,"num_per_pool_osds":8,"num_per_pool_omap_osds":8,"kb":754974720,"kb_used":4716776,"kb_used_data":39528,"kb_used_omap":12,"kb_used_meta":4677171,"kb_avail":750257944,"statfs":{"total":773094113280,"available":768264134656,"internally_reserved":0,"allocated":40476672,"data_stored":39046538,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":12751,"internal_metadata":4789423665},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003551"},"pg_stats":[{"pgid":"3.3f","version":"23'5187","reported_seq":5220,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:10.285043+0000","last_change":"2024-05-09T05:24:10.285104+0000","last_active":"2024-05-09T05:24:10.285043+0000","last_peered":"2024-05-09T05:24:10.285043+0000","last_clean":"2024-05-09T05:24:10.285043+0000","last_became_active":"2024-05-09T05:06:05.068918+0000","last_became_peered":"2024-05-09T05:06:05.068918+0000","last_unstale":"2024-05-09T05:24:10.285043+0000","last_undegraded":"2024-05-09T05:24:10.285043+0000","last_fullsized":"2024-05-09T05:24:10.285043+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5187","last_scrub_stamp":"2024-05-09T05:24:10.284882+0000","last_deep_scrub":"23'5187","last_deep_scrub_stamp":"2024-05-09T05:24:10.284882+0000","last_clean_scrub_stamp":"2024-05-09T05:24:10.284882+0000","objects_scrubbed":0,"log_size":5187,"log_dups_size":0,"ondisk_log_size":5187,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:27:12.429791+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26135,"num_write_kb":280897,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3e","version":"23'6912","reported_seq":6945,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:00.073146+0000","last_change":"2024-05-09T05:24:00.073180+0000","last_active":"2024-05-09T05:24:00.073146+0000","last_peered":"2024-05-09T05:24:00.073146+0000","last_clean":"2024-05-09T05:24:00.073146+0000","last_became_active":"2024-05-09T05:06:05.058468+0000","last_became_peered":"2024-05-09T05:06:05.058468+0000","last_unstale":"2024-05-09T05:24:00.073146+0000","last_undegraded":"2024-05-09T05:24:00.073146+0000","last_fullsized":"2024-05-09T05:24:00.073146+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6912","last_scrub_stamp":"2024-05-09T05:24:00.073015+0000","last_deep_scrub":"23'6912","last_deep_scrub_stamp":"2024-05-09T05:24:00.073015+0000","last_clean_scrub_stamp":"2024-05-09T05:24:00.073015+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:08:30.106518+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":35479,"num_write_kb":400336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.3d","version":"23'5571","reported_seq":5604,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:53.775097+0000","last_change":"2024-05-09T05:23:53.775125+0000","last_active":"2024-05-09T05:23:53.775097+0000","last_peered":"2024-05-09T05:23:53.775097+0000","last_clean":"2024-05-09T05:23:53.775097+0000","last_became_active":"2024-05-09T05:06:05.054451+0000","last_became_peered":"2024-05-09T05:06:05.054451+0000","last_unstale":"2024-05-09T05:23:53.775097+0000","last_undegraded":"2024-05-09T05:23:53.775097+0000","last_fullsized":"2024-05-09T05:23:53.775097+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5571","last_scrub_stamp":"2024-05-09T05:23:53.774986+0000","last_deep_scrub":"23'5571","last_deep_scrub_stamp":"2024-05-09T05:23:53.774986+0000","last_clean_scrub_stamp":"2024-05-09T05:23:53.774986+0000","objects_scrubbed":0,"log_size":5571,"log_dups_size":0,"ondisk_log_size":5571,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:07.153648+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27199,"num_write_kb":317974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2],"acting":[0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"23'5494","reported_seq":5528,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:03.884272+0000","last_change":"2024-05-09T05:24:03.884272+0000","last_active":"2024-05-09T05:24:03.884272+0000","last_peered":"2024-05-09T05:24:03.884272+0000","last_clean":"2024-05-09T05:24:03.884272+0000","last_became_active":"2024-05-09T05:06:05.065735+0000","last_became_peered":"2024-05-09T05:06:05.065735+0000","last_unstale":"2024-05-09T05:24:03.884272+0000","last_undegraded":"2024-05-09T05:24:03.884272+0000","last_fullsized":"2024-05-09T05:24:03.884272+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5494","last_scrub_stamp":"2024-05-09T05:24:03.884050+0000","last_deep_scrub":"23'5494","last_deep_scrub_stamp":"2024-05-09T05:24:03.884050+0000","last_clean_scrub_stamp":"2024-05-09T05:24:03.884050+0000","objects_scrubbed":0,"log_size":5494,"log_dups_size":0,"ondisk_log_size":5494,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:45:36.597402+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28054,"num_write_kb":296869,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3b","version":"23'7058","reported_seq":7090,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:04.337152+0000","last_change":"2024-05-09T05:24:04.337221+0000","last_active":"2024-05-09T05:24:04.337152+0000","last_peered":"2024-05-09T05:24:04.337152+0000","last_clean":"2024-05-09T05:24:04.337152+0000","last_became_active":"2024-05-09T05:06:05.063129+0000","last_became_peered":"2024-05-09T05:06:05.063129+0000","last_unstale":"2024-05-09T05:24:04.337152+0000","last_undegraded":"2024-05-09T05:24:04.337152+0000","last_fullsized":"2024-05-09T05:24:04.337152+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7058","last_scrub_stamp":"2024-05-09T05:24:04.336994+0000","last_deep_scrub":"23'7058","last_deep_scrub_stamp":"2024-05-09T05:24:04.336994+0000","last_clean_scrub_stamp":"2024-05-09T05:24:04.336994+0000","objects_scrubbed":0,"log_size":7058,"log_dups_size":0,"ondisk_log_size":7058,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:01:15.546162+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34281,"num_write_kb":379027,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3a","version":"23'4948","reported_seq":4980,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:59.817331+0000","last_change":"2024-05-09T05:23:59.817359+0000","last_active":"2024-05-09T05:23:59.817331+0000","last_peered":"2024-05-09T05:23:59.817331+0000","last_clean":"2024-05-09T05:23:59.817331+0000","last_became_active":"2024-05-09T05:06:05.057923+0000","last_became_peered":"2024-05-09T05:06:05.057923+0000","last_unstale":"2024-05-09T05:23:59.817331+0000","last_undegraded":"2024-05-09T05:23:59.817331+0000","last_fullsized":"2024-05-09T05:23:59.817331+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4948","last_scrub_stamp":"2024-05-09T05:23:59.817228+0000","last_deep_scrub":"23'4948","last_deep_scrub_stamp":"2024-05-09T05:23:59.817228+0000","last_clean_scrub_stamp":"2024-05-09T05:23:59.817228+0000","objects_scrubbed":0,"log_size":4948,"log_dups_size":0,"ondisk_log_size":4948,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:13:00.347109+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24780,"num_write_kb":275405,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.39","version":"23'5454","reported_seq":5486,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:09.305429+0000","last_change":"2024-05-09T05:24:09.305501+0000","last_active":"2024-05-09T05:24:09.305429+0000","last_peered":"2024-05-09T05:24:09.305429+0000","last_clean":"2024-05-09T05:24:09.305429+0000","last_became_active":"2024-05-09T05:06:05.068228+0000","last_became_peered":"2024-05-09T05:06:05.068228+0000","last_unstale":"2024-05-09T05:24:09.305429+0000","last_undegraded":"2024-05-09T05:24:09.305429+0000","last_fullsized":"2024-05-09T05:24:09.305429+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5454","last_scrub_stamp":"2024-05-09T05:24:09.305294+0000","last_deep_scrub":"23'5454","last_deep_scrub_stamp":"2024-05-09T05:24:09.305294+0000","last_clean_scrub_stamp":"2024-05-09T05:24:09.305294+0000","objects_scrubbed":0,"log_size":5454,"log_dups_size":0,"ondisk_log_size":5454,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:26:41.190573+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27432,"num_write_kb":306787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"23'6053","reported_seq":6085,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:00.220556+0000","last_change":"2024-05-09T05:24:00.220620+0000","last_active":"2024-05-09T05:24:00.220556+0000","last_peered":"2024-05-09T05:24:00.220556+0000","last_clean":"2024-05-09T05:24:00.220556+0000","last_became_active":"2024-05-09T05:06:05.067156+0000","last_became_peered":"2024-05-09T05:06:05.067156+0000","last_unstale":"2024-05-09T05:24:00.220556+0000","last_undegraded":"2024-05-09T05:24:00.220556+0000","last_fullsized":"2024-05-09T05:24:00.220556+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6053","last_scrub_stamp":"2024-05-09T05:24:00.220417+0000","last_deep_scrub":"23'6053","last_deep_scrub_stamp":"2024-05-09T05:24:00.220417+0000","last_clean_scrub_stamp":"2024-05-09T05:24:00.220417+0000","objects_scrubbed":0,"log_size":6053,"log_dups_size":0,"ondisk_log_size":6053,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:29:05.833834+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29445,"num_write_kb":339212,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"23'7129","reported_seq":7161,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:08.317663+0000","last_change":"2024-05-09T05:24:08.317696+0000","last_active":"2024-05-09T05:24:08.317663+0000","last_peered":"2024-05-09T05:24:08.317663+0000","last_clean":"2024-05-09T05:24:08.317663+0000","last_became_active":"2024-05-09T05:06:05.068557+0000","last_became_peered":"2024-05-09T05:06:05.068557+0000","last_unstale":"2024-05-09T05:24:08.317663+0000","last_undegraded":"2024-05-09T05:24:08.317663+0000","last_fullsized":"2024-05-09T05:24:08.317663+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7129","last_scrub_stamp":"2024-05-09T05:24:08.317534+0000","last_deep_scrub":"23'7129","last_deep_scrub_stamp":"2024-05-09T05:24:08.317534+0000","last_clean_scrub_stamp":"2024-05-09T05:24:08.317534+0000","objects_scrubbed":0,"log_size":7129,"log_dups_size":0,"ondisk_log_size":7129,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:57:58.489139+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36611,"num_write_kb":392539,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.36","version":"23'6358","reported_seq":6390,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:02.682162+0000","last_change":"2024-05-09T05:24:02.682192+0000","last_active":"2024-05-09T05:24:02.682162+0000","last_peered":"2024-05-09T05:24:02.682162+0000","last_clean":"2024-05-09T05:24:02.682162+0000","last_became_active":"2024-05-09T05:06:05.062710+0000","last_became_peered":"2024-05-09T05:06:05.062710+0000","last_unstale":"2024-05-09T05:24:02.682162+0000","last_undegraded":"2024-05-09T05:24:02.682162+0000","last_fullsized":"2024-05-09T05:24:02.682162+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6358","last_scrub_stamp":"2024-05-09T05:24:02.682050+0000","last_deep_scrub":"23'6358","last_deep_scrub_stamp":"2024-05-09T05:24:02.682050+0000","last_clean_scrub_stamp":"2024-05-09T05:24:02.682050+0000","objects_scrubbed":0,"log_size":6358,"log_dups_size":0,"ondisk_log_size":6358,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:08:32.715554+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32647,"num_write_kb":360281,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.35","version":"23'4747","reported_seq":4779,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:04.612302+0000","last_change":"2024-05-09T05:24:04.612362+0000","last_active":"2024-05-09T05:24:04.612302+0000","last_peered":"2024-05-09T05:24:04.612302+0000","last_clean":"2024-05-09T05:24:04.612302+0000","last_became_active":"2024-05-09T05:06:05.065753+0000","last_became_peered":"2024-05-09T05:06:05.065753+0000","last_unstale":"2024-05-09T05:24:04.612302+0000","last_undegraded":"2024-05-09T05:24:04.612302+0000","last_fullsized":"2024-05-09T05:24:04.612302+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4747","last_scrub_stamp":"2024-05-09T05:24:04.612153+0000","last_deep_scrub":"23'4747","last_deep_scrub_stamp":"2024-05-09T05:24:04.612153+0000","last_clean_scrub_stamp":"2024-05-09T05:24:04.612153+0000","objects_scrubbed":0,"log_size":4747,"log_dups_size":0,"ondisk_log_size":4747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:36.450482+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22850,"num_write_kb":288147,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.34","version":"23'4814","reported_seq":4846,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:57.278486+0000","last_change":"2024-05-09T05:23:57.278532+0000","last_active":"2024-05-09T05:23:57.278486+0000","last_peered":"2024-05-09T05:23:57.278486+0000","last_clean":"2024-05-09T05:23:57.278486+0000","last_became_active":"2024-05-09T05:06:05.061083+0000","last_became_peered":"2024-05-09T05:06:05.061083+0000","last_unstale":"2024-05-09T05:23:57.278486+0000","last_undegraded":"2024-05-09T05:23:57.278486+0000","last_fullsized":"2024-05-09T05:23:57.278486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4814","last_scrub_stamp":"2024-05-09T05:23:57.278377+0000","last_deep_scrub":"23'4814","last_deep_scrub_stamp":"2024-05-09T05:23:57.278377+0000","last_clean_scrub_stamp":"2024-05-09T05:23:57.278377+0000","objects_scrubbed":0,"log_size":4814,"log_dups_size":0,"ondisk_log_size":4814,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:20:12.159569+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23824,"num_write_kb":267591,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6],"acting":[4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"23'5105","reported_seq":5137,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:01.198644+0000","last_change":"2024-05-09T05:24:01.198679+0000","last_active":"2024-05-09T05:24:01.198644+0000","last_peered":"2024-05-09T05:24:01.198644+0000","last_clean":"2024-05-09T05:24:01.198644+0000","last_became_active":"2024-05-09T05:06:05.069044+0000","last_became_peered":"2024-05-09T05:06:05.069044+0000","last_unstale":"2024-05-09T05:24:01.198644+0000","last_undegraded":"2024-05-09T05:24:01.198644+0000","last_fullsized":"2024-05-09T05:24:01.198644+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5105","last_scrub_stamp":"2024-05-09T05:24:01.198528+0000","last_deep_scrub":"23'5105","last_deep_scrub_stamp":"2024-05-09T05:24:01.198528+0000","last_clean_scrub_stamp":"2024-05-09T05:24:01.198528+0000","objects_scrubbed":0,"log_size":5105,"log_dups_size":0,"ondisk_log_size":5105,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:52:26.481159+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25963,"num_write_kb":288363,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.32","version":"23'5957","reported_seq":5989,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:55.788219+0000","last_change":"2024-05-09T05:23:55.788275+0000","last_active":"2024-05-09T05:23:55.788219+0000","last_peered":"2024-05-09T05:23:55.788219+0000","last_clean":"2024-05-09T05:23:55.788219+0000","last_became_active":"2024-05-09T05:06:05.057725+0000","last_became_peered":"2024-05-09T05:06:05.057725+0000","last_unstale":"2024-05-09T05:23:55.788219+0000","last_undegraded":"2024-05-09T05:23:55.788219+0000","last_fullsized":"2024-05-09T05:23:55.788219+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5957","last_scrub_stamp":"2024-05-09T05:23:55.788053+0000","last_deep_scrub":"23'5957","last_deep_scrub_stamp":"2024-05-09T05:23:55.788053+0000","last_clean_scrub_stamp":"2024-05-09T05:23:55.788053+0000","objects_scrubbed":0,"log_size":5957,"log_dups_size":0,"ondisk_log_size":5957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:13.448757+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29976,"num_write_kb":329157,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"23'5739","reported_seq":5771,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:02.113228+0000","last_change":"2024-05-09T05:24:02.113260+0000","last_active":"2024-05-09T05:24:02.113228+0000","last_peered":"2024-05-09T05:24:02.113228+0000","last_clean":"2024-05-09T05:24:02.113228+0000","last_became_active":"2024-05-09T05:06:05.062841+0000","last_became_peered":"2024-05-09T05:06:05.062841+0000","last_unstale":"2024-05-09T05:24:02.113228+0000","last_undegraded":"2024-05-09T05:24:02.113228+0000","last_fullsized":"2024-05-09T05:24:02.113228+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5739","last_scrub_stamp":"2024-05-09T05:24:02.113110+0000","last_deep_scrub":"23'5739","last_deep_scrub_stamp":"2024-05-09T05:24:02.113110+0000","last_clean_scrub_stamp":"2024-05-09T05:24:02.113110+0000","objects_scrubbed":0,"log_size":5739,"log_dups_size":0,"ondisk_log_size":5739,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:43:33.951438+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28226,"num_write_kb":322058,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.30","version":"23'5341","reported_seq":5373,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:58.793170+0000","last_change":"2024-05-09T05:23:58.793220+0000","last_active":"2024-05-09T05:23:58.793170+0000","last_peered":"2024-05-09T05:23:58.793170+0000","last_clean":"2024-05-09T05:23:58.793170+0000","last_became_active":"2024-05-09T05:06:05.052156+0000","last_became_peered":"2024-05-09T05:06:05.052156+0000","last_unstale":"2024-05-09T05:23:58.793170+0000","last_undegraded":"2024-05-09T05:23:58.793170+0000","last_fullsized":"2024-05-09T05:23:58.793170+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5341","last_scrub_stamp":"2024-05-09T05:23:58.793032+0000","last_deep_scrub":"23'5341","last_deep_scrub_stamp":"2024-05-09T05:23:58.793032+0000","last_clean_scrub_stamp":"2024-05-09T05:23:58.793032+0000","objects_scrubbed":0,"log_size":5341,"log_dups_size":0,"ondisk_log_size":5341,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:51.883731+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27307,"num_write_kb":289793,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0],"acting":[5,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2f","version":"23'6965","reported_seq":6991,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:59.676912+0000","last_change":"2024-05-09T05:23:59.676946+0000","last_active":"2024-05-09T05:23:59.676912+0000","last_peered":"2024-05-09T05:23:59.676912+0000","last_clean":"2024-05-09T05:23:59.676912+0000","last_became_active":"2024-05-09T05:07:01.750492+0000","last_became_peered":"2024-05-09T05:07:01.750492+0000","last_unstale":"2024-05-09T05:23:59.676912+0000","last_undegraded":"2024-05-09T05:23:59.676912+0000","last_fullsized":"2024-05-09T05:23:59.676912+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6965","last_scrub_stamp":"2024-05-09T05:23:59.676782+0000","last_deep_scrub":"23'6965","last_deep_scrub_stamp":"2024-05-09T05:23:59.676782+0000","last_clean_scrub_stamp":"2024-05-09T05:23:59.676782+0000","objects_scrubbed":0,"log_size":6965,"log_dups_size":0,"ondisk_log_size":6965,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:07.694831+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34877,"num_write_kb":385595,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,7],"acting":[2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2e","version":"23'4549","reported_seq":4581,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:57.735383+0000","last_change":"2024-05-09T05:23:57.735413+0000","last_active":"2024-05-09T05:23:57.735383+0000","last_peered":"2024-05-09T05:23:57.735383+0000","last_clean":"2024-05-09T05:23:57.735383+0000","last_became_active":"2024-05-09T05:06:05.064520+0000","last_became_peered":"2024-05-09T05:06:05.064520+0000","last_unstale":"2024-05-09T05:23:57.735383+0000","last_undegraded":"2024-05-09T05:23:57.735383+0000","last_fullsized":"2024-05-09T05:23:57.735383+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4549","last_scrub_stamp":"2024-05-09T05:23:57.735271+0000","last_deep_scrub":"23'4549","last_deep_scrub_stamp":"2024-05-09T05:23:57.735271+0000","last_clean_scrub_stamp":"2024-05-09T05:23:57.735271+0000","objects_scrubbed":0,"log_size":4549,"log_dups_size":0,"ondisk_log_size":4549,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:56:53.123595+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22029,"num_write_kb":252128,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6],"acting":[2,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.2d","version":"23'5335","reported_seq":5368,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:05.347368+0000","last_change":"2024-05-09T05:24:05.347368+0000","last_active":"2024-05-09T05:24:05.347368+0000","last_peered":"2024-05-09T05:24:05.347368+0000","last_clean":"2024-05-09T05:24:05.347368+0000","last_became_active":"2024-05-09T05:06:05.064010+0000","last_became_peered":"2024-05-09T05:06:05.064010+0000","last_unstale":"2024-05-09T05:24:05.347368+0000","last_undegraded":"2024-05-09T05:24:05.347368+0000","last_fullsized":"2024-05-09T05:24:05.347368+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5335","last_scrub_stamp":"2024-05-09T05:24:05.347166+0000","last_deep_scrub":"23'5335","last_deep_scrub_stamp":"2024-05-09T05:24:05.347166+0000","last_clean_scrub_stamp":"2024-05-09T05:24:05.347166+0000","objects_scrubbed":0,"log_size":5335,"log_dups_size":0,"ondisk_log_size":5335,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:33:52.041011+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26106,"num_write_kb":309468,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,2],"acting":[7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2c","version":"23'5370","reported_seq":5403,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:59.652830+0000","last_change":"2024-05-09T05:23:59.652830+0000","last_active":"2024-05-09T05:23:59.652830+0000","last_peered":"2024-05-09T05:23:59.652830+0000","last_clean":"2024-05-09T05:23:59.652830+0000","last_became_active":"2024-05-09T05:06:05.061196+0000","last_became_peered":"2024-05-09T05:06:05.061196+0000","last_unstale":"2024-05-09T05:23:59.652830+0000","last_undegraded":"2024-05-09T05:23:59.652830+0000","last_fullsized":"2024-05-09T05:23:59.652830+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5370","last_scrub_stamp":"2024-05-09T05:23:59.652665+0000","last_deep_scrub":"23'5370","last_deep_scrub_stamp":"2024-05-09T05:23:59.652665+0000","last_clean_scrub_stamp":"2024-05-09T05:23:59.652665+0000","objects_scrubbed":0,"log_size":5370,"log_dups_size":0,"ondisk_log_size":5370,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:48:15.786021+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26990,"num_write_kb":308937,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2b","version":"23'5403","reported_seq":5436,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:00.792933+0000","last_change":"2024-05-09T05:24:00.792933+0000","last_active":"2024-05-09T05:24:00.792933+0000","last_peered":"2024-05-09T05:24:00.792933+0000","last_clean":"2024-05-09T05:24:00.792933+0000","last_became_active":"2024-05-09T05:06:05.064106+0000","last_became_peered":"2024-05-09T05:06:05.064106+0000","last_unstale":"2024-05-09T05:24:00.792933+0000","last_undegraded":"2024-05-09T05:24:00.792933+0000","last_fullsized":"2024-05-09T05:24:00.792933+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5403","last_scrub_stamp":"2024-05-09T05:24:00.792759+0000","last_deep_scrub":"23'5403","last_deep_scrub_stamp":"2024-05-09T05:24:00.792759+0000","last_clean_scrub_stamp":"2024-05-09T05:24:00.792759+0000","objects_scrubbed":0,"log_size":5403,"log_dups_size":0,"ondisk_log_size":5403,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:57:23.795180+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26698,"num_write_kb":315887,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.2a","version":"23'6082","reported_seq":6115,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:07.955840+0000","last_change":"2024-05-09T05:24:07.955871+0000","last_active":"2024-05-09T05:24:07.955840+0000","last_peered":"2024-05-09T05:24:07.955840+0000","last_clean":"2024-05-09T05:24:07.955840+0000","last_became_active":"2024-05-09T05:06:05.071411+0000","last_became_peered":"2024-05-09T05:06:05.071411+0000","last_unstale":"2024-05-09T05:24:07.955840+0000","last_undegraded":"2024-05-09T05:24:07.955840+0000","last_fullsized":"2024-05-09T05:24:07.955840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6082","last_scrub_stamp":"2024-05-09T05:24:07.955722+0000","last_deep_scrub":"23'6082","last_deep_scrub_stamp":"2024-05-09T05:24:07.955722+0000","last_clean_scrub_stamp":"2024-05-09T05:24:07.955722+0000","objects_scrubbed":0,"log_size":6082,"log_dups_size":0,"ondisk_log_size":6082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:57:58.127327+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30462,"num_write_kb":343332,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"23'5512","reported_seq":5545,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:57.233514+0000","last_change":"2024-05-09T05:23:57.233514+0000","last_active":"2024-05-09T05:23:57.233514+0000","last_peered":"2024-05-09T05:23:57.233514+0000","last_clean":"2024-05-09T05:23:57.233514+0000","last_became_active":"2024-05-09T05:06:05.062570+0000","last_became_peered":"2024-05-09T05:06:05.062570+0000","last_unstale":"2024-05-09T05:23:57.233514+0000","last_undegraded":"2024-05-09T05:23:57.233514+0000","last_fullsized":"2024-05-09T05:23:57.233514+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5512","last_scrub_stamp":"2024-05-09T05:23:57.233297+0000","last_deep_scrub":"23'5512","last_deep_scrub_stamp":"2024-05-09T05:23:57.233297+0000","last_clean_scrub_stamp":"2024-05-09T05:23:57.233297+0000","objects_scrubbed":0,"log_size":5512,"log_dups_size":0,"ondisk_log_size":5512,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:02:23.073247+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27471,"num_write_kb":310767,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"23'5065","reported_seq":5098,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:02.894550+0000","last_change":"2024-05-09T05:24:02.894582+0000","last_active":"2024-05-09T05:24:02.894550+0000","last_peered":"2024-05-09T05:24:02.894550+0000","last_clean":"2024-05-09T05:24:02.894550+0000","last_became_active":"2024-05-09T05:06:05.065773+0000","last_became_peered":"2024-05-09T05:06:05.065773+0000","last_unstale":"2024-05-09T05:24:02.894550+0000","last_undegraded":"2024-05-09T05:24:02.894550+0000","last_fullsized":"2024-05-09T05:24:02.894550+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5065","last_scrub_stamp":"2024-05-09T05:24:02.894445+0000","last_deep_scrub":"23'5065","last_deep_scrub_stamp":"2024-05-09T05:24:02.894445+0000","last_clean_scrub_stamp":"2024-05-09T05:24:02.894445+0000","objects_scrubbed":0,"log_size":5065,"log_dups_size":0,"ondisk_log_size":5065,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:01:14.739326+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24983,"num_write_kb":280386,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.27","version":"23'6632","reported_seq":6665,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:56.272843+0000","last_change":"2024-05-09T05:23:56.272876+0000","last_active":"2024-05-09T05:23:56.272843+0000","last_peered":"2024-05-09T05:23:56.272843+0000","last_clean":"2024-05-09T05:23:56.272843+0000","last_became_active":"2024-05-09T05:06:05.063155+0000","last_became_peered":"2024-05-09T05:06:05.063155+0000","last_unstale":"2024-05-09T05:23:56.272843+0000","last_undegraded":"2024-05-09T05:23:56.272843+0000","last_fullsized":"2024-05-09T05:23:56.272843+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6632","last_scrub_stamp":"2024-05-09T05:23:56.272716+0000","last_deep_scrub":"23'6632","last_deep_scrub_stamp":"2024-05-09T05:23:56.272716+0000","last_clean_scrub_stamp":"2024-05-09T05:23:56.272716+0000","objects_scrubbed":0,"log_size":6632,"log_dups_size":0,"ondisk_log_size":6632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:07:14.248435+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33114,"num_write_kb":370371,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4],"acting":[3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.26","version":"23'5121","reported_seq":5159,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:05.636648+0000","last_change":"2024-05-09T05:24:05.636718+0000","last_active":"2024-05-09T05:24:05.636648+0000","last_peered":"2024-05-09T05:24:05.636648+0000","last_clean":"2024-05-09T05:24:05.636648+0000","last_became_active":"2024-05-09T05:07:01.662096+0000","last_became_peered":"2024-05-09T05:07:01.662096+0000","last_unstale":"2024-05-09T05:24:05.636648+0000","last_undegraded":"2024-05-09T05:24:05.636648+0000","last_fullsized":"2024-05-09T05:24:05.636648+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5121","last_scrub_stamp":"2024-05-09T05:24:05.636531+0000","last_deep_scrub":"23'5121","last_deep_scrub_stamp":"2024-05-09T05:24:05.636531+0000","last_clean_scrub_stamp":"2024-05-09T05:24:05.636531+0000","objects_scrubbed":0,"log_size":5121,"log_dups_size":0,"ondisk_log_size":5121,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:43.586805+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25567,"num_write_kb":287181,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2],"acting":[6,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"23'6014","reported_seq":6047,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:01.096479+0000","last_change":"2024-05-09T05:24:01.096512+0000","last_active":"2024-05-09T05:24:01.096479+0000","last_peered":"2024-05-09T05:24:01.096479+0000","last_clean":"2024-05-09T05:24:01.096479+0000","last_became_active":"2024-05-09T05:06:05.061202+0000","last_became_peered":"2024-05-09T05:06:05.061202+0000","last_unstale":"2024-05-09T05:24:01.096479+0000","last_undegraded":"2024-05-09T05:24:01.096479+0000","last_fullsized":"2024-05-09T05:24:01.096479+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6014","last_scrub_stamp":"2024-05-09T05:24:01.096344+0000","last_deep_scrub":"23'6014","last_deep_scrub_stamp":"2024-05-09T05:24:01.096344+0000","last_clean_scrub_stamp":"2024-05-09T05:24:01.096344+0000","objects_scrubbed":0,"log_size":6014,"log_dups_size":0,"ondisk_log_size":6014,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:25.645360+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30657,"num_write_kb":340145,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.24","version":"23'5132","reported_seq":5158,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:04.157736+0000","last_change":"2024-05-09T05:24:04.157772+0000","last_active":"2024-05-09T05:24:04.157736+0000","last_peered":"2024-05-09T05:24:04.157736+0000","last_clean":"2024-05-09T05:24:04.157736+0000","last_became_active":"2024-05-09T05:07:01.722736+0000","last_became_peered":"2024-05-09T05:07:01.722736+0000","last_unstale":"2024-05-09T05:24:04.157736+0000","last_undegraded":"2024-05-09T05:24:04.157736+0000","last_fullsized":"2024-05-09T05:24:04.157736+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5132","last_scrub_stamp":"2024-05-09T05:24:04.157597+0000","last_deep_scrub":"23'5132","last_deep_scrub_stamp":"2024-05-09T05:24:04.157597+0000","last_clean_scrub_stamp":"2024-05-09T05:24:04.157597+0000","objects_scrubbed":0,"log_size":5132,"log_dups_size":0,"ondisk_log_size":5132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:05:55.781261+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25502,"num_write_kb":286500,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.23","version":"23'4664","reported_seq":4697,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:01.748486+0000","last_change":"2024-05-09T05:24:01.748542+0000","last_active":"2024-05-09T05:24:01.748486+0000","last_peered":"2024-05-09T05:24:01.748486+0000","last_clean":"2024-05-09T05:24:01.748486+0000","last_became_active":"2024-05-09T05:06:05.066089+0000","last_became_peered":"2024-05-09T05:06:05.066089+0000","last_unstale":"2024-05-09T05:24:01.748486+0000","last_undegraded":"2024-05-09T05:24:01.748486+0000","last_fullsized":"2024-05-09T05:24:01.748486+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4664","last_scrub_stamp":"2024-05-09T05:24:01.748344+0000","last_deep_scrub":"23'4664","last_deep_scrub_stamp":"2024-05-09T05:24:01.748344+0000","last_clean_scrub_stamp":"2024-05-09T05:24:01.748344+0000","objects_scrubbed":0,"log_size":4664,"log_dups_size":0,"ondisk_log_size":4664,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:50.510382+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23461,"num_write_kb":270138,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3],"acting":[0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.22","version":"23'6545","reported_seq":6578,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:05.892879+0000","last_change":"2024-05-09T05:24:05.892911+0000","last_active":"2024-05-09T05:24:05.892879+0000","last_peered":"2024-05-09T05:24:05.892879+0000","last_clean":"2024-05-09T05:24:05.892879+0000","last_became_active":"2024-05-09T05:06:05.066500+0000","last_became_peered":"2024-05-09T05:06:05.066500+0000","last_unstale":"2024-05-09T05:24:05.892879+0000","last_undegraded":"2024-05-09T05:24:05.892879+0000","last_fullsized":"2024-05-09T05:24:05.892879+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6545","last_scrub_stamp":"2024-05-09T05:24:05.892771+0000","last_deep_scrub":"23'6545","last_deep_scrub_stamp":"2024-05-09T05:24:05.892771+0000","last_clean_scrub_stamp":"2024-05-09T05:24:05.892771+0000","objects_scrubbed":0,"log_size":6545,"log_dups_size":0,"ondisk_log_size":6545,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:35:03.628721+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32981,"num_write_kb":377918,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.21","version":"23'6387","reported_seq":6421,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:56.751497+0000","last_change":"2024-05-09T05:23:56.751529+0000","last_active":"2024-05-09T05:23:56.751497+0000","last_peered":"2024-05-09T05:23:56.751497+0000","last_clean":"2024-05-09T05:23:56.751497+0000","last_became_active":"2024-05-09T05:06:05.062602+0000","last_became_peered":"2024-05-09T05:06:05.062602+0000","last_unstale":"2024-05-09T05:23:56.751497+0000","last_undegraded":"2024-05-09T05:23:56.751497+0000","last_fullsized":"2024-05-09T05:23:56.751497+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6387","last_scrub_stamp":"2024-05-09T05:23:56.751384+0000","last_deep_scrub":"23'6387","last_deep_scrub_stamp":"2024-05-09T05:23:56.751384+0000","last_clean_scrub_stamp":"2024-05-09T05:23:56.751384+0000","objects_scrubbed":0,"log_size":6387,"log_dups_size":0,"ondisk_log_size":6387,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:31:47.501781+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":32133,"num_write_kb":346002,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.20","version":"23'7361","reported_seq":7394,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:01.645621+0000","last_change":"2024-05-09T05:24:01.645652+0000","last_active":"2024-05-09T05:24:01.645621+0000","last_peered":"2024-05-09T05:24:01.645621+0000","last_clean":"2024-05-09T05:24:01.645621+0000","last_became_active":"2024-05-09T05:06:05.061714+0000","last_became_peered":"2024-05-09T05:06:05.061714+0000","last_unstale":"2024-05-09T05:24:01.645621+0000","last_undegraded":"2024-05-09T05:24:01.645621+0000","last_fullsized":"2024-05-09T05:24:01.645621+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7361","last_scrub_stamp":"2024-05-09T05:24:01.645492+0000","last_deep_scrub":"23'7361","last_deep_scrub_stamp":"2024-05-09T05:24:01.645492+0000","last_clean_scrub_stamp":"2024-05-09T05:24:01.645492+0000","objects_scrubbed":0,"log_size":7361,"log_dups_size":0,"ondisk_log_size":7361,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:36.888211+0000","scrub_duration":14,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36142,"num_write_kb":398870,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1a","version":"23'5228","reported_seq":5260,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:55.292744+0000","last_change":"2024-05-09T05:23:55.292816+0000","last_active":"2024-05-09T05:23:55.292744+0000","last_peered":"2024-05-09T05:23:55.292744+0000","last_clean":"2024-05-09T05:23:55.292744+0000","last_became_active":"2024-05-09T05:06:05.058394+0000","last_became_peered":"2024-05-09T05:06:05.058394+0000","last_unstale":"2024-05-09T05:23:55.292744+0000","last_undegraded":"2024-05-09T05:23:55.292744+0000","last_fullsized":"2024-05-09T05:23:55.292744+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5228","last_scrub_stamp":"2024-05-09T05:23:55.292591+0000","last_deep_scrub":"23'5228","last_deep_scrub_stamp":"2024-05-09T05:23:55.292591+0000","last_clean_scrub_stamp":"2024-05-09T05:23:55.292591+0000","objects_scrubbed":0,"log_size":5228,"log_dups_size":0,"ondisk_log_size":5228,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:21.184665+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25537,"num_write_kb":278419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1],"acting":[4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1b","version":"23'579","reported_seq":613,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:55.307934+0000","last_change":"2024-05-09T05:23:55.307966+0000","last_active":"2024-05-09T05:23:55.307934+0000","last_peered":"2024-05-09T05:23:55.307934+0000","last_clean":"2024-05-09T05:23:55.307934+0000","last_became_active":"2024-05-09T05:06:04.069184+0000","last_became_peered":"2024-05-09T05:06:04.069184+0000","last_unstale":"2024-05-09T05:23:55.307934+0000","last_undegraded":"2024-05-09T05:23:55.307934+0000","last_fullsized":"2024-05-09T05:23:55.307934+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'579","last_scrub_stamp":"2024-05-09T05:23:55.307829+0000","last_deep_scrub":"23'579","last_deep_scrub_stamp":"2024-05-09T05:23:55.307829+0000","last_clean_scrub_stamp":"2024-05-09T05:23:55.307829+0000","objects_scrubbed":3,"log_size":579,"log_dups_size":0,"ondisk_log_size":579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:40.529430+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4195508,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":573,"num_write_kb":4218,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1b","version":"23'4521","reported_seq":4553,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:58.804524+0000","last_change":"2024-05-09T05:23:58.804557+0000","last_active":"2024-05-09T05:23:58.804524+0000","last_peered":"2024-05-09T05:23:58.804524+0000","last_clean":"2024-05-09T05:23:58.804524+0000","last_became_active":"2024-05-09T05:06:05.060822+0000","last_became_peered":"2024-05-09T05:06:05.060822+0000","last_unstale":"2024-05-09T05:23:58.804524+0000","last_undegraded":"2024-05-09T05:23:58.804524+0000","last_fullsized":"2024-05-09T05:23:58.804524+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4521","last_scrub_stamp":"2024-05-09T05:23:58.804421+0000","last_deep_scrub":"23'4521","last_deep_scrub_stamp":"2024-05-09T05:23:58.804421+0000","last_clean_scrub_stamp":"2024-05-09T05:23:58.804421+0000","objects_scrubbed":0,"log_size":4521,"log_dups_size":0,"ondisk_log_size":4521,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:39.431449+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22336,"num_write_kb":249223,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1a","version":"23'114","reported_seq":148,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:55.606438+0000","last_change":"2024-05-09T05:23:55.606480+0000","last_active":"2024-05-09T05:23:55.606438+0000","last_peered":"2024-05-09T05:23:55.606438+0000","last_clean":"2024-05-09T05:23:55.606438+0000","last_became_active":"2024-05-09T05:06:04.058565+0000","last_became_peered":"2024-05-09T05:06:04.058565+0000","last_unstale":"2024-05-09T05:23:55.606438+0000","last_undegraded":"2024-05-09T05:23:55.606438+0000","last_fullsized":"2024-05-09T05:23:55.606438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'114","last_scrub_stamp":"2024-05-09T05:23:55.606329+0000","last_deep_scrub":"23'114","last_deep_scrub_stamp":"2024-05-09T05:23:55.606329+0000","last_clean_scrub_stamp":"2024-05-09T05:23:55.606329+0000","objects_scrubbed":4,"log_size":114,"log_dups_size":0,"ondisk_log_size":114,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:36.055954+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1715788,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":110,"num_write_kb":1741,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.19","version":"23'8","reported_seq":42,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:53.307431+0000","last_change":"2024-05-09T05:23:53.307460+0000","last_active":"2024-05-09T05:23:53.307431+0000","last_peered":"2024-05-09T05:23:53.307431+0000","last_clean":"2024-05-09T05:23:53.307431+0000","last_became_active":"2024-05-09T05:06:04.057809+0000","last_became_peered":"2024-05-09T05:06:04.057809+0000","last_unstale":"2024-05-09T05:23:53.307431+0000","last_undegraded":"2024-05-09T05:23:53.307431+0000","last_fullsized":"2024-05-09T05:23:53.307431+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'8","last_scrub_stamp":"2024-05-09T05:23:53.307355+0000","last_deep_scrub":"23'8","last_deep_scrub_stamp":"2024-05-09T05:23:53.307355+0000","last_clean_scrub_stamp":"2024-05-09T05:23:53.307355+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:14:58.965917+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.18","version":"23'5503","reported_seq":5535,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:03.183050+0000","last_change":"2024-05-09T05:24:03.183082+0000","last_active":"2024-05-09T05:24:03.183050+0000","last_peered":"2024-05-09T05:24:03.183050+0000","last_clean":"2024-05-09T05:24:03.183050+0000","last_became_active":"2024-05-09T05:06:05.070646+0000","last_became_peered":"2024-05-09T05:06:05.070646+0000","last_unstale":"2024-05-09T05:24:03.183050+0000","last_undegraded":"2024-05-09T05:24:03.183050+0000","last_fullsized":"2024-05-09T05:24:03.183050+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5503","last_scrub_stamp":"2024-05-09T05:24:03.182930+0000","last_deep_scrub":"23'5503","last_deep_scrub_stamp":"2024-05-09T05:24:03.182930+0000","last_clean_scrub_stamp":"2024-05-09T05:24:03.182930+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:59:31.033412+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27748,"num_write_kb":311965,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"23'7","reported_seq":41,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:54.846197+0000","last_change":"2024-05-09T05:23:54.846259+0000","last_active":"2024-05-09T05:23:54.846197+0000","last_peered":"2024-05-09T05:23:54.846197+0000","last_clean":"2024-05-09T05:23:54.846197+0000","last_became_active":"2024-05-09T05:06:04.056048+0000","last_became_peered":"2024-05-09T05:06:04.056048+0000","last_unstale":"2024-05-09T05:23:54.846197+0000","last_undegraded":"2024-05-09T05:23:54.846197+0000","last_fullsized":"2024-05-09T05:23:54.846197+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7","last_scrub_stamp":"2024-05-09T05:23:54.846050+0000","last_deep_scrub":"23'7","last_deep_scrub_stamp":"2024-05-09T05:23:54.846050+0000","last_clean_scrub_stamp":"2024-05-09T05:23:54.846050+0000","objects_scrubbed":1,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:12.506755+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":10,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"23'6912","reported_seq":6944,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:03.135109+0000","last_change":"2024-05-09T05:24:03.135138+0000","last_active":"2024-05-09T05:24:03.135109+0000","last_peered":"2024-05-09T05:24:03.135109+0000","last_clean":"2024-05-09T05:24:03.135109+0000","last_became_active":"2024-05-09T05:06:05.065746+0000","last_became_peered":"2024-05-09T05:06:05.065746+0000","last_unstale":"2024-05-09T05:24:03.135109+0000","last_undegraded":"2024-05-09T05:24:03.135109+0000","last_fullsized":"2024-05-09T05:24:03.135109+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6912","last_scrub_stamp":"2024-05-09T05:24:03.134991+0000","last_deep_scrub":"23'6912","last_deep_scrub_stamp":"2024-05-09T05:24:03.134991+0000","last_clean_scrub_stamp":"2024-05-09T05:24:03.134991+0000","objects_scrubbed":0,"log_size":6912,"log_dups_size":0,"ondisk_log_size":6912,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:30:26.185257+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34348,"num_write_kb":375438,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.17","version":"23'10","reported_seq":44,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:57.669432+0000","last_change":"2024-05-09T05:23:57.669490+0000","last_active":"2024-05-09T05:23:57.669432+0000","last_peered":"2024-05-09T05:23:57.669432+0000","last_clean":"2024-05-09T05:23:57.669432+0000","last_became_active":"2024-05-09T05:06:04.061137+0000","last_became_peered":"2024-05-09T05:06:04.061137+0000","last_unstale":"2024-05-09T05:23:57.669432+0000","last_undegraded":"2024-05-09T05:23:57.669432+0000","last_fullsized":"2024-05-09T05:23:57.669432+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'10","last_scrub_stamp":"2024-05-09T05:23:57.669298+0000","last_deep_scrub":"23'10","last_deep_scrub_stamp":"2024-05-09T05:23:57.669298+0000","last_clean_scrub_stamp":"2024-05-09T05:23:57.669298+0000","objects_scrubbed":3,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:07:15.645017+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5],"acting":[6,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.16","version":"23'5546","reported_seq":5578,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:06.917091+0000","last_change":"2024-05-09T05:24:06.917123+0000","last_active":"2024-05-09T05:24:06.917091+0000","last_peered":"2024-05-09T05:24:06.917091+0000","last_clean":"2024-05-09T05:24:06.917091+0000","last_became_active":"2024-05-09T05:06:05.071032+0000","last_became_peered":"2024-05-09T05:06:05.071032+0000","last_unstale":"2024-05-09T05:24:06.917091+0000","last_undegraded":"2024-05-09T05:24:06.917091+0000","last_fullsized":"2024-05-09T05:24:06.917091+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5546","last_scrub_stamp":"2024-05-09T05:24:06.916965+0000","last_deep_scrub":"23'5546","last_deep_scrub_stamp":"2024-05-09T05:24:06.916965+0000","last_clean_scrub_stamp":"2024-05-09T05:24:06.916965+0000","objects_scrubbed":0,"log_size":5546,"log_dups_size":0,"ondisk_log_size":5546,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:19:39.427885+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27252,"num_write_kb":314086,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7],"acting":[5,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.16","version":"23'8","reported_seq":42,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:53.858205+0000","last_change":"2024-05-09T05:23:53.858289+0000","last_active":"2024-05-09T05:23:53.858205+0000","last_peered":"2024-05-09T05:23:53.858205+0000","last_clean":"2024-05-09T05:23:53.858205+0000","last_became_active":"2024-05-09T05:06:04.055568+0000","last_became_peered":"2024-05-09T05:06:04.055568+0000","last_unstale":"2024-05-09T05:23:53.858205+0000","last_undegraded":"2024-05-09T05:23:53.858205+0000","last_fullsized":"2024-05-09T05:23:53.858205+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'8","last_scrub_stamp":"2024-05-09T05:23:53.858086+0000","last_deep_scrub":"23'8","last_deep_scrub_stamp":"2024-05-09T05:23:53.858086+0000","last_clean_scrub_stamp":"2024-05-09T05:23:53.858086+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:37:36.652958+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6],"acting":[5,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.17","version":"23'5049","reported_seq":5081,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:56.799904+0000","last_change":"2024-05-09T05:23:56.799982+0000","last_active":"2024-05-09T05:23:56.799904+0000","last_peered":"2024-05-09T05:23:56.799904+0000","last_clean":"2024-05-09T05:23:56.799904+0000","last_became_active":"2024-05-09T05:06:05.058612+0000","last_became_peered":"2024-05-09T05:06:05.058612+0000","last_unstale":"2024-05-09T05:23:56.799904+0000","last_undegraded":"2024-05-09T05:23:56.799904+0000","last_fullsized":"2024-05-09T05:23:56.799904+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5049","last_scrub_stamp":"2024-05-09T05:23:56.799775+0000","last_deep_scrub":"23'5049","last_deep_scrub_stamp":"2024-05-09T05:23:56.799775+0000","last_clean_scrub_stamp":"2024-05-09T05:23:56.799775+0000","objects_scrubbed":0,"log_size":5049,"log_dups_size":0,"ondisk_log_size":5049,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:33:20.063351+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25089,"num_write_kb":276787,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"23'29","reported_seq":63,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:56.115962+0000","last_change":"2024-05-09T05:23:56.116030+0000","last_active":"2024-05-09T05:23:56.115962+0000","last_peered":"2024-05-09T05:23:56.115962+0000","last_clean":"2024-05-09T05:23:56.115962+0000","last_became_active":"2024-05-09T05:06:04.059744+0000","last_became_peered":"2024-05-09T05:06:04.059744+0000","last_unstale":"2024-05-09T05:23:56.115962+0000","last_undegraded":"2024-05-09T05:23:56.115962+0000","last_fullsized":"2024-05-09T05:23:56.115962+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'29","last_scrub_stamp":"2024-05-09T05:23:56.115803+0000","last_deep_scrub":"23'29","last_deep_scrub_stamp":"2024-05-09T05:23:56.115803+0000","last_clean_scrub_stamp":"2024-05-09T05:23:56.115803+0000","objects_scrubbed":4,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:02:21.955753+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25,"num_write_kb":26,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.14","version":"23'7381","reported_seq":7413,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:58.259881+0000","last_change":"2024-05-09T05:23:58.259930+0000","last_active":"2024-05-09T05:23:58.259881+0000","last_peered":"2024-05-09T05:23:58.259881+0000","last_clean":"2024-05-09T05:23:58.259881+0000","last_became_active":"2024-05-09T05:06:05.066593+0000","last_became_peered":"2024-05-09T05:06:05.066593+0000","last_unstale":"2024-05-09T05:23:58.259881+0000","last_undegraded":"2024-05-09T05:23:58.259881+0000","last_fullsized":"2024-05-09T05:23:58.259881+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'7381","last_scrub_stamp":"2024-05-09T05:23:58.259703+0000","last_deep_scrub":"23'7381","last_deep_scrub_stamp":"2024-05-09T05:23:58.259703+0000","last_clean_scrub_stamp":"2024-05-09T05:23:58.259703+0000","objects_scrubbed":0,"log_size":7381,"log_dups_size":0,"ondisk_log_size":7381,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:03.918266+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":36744,"num_write_kb":418024,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7],"acting":[4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.14","version":"23'338","reported_seq":372,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:54.600096+0000","last_change":"2024-05-09T05:23:54.600146+0000","last_active":"2024-05-09T05:23:54.600096+0000","last_peered":"2024-05-09T05:23:54.600096+0000","last_clean":"2024-05-09T05:23:54.600096+0000","last_became_active":"2024-05-09T05:06:04.058530+0000","last_became_peered":"2024-05-09T05:06:04.058530+0000","last_unstale":"2024-05-09T05:23:54.600096+0000","last_undegraded":"2024-05-09T05:23:54.600096+0000","last_fullsized":"2024-05-09T05:23:54.600096+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'338","last_scrub_stamp":"2024-05-09T05:23:54.599984+0000","last_deep_scrub":"23'338","last_deep_scrub_stamp":"2024-05-09T05:23:54.599984+0000","last_clean_scrub_stamp":"2024-05-09T05:23:54.599984+0000","objects_scrubbed":6,"log_size":338,"log_dups_size":0,"ondisk_log_size":338,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:00.258546+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":331,"num_write_kb":4250,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3],"acting":[6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.15","version":"23'5538","reported_seq":5570,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:55.327334+0000","last_change":"2024-05-09T05:23:55.327365+0000","last_active":"2024-05-09T05:23:55.327334+0000","last_peered":"2024-05-09T05:23:55.327334+0000","last_clean":"2024-05-09T05:23:55.327334+0000","last_became_active":"2024-05-09T05:06:05.064850+0000","last_became_peered":"2024-05-09T05:06:05.064850+0000","last_unstale":"2024-05-09T05:23:55.327334+0000","last_undegraded":"2024-05-09T05:23:55.327334+0000","last_fullsized":"2024-05-09T05:23:55.327334+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5538","last_scrub_stamp":"2024-05-09T05:23:55.327222+0000","last_deep_scrub":"23'5538","last_deep_scrub_stamp":"2024-05-09T05:23:55.327222+0000","last_clean_scrub_stamp":"2024-05-09T05:23:55.327222+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:02:12.987926+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27557,"num_write_kb":302974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3],"acting":[7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.13","version":"23'21","reported_seq":55,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:51.849333+0000","last_change":"2024-05-09T05:23:51.849368+0000","last_active":"2024-05-09T05:23:51.849333+0000","last_peered":"2024-05-09T05:23:51.849333+0000","last_clean":"2024-05-09T05:23:51.849333+0000","last_became_active":"2024-05-09T05:06:04.057249+0000","last_became_peered":"2024-05-09T05:06:04.057249+0000","last_unstale":"2024-05-09T05:23:51.849333+0000","last_undegraded":"2024-05-09T05:23:51.849333+0000","last_fullsized":"2024-05-09T05:23:51.849333+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'21","last_scrub_stamp":"2024-05-09T05:23:51.849249+0000","last_deep_scrub":"23'21","last_deep_scrub_stamp":"2024-05-09T05:23:51.849249+0000","last_clean_scrub_stamp":"2024-05-09T05:23:51.849249+0000","objects_scrubbed":6,"log_size":21,"log_dups_size":0,"ondisk_log_size":21,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:45:41.405224+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.12","version":"23'5579","reported_seq":5611,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:02.758907+0000","last_change":"2024-05-09T05:24:02.759008+0000","last_active":"2024-05-09T05:24:02.758907+0000","last_peered":"2024-05-09T05:24:02.758907+0000","last_clean":"2024-05-09T05:24:02.758907+0000","last_became_active":"2024-05-09T05:06:05.068877+0000","last_became_peered":"2024-05-09T05:06:05.068877+0000","last_unstale":"2024-05-09T05:24:02.758907+0000","last_undegraded":"2024-05-09T05:24:02.758907+0000","last_fullsized":"2024-05-09T05:24:02.758907+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5579","last_scrub_stamp":"2024-05-09T05:24:02.758759+0000","last_deep_scrub":"23'5579","last_deep_scrub_stamp":"2024-05-09T05:24:02.758759+0000","last_clean_scrub_stamp":"2024-05-09T05:24:02.758759+0000","objects_scrubbed":0,"log_size":5579,"log_dups_size":0,"ondisk_log_size":5579,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:13:09.437691+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27823,"num_write_kb":303973,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.12","version":"23'13","reported_seq":47,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:57.834598+0000","last_change":"2024-05-09T05:23:57.834637+0000","last_active":"2024-05-09T05:23:57.834598+0000","last_peered":"2024-05-09T05:23:57.834598+0000","last_clean":"2024-05-09T05:23:57.834598+0000","last_became_active":"2024-05-09T05:06:04.061870+0000","last_became_peered":"2024-05-09T05:06:04.061870+0000","last_unstale":"2024-05-09T05:23:57.834598+0000","last_undegraded":"2024-05-09T05:23:57.834598+0000","last_fullsized":"2024-05-09T05:23:57.834598+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'13","last_scrub_stamp":"2024-05-09T05:23:57.834491+0000","last_deep_scrub":"23'13","last_deep_scrub_stamp":"2024-05-09T05:23:57.834491+0000","last_clean_scrub_stamp":"2024-05-09T05:23:57.834491+0000","objects_scrubbed":3,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:38.461519+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":12,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"23'6303","reported_seq":6335,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:06.307859+0000","last_change":"2024-05-09T05:24:06.307893+0000","last_active":"2024-05-09T05:24:06.307859+0000","last_peered":"2024-05-09T05:24:06.307859+0000","last_clean":"2024-05-09T05:24:06.307859+0000","last_became_active":"2024-05-09T05:06:05.066676+0000","last_became_peered":"2024-05-09T05:06:05.066676+0000","last_unstale":"2024-05-09T05:24:06.307859+0000","last_undegraded":"2024-05-09T05:24:06.307859+0000","last_fullsized":"2024-05-09T05:24:06.307859+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6303","last_scrub_stamp":"2024-05-09T05:24:06.307711+0000","last_deep_scrub":"23'6303","last_deep_scrub_stamp":"2024-05-09T05:24:06.307711+0000","last_clean_scrub_stamp":"2024-05-09T05:24:06.307711+0000","objects_scrubbed":0,"log_size":6303,"log_dups_size":0,"ondisk_log_size":6303,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:35:04.043662+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":30554,"num_write_kb":351862,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"23'15","reported_seq":49,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:56.635438+0000","last_change":"2024-05-09T05:23:56.635479+0000","last_active":"2024-05-09T05:23:56.635438+0000","last_peered":"2024-05-09T05:23:56.635438+0000","last_clean":"2024-05-09T05:23:56.635438+0000","last_became_active":"2024-05-09T05:06:04.061027+0000","last_became_peered":"2024-05-09T05:06:04.061027+0000","last_unstale":"2024-05-09T05:23:56.635438+0000","last_undegraded":"2024-05-09T05:23:56.635438+0000","last_fullsized":"2024-05-09T05:23:56.635438+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'15","last_scrub_stamp":"2024-05-09T05:23:56.635340+0000","last_deep_scrub":"23'15","last_deep_scrub_stamp":"2024-05-09T05:23:56.635340+0000","last_clean_scrub_stamp":"2024-05-09T05:23:56.635340+0000","objects_scrubbed":3,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:41.856941+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1170,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":56,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.10","version":"23'5877","reported_seq":5909,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:00.661840+0000","last_change":"2024-05-09T05:24:00.661871+0000","last_active":"2024-05-09T05:24:00.661840+0000","last_peered":"2024-05-09T05:24:00.661840+0000","last_clean":"2024-05-09T05:24:00.661840+0000","last_became_active":"2024-05-09T05:06:05.060586+0000","last_became_peered":"2024-05-09T05:06:05.060586+0000","last_unstale":"2024-05-09T05:24:00.661840+0000","last_undegraded":"2024-05-09T05:24:00.661840+0000","last_fullsized":"2024-05-09T05:24:00.661840+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5877","last_scrub_stamp":"2024-05-09T05:24:00.661708+0000","last_deep_scrub":"23'5877","last_deep_scrub_stamp":"2024-05-09T05:24:00.661708+0000","last_clean_scrub_stamp":"2024-05-09T05:24:00.661708+0000","objects_scrubbed":0,"log_size":5877,"log_dups_size":0,"ondisk_log_size":5877,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:14:35.984690+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29217,"num_write_kb":334206,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0],"acting":[6,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"23'29","reported_seq":63,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:52.684826+0000","last_change":"2024-05-09T05:23:52.684906+0000","last_active":"2024-05-09T05:23:52.684826+0000","last_peered":"2024-05-09T05:23:52.684826+0000","last_clean":"2024-05-09T05:23:52.684826+0000","last_became_active":"2024-05-09T05:06:04.050636+0000","last_became_peered":"2024-05-09T05:06:04.050636+0000","last_unstale":"2024-05-09T05:23:52.684826+0000","last_undegraded":"2024-05-09T05:23:52.684826+0000","last_fullsized":"2024-05-09T05:23:52.684826+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'29","last_scrub_stamp":"2024-05-09T05:23:52.684732+0000","last_deep_scrub":"23'29","last_deep_scrub_stamp":"2024-05-09T05:23:52.684732+0000","last_clean_scrub_stamp":"2024-05-09T05:23:52.684732+0000","objects_scrubbed":8,"log_size":29,"log_dups_size":0,"ondisk_log_size":29,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:00:23.482496+0000","scrub_duration":5,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.11","version":"23'5443","reported_seq":5475,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:07.276921+0000","last_change":"2024-05-09T05:24:07.276964+0000","last_active":"2024-05-09T05:24:07.276921+0000","last_peered":"2024-05-09T05:24:07.276921+0000","last_clean":"2024-05-09T05:24:07.276921+0000","last_became_active":"2024-05-09T05:06:05.068540+0000","last_became_peered":"2024-05-09T05:06:05.068540+0000","last_unstale":"2024-05-09T05:24:07.276921+0000","last_undegraded":"2024-05-09T05:24:07.276921+0000","last_fullsized":"2024-05-09T05:24:07.276921+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5443","last_scrub_stamp":"2024-05-09T05:24:07.276802+0000","last_deep_scrub":"23'5443","last_deep_scrub_stamp":"2024-05-09T05:24:07.276802+0000","last_clean_scrub_stamp":"2024-05-09T05:24:07.276802+0000","objects_scrubbed":0,"log_size":5443,"log_dups_size":0,"ondisk_log_size":5443,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:27:32.706622+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27739,"num_write_kb":311936,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.f","version":"23'8","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:53.328737+0000","last_change":"2024-05-09T05:23:53.328737+0000","last_active":"2024-05-09T05:23:53.328737+0000","last_peered":"2024-05-09T05:23:53.328737+0000","last_clean":"2024-05-09T05:23:53.328737+0000","last_became_active":"2024-05-09T05:06:04.060554+0000","last_became_peered":"2024-05-09T05:06:04.060554+0000","last_unstale":"2024-05-09T05:23:53.328737+0000","last_undegraded":"2024-05-09T05:23:53.328737+0000","last_fullsized":"2024-05-09T05:23:53.328737+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'8","last_scrub_stamp":"2024-05-09T05:23:53.328196+0000","last_deep_scrub":"23'8","last_deep_scrub_stamp":"2024-05-09T05:23:53.328196+0000","last_clean_scrub_stamp":"2024-05-09T05:23:53.328196+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:46:19.126082+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.e","version":"23'5295","reported_seq":5328,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:56.296637+0000","last_change":"2024-05-09T05:23:56.296637+0000","last_active":"2024-05-09T05:23:56.296637+0000","last_peered":"2024-05-09T05:23:56.296637+0000","last_clean":"2024-05-09T05:23:56.296637+0000","last_became_active":"2024-05-09T05:06:05.063829+0000","last_became_peered":"2024-05-09T05:06:05.063829+0000","last_unstale":"2024-05-09T05:23:56.296637+0000","last_undegraded":"2024-05-09T05:23:56.296637+0000","last_fullsized":"2024-05-09T05:23:56.296637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5295","last_scrub_stamp":"2024-05-09T05:23:56.296500+0000","last_deep_scrub":"23'5295","last_deep_scrub_stamp":"2024-05-09T05:23:56.296500+0000","last_clean_scrub_stamp":"2024-05-09T05:23:56.296500+0000","objects_scrubbed":0,"log_size":5295,"log_dups_size":0,"ondisk_log_size":5295,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:33:19.560076+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25732,"num_write_kb":299773,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"23'11","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:55.711211+0000","last_change":"2024-05-09T05:23:55.711211+0000","last_active":"2024-05-09T05:23:55.711211+0000","last_peered":"2024-05-09T05:23:55.711211+0000","last_clean":"2024-05-09T05:23:55.711211+0000","last_became_active":"2024-05-09T05:06:04.058575+0000","last_became_peered":"2024-05-09T05:06:04.058575+0000","last_unstale":"2024-05-09T05:23:55.711211+0000","last_undegraded":"2024-05-09T05:23:55.711211+0000","last_fullsized":"2024-05-09T05:23:55.711211+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'11","last_scrub_stamp":"2024-05-09T05:23:55.711018+0000","last_deep_scrub":"23'11","last_deep_scrub_stamp":"2024-05-09T05:23:55.711018+0000","last_clean_scrub_stamp":"2024-05-09T05:23:55.711018+0000","objects_scrubbed":2,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:31:00.076883+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.f","version":"23'6402","reported_seq":6435,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:11.304359+0000","last_change":"2024-05-09T05:24:11.304359+0000","last_active":"2024-05-09T05:24:11.304359+0000","last_peered":"2024-05-09T05:24:11.304359+0000","last_clean":"2024-05-09T05:24:11.304359+0000","last_became_active":"2024-05-09T05:06:05.069954+0000","last_became_peered":"2024-05-09T05:06:05.069954+0000","last_unstale":"2024-05-09T05:24:11.304359+0000","last_undegraded":"2024-05-09T05:24:11.304359+0000","last_fullsized":"2024-05-09T05:24:11.304359+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6402","last_scrub_stamp":"2024-05-09T05:24:11.304184+0000","last_deep_scrub":"23'6402","last_deep_scrub_stamp":"2024-05-09T05:24:11.304184+0000","last_clean_scrub_stamp":"2024-05-09T05:24:11.304184+0000","objects_scrubbed":0,"log_size":6402,"log_dups_size":0,"ondisk_log_size":6402,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:50.016685+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":31156,"num_write_kb":362485,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"23'5","reported_seq":40,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:53.120848+0000","last_change":"2024-05-09T05:23:53.120848+0000","last_active":"2024-05-09T05:23:53.120848+0000","last_peered":"2024-05-09T05:23:53.120848+0000","last_clean":"2024-05-09T05:23:53.120848+0000","last_became_active":"2024-05-09T05:06:04.056484+0000","last_became_peered":"2024-05-09T05:06:04.056484+0000","last_unstale":"2024-05-09T05:23:53.120848+0000","last_undegraded":"2024-05-09T05:23:53.120848+0000","last_fullsized":"2024-05-09T05:23:53.120848+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5","last_scrub_stamp":"2024-05-09T05:23:53.120682+0000","last_deep_scrub":"23'5","last_deep_scrub_stamp":"2024-05-09T05:23:53.120682+0000","last_clean_scrub_stamp":"2024-05-09T05:23:53.120682+0000","objects_scrubbed":1,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:31:43.871080+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,4],"acting":[1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.c","version":"23'5559","reported_seq":5592,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:01.857664+0000","last_change":"2024-05-09T05:24:01.857664+0000","last_active":"2024-05-09T05:24:01.857664+0000","last_peered":"2024-05-09T05:24:01.857664+0000","last_clean":"2024-05-09T05:24:01.857664+0000","last_became_active":"2024-05-09T05:06:05.065651+0000","last_became_peered":"2024-05-09T05:06:05.065651+0000","last_unstale":"2024-05-09T05:24:01.857664+0000","last_undegraded":"2024-05-09T05:24:01.857664+0000","last_fullsized":"2024-05-09T05:24:01.857664+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5559","last_scrub_stamp":"2024-05-09T05:24:01.857522+0000","last_deep_scrub":"23'5559","last_deep_scrub_stamp":"2024-05-09T05:24:01.857522+0000","last_clean_scrub_stamp":"2024-05-09T05:24:01.857522+0000","objects_scrubbed":0,"log_size":5559,"log_dups_size":0,"ondisk_log_size":5559,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:08.750794+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28130,"num_write_kb":325652,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.c","version":"23'12","reported_seq":47,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:53.704332+0000","last_change":"2024-05-09T05:23:53.704332+0000","last_active":"2024-05-09T05:23:53.704332+0000","last_peered":"2024-05-09T05:23:53.704332+0000","last_clean":"2024-05-09T05:23:53.704332+0000","last_became_active":"2024-05-09T05:06:04.055029+0000","last_became_peered":"2024-05-09T05:06:04.055029+0000","last_unstale":"2024-05-09T05:23:53.704332+0000","last_undegraded":"2024-05-09T05:23:53.704332+0000","last_fullsized":"2024-05-09T05:23:53.704332+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'12","last_scrub_stamp":"2024-05-09T05:23:53.704184+0000","last_deep_scrub":"23'12","last_deep_scrub_stamp":"2024-05-09T05:23:53.704184+0000","last_clean_scrub_stamp":"2024-05-09T05:23:53.704184+0000","objects_scrubbed":3,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:19.596258+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1752,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5],"acting":[2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"23'5794","reported_seq":5827,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:12.318596+0000","last_change":"2024-05-09T05:24:12.318630+0000","last_active":"2024-05-09T05:24:12.318596+0000","last_peered":"2024-05-09T05:24:12.318596+0000","last_clean":"2024-05-09T05:24:12.318596+0000","last_became_active":"2024-05-09T05:06:05.070558+0000","last_became_peered":"2024-05-09T05:06:05.070558+0000","last_unstale":"2024-05-09T05:24:12.318596+0000","last_undegraded":"2024-05-09T05:24:12.318596+0000","last_fullsized":"2024-05-09T05:24:12.318596+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5794","last_scrub_stamp":"2024-05-09T05:24:12.318471+0000","last_deep_scrub":"23'5794","last_deep_scrub_stamp":"2024-05-09T05:24:12.318471+0000","last_clean_scrub_stamp":"2024-05-09T05:24:12.318471+0000","objects_scrubbed":0,"log_size":5794,"log_dups_size":0,"ondisk_log_size":5794,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:06:44.121650+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28847,"num_write_kb":320571,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"1.0","version":"23'74","reported_seq":134,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:57.333074+0000","last_change":"2024-05-09T05:23:57.333074+0000","last_active":"2024-05-09T05:23:57.333074+0000","last_peered":"2024-05-09T05:23:57.333074+0000","last_clean":"2024-05-09T05:23:57.333074+0000","last_became_active":"2024-05-09T05:05:53.329257+0000","last_became_peered":"2024-05-09T05:05:53.329257+0000","last_unstale":"2024-05-09T05:23:57.333074+0000","last_undegraded":"2024-05-09T05:23:57.333074+0000","last_fullsized":"2024-05-09T05:23:57.333074+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":15,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'74","last_scrub_stamp":"2024-05-09T05:23:57.332931+0000","last_deep_scrub":"23'74","last_deep_scrub_stamp":"2024-05-09T05:23:57.332931+0000","last_clean_scrub_stamp":"2024-05-09T05:23:57.332931+0000","objects_scrubbed":2,"log_size":74,"log_dups_size":0,"ondisk_log_size":74,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:28:52.004788+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"23'12","reported_seq":47,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:56.866702+0000","last_change":"2024-05-09T05:23:56.866753+0000","last_active":"2024-05-09T05:23:56.866702+0000","last_peered":"2024-05-09T05:23:56.866702+0000","last_clean":"2024-05-09T05:23:56.866702+0000","last_became_active":"2024-05-09T05:06:04.053479+0000","last_became_peered":"2024-05-09T05:06:04.053479+0000","last_unstale":"2024-05-09T05:23:56.866702+0000","last_undegraded":"2024-05-09T05:23:56.866702+0000","last_fullsized":"2024-05-09T05:23:56.866702+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'12","last_scrub_stamp":"2024-05-09T05:23:56.866606+0000","last_deep_scrub":"23'12","last_deep_scrub_stamp":"2024-05-09T05:23:56.866606+0000","last_clean_scrub_stamp":"2024-05-09T05:23:56.866606+0000","objects_scrubbed":5,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:28:51.538464+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"23'5510","reported_seq":5543,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:58.200250+0000","last_change":"2024-05-09T05:23:58.200284+0000","last_active":"2024-05-09T05:23:58.200250+0000","last_peered":"2024-05-09T05:23:58.200250+0000","last_clean":"2024-05-09T05:23:58.200250+0000","last_became_active":"2024-05-09T05:06:05.065619+0000","last_became_peered":"2024-05-09T05:06:05.065619+0000","last_unstale":"2024-05-09T05:23:58.200250+0000","last_undegraded":"2024-05-09T05:23:58.200250+0000","last_fullsized":"2024-05-09T05:23:58.200250+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5510","last_scrub_stamp":"2024-05-09T05:23:58.200114+0000","last_deep_scrub":"23'5510","last_deep_scrub_stamp":"2024-05-09T05:23:58.200114+0000","last_clean_scrub_stamp":"2024-05-09T05:23:58.200114+0000","objects_scrubbed":0,"log_size":5510,"log_dups_size":0,"ondisk_log_size":5510,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:02.863318+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27376,"num_write_kb":322728,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.0","version":"23'10","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:59.338593+0000","last_change":"2024-05-09T05:23:59.338593+0000","last_active":"2024-05-09T05:23:59.338593+0000","last_peered":"2024-05-09T05:23:59.338593+0000","last_clean":"2024-05-09T05:23:59.338593+0000","last_became_active":"2024-05-09T05:06:04.068139+0000","last_became_peered":"2024-05-09T05:06:04.068139+0000","last_unstale":"2024-05-09T05:23:59.338593+0000","last_undegraded":"2024-05-09T05:23:59.338593+0000","last_fullsized":"2024-05-09T05:23:59.338593+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'10","last_scrub_stamp":"2024-05-09T05:23:59.338440+0000","last_deep_scrub":"23'10","last_deep_scrub_stamp":"2024-05-09T05:23:59.338440+0000","last_clean_scrub_stamp":"2024-05-09T05:23:59.338440+0000","objects_scrubbed":1,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:33:52.429138+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"23'5383","reported_seq":5416,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:59.779116+0000","last_change":"2024-05-09T05:23:59.779148+0000","last_active":"2024-05-09T05:23:59.779116+0000","last_peered":"2024-05-09T05:23:59.779116+0000","last_clean":"2024-05-09T05:23:59.779116+0000","last_became_active":"2024-05-09T05:06:05.061265+0000","last_became_peered":"2024-05-09T05:06:05.061265+0000","last_unstale":"2024-05-09T05:23:59.779116+0000","last_undegraded":"2024-05-09T05:23:59.779116+0000","last_fullsized":"2024-05-09T05:23:59.779116+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5383","last_scrub_stamp":"2024-05-09T05:23:59.779005+0000","last_deep_scrub":"23'5383","last_deep_scrub_stamp":"2024-05-09T05:23:59.779005+0000","last_clean_scrub_stamp":"2024-05-09T05:23:59.779005+0000","objects_scrubbed":0,"log_size":5383,"log_dups_size":0,"ondisk_log_size":5383,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:23:11.364019+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27075,"num_write_kb":305200,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"23'48","reported_seq":119,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:54.680134+0000","last_change":"2024-05-09T05:23:54.680195+0000","last_active":"2024-05-09T05:23:54.680134+0000","last_peered":"2024-05-09T05:23:54.680134+0000","last_clean":"2024-05-09T05:23:54.680134+0000","last_became_active":"2024-05-09T05:06:04.058461+0000","last_became_peered":"2024-05-09T05:06:04.058461+0000","last_unstale":"2024-05-09T05:23:54.680134+0000","last_undegraded":"2024-05-09T05:23:54.680134+0000","last_fullsized":"2024-05-09T05:23:54.680134+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'48","last_scrub_stamp":"2024-05-09T05:23:54.680000+0000","last_deep_scrub":"23'48","last_deep_scrub_stamp":"2024-05-09T05:23:54.680000+0000","last_clean_scrub_stamp":"2024-05-09T05:23:54.680000+0000","objects_scrubbed":4,"log_size":48,"log_dups_size":0,"ondisk_log_size":48,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:20:09.561192+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":88293,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":43,"num_write_kb":116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"23'5441","reported_seq":5475,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:59.070225+0000","last_change":"2024-05-09T05:23:59.070225+0000","last_active":"2024-05-09T05:23:59.070225+0000","last_peered":"2024-05-09T05:23:59.070225+0000","last_clean":"2024-05-09T05:23:59.070225+0000","last_became_active":"2024-05-09T05:06:05.055515+0000","last_became_peered":"2024-05-09T05:06:05.055515+0000","last_unstale":"2024-05-09T05:23:59.070225+0000","last_undegraded":"2024-05-09T05:23:59.070225+0000","last_fullsized":"2024-05-09T05:23:59.070225+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5441","last_scrub_stamp":"2024-05-09T05:23:59.070069+0000","last_deep_scrub":"23'5441","last_deep_scrub_stamp":"2024-05-09T05:23:59.070069+0000","last_clean_scrub_stamp":"2024-05-09T05:23:59.070069+0000","objects_scrubbed":0,"log_size":5441,"log_dups_size":0,"ondisk_log_size":5441,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:29:04.683486+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27385,"num_write_kb":302253,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"23'12","reported_seq":47,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:55.837528+0000","last_change":"2024-05-09T05:23:55.837565+0000","last_active":"2024-05-09T05:23:55.837528+0000","last_peered":"2024-05-09T05:23:55.837528+0000","last_clean":"2024-05-09T05:23:55.837528+0000","last_became_active":"2024-05-09T05:06:04.054469+0000","last_became_peered":"2024-05-09T05:06:04.054469+0000","last_unstale":"2024-05-09T05:23:55.837528+0000","last_undegraded":"2024-05-09T05:23:55.837528+0000","last_fullsized":"2024-05-09T05:23:55.837528+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'12","last_scrub_stamp":"2024-05-09T05:23:55.837425+0000","last_deep_scrub":"23'12","last_deep_scrub_stamp":"2024-05-09T05:23:55.837425+0000","last_clean_scrub_stamp":"2024-05-09T05:23:55.837425+0000","objects_scrubbed":2,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:33:19.101002+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"23'5472","reported_seq":5505,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:54.298309+0000","last_change":"2024-05-09T05:23:54.298363+0000","last_active":"2024-05-09T05:23:54.298309+0000","last_peered":"2024-05-09T05:23:54.298309+0000","last_clean":"2024-05-09T05:23:54.298309+0000","last_became_active":"2024-05-09T05:06:05.058370+0000","last_became_peered":"2024-05-09T05:06:05.058370+0000","last_unstale":"2024-05-09T05:23:54.298309+0000","last_undegraded":"2024-05-09T05:23:54.298309+0000","last_fullsized":"2024-05-09T05:23:54.298309+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5472","last_scrub_stamp":"2024-05-09T05:23:54.298161+0000","last_deep_scrub":"23'5472","last_deep_scrub_stamp":"2024-05-09T05:23:54.298161+0000","last_clean_scrub_stamp":"2024-05-09T05:23:54.298161+0000","objects_scrubbed":0,"log_size":5472,"log_dups_size":0,"ondisk_log_size":5472,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:00:25.095925+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":28049,"num_write_kb":304974,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.4","version":"23'6","reported_seq":41,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:52.102285+0000","last_change":"2024-05-09T05:23:52.102315+0000","last_active":"2024-05-09T05:23:52.102285+0000","last_peered":"2024-05-09T05:23:52.102285+0000","last_clean":"2024-05-09T05:23:52.102285+0000","last_became_active":"2024-05-09T05:06:04.051954+0000","last_became_peered":"2024-05-09T05:06:04.051954+0000","last_unstale":"2024-05-09T05:23:52.102285+0000","last_undegraded":"2024-05-09T05:23:52.102285+0000","last_fullsized":"2024-05-09T05:23:52.102285+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-05-09T05:23:52.102163+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-05-09T05:23:52.102163+0000","last_clean_scrub_stamp":"2024-05-09T05:23:52.102163+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:56.468028+0000","scrub_duration":1,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"23'4678","reported_seq":4711,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:00.859637+0000","last_change":"2024-05-09T05:24:00.859678+0000","last_active":"2024-05-09T05:24:00.859637+0000","last_peered":"2024-05-09T05:24:00.859637+0000","last_clean":"2024-05-09T05:24:00.859637+0000","last_became_active":"2024-05-09T05:06:05.064196+0000","last_became_peered":"2024-05-09T05:06:05.064196+0000","last_unstale":"2024-05-09T05:24:00.859637+0000","last_undegraded":"2024-05-09T05:24:00.859637+0000","last_fullsized":"2024-05-09T05:24:00.859637+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4678","last_scrub_stamp":"2024-05-09T05:24:00.859507+0000","last_deep_scrub":"23'4678","last_deep_scrub_stamp":"2024-05-09T05:24:00.859507+0000","last_clean_scrub_stamp":"2024-05-09T05:24:00.859507+0000","objects_scrubbed":0,"log_size":4678,"log_dups_size":0,"ondisk_log_size":4678,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:29:36.190946+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":23621,"num_write_kb":274908,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.5","version":"23'11","reported_seq":46,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:58.378995+0000","last_change":"2024-05-09T05:23:58.379040+0000","last_active":"2024-05-09T05:23:58.378995+0000","last_peered":"2024-05-09T05:23:58.378995+0000","last_clean":"2024-05-09T05:23:58.378995+0000","last_became_active":"2024-05-09T05:06:04.068132+0000","last_became_peered":"2024-05-09T05:06:04.068132+0000","last_unstale":"2024-05-09T05:23:58.378995+0000","last_undegraded":"2024-05-09T05:23:58.378995+0000","last_fullsized":"2024-05-09T05:23:58.378995+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'11","last_scrub_stamp":"2024-05-09T05:23:58.378880+0000","last_deep_scrub":"23'11","last_deep_scrub_stamp":"2024-05-09T05:23:58.378880+0000","last_clean_scrub_stamp":"2024-05-09T05:23:58.378880+0000","objects_scrubbed":3,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:39.005908+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.4","version":"23'5941","reported_seq":5974,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:58.081827+0000","last_change":"2024-05-09T05:23:58.081861+0000","last_active":"2024-05-09T05:23:58.081827+0000","last_peered":"2024-05-09T05:23:58.081827+0000","last_clean":"2024-05-09T05:23:58.081827+0000","last_became_active":"2024-05-09T05:06:05.054726+0000","last_became_peered":"2024-05-09T05:06:05.054726+0000","last_unstale":"2024-05-09T05:23:58.081827+0000","last_undegraded":"2024-05-09T05:23:58.081827+0000","last_fullsized":"2024-05-09T05:23:58.081827+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5941","last_scrub_stamp":"2024-05-09T05:23:58.081643+0000","last_deep_scrub":"23'5941","last_deep_scrub_stamp":"2024-05-09T05:23:58.081643+0000","last_clean_scrub_stamp":"2024-05-09T05:23:58.081643+0000","objects_scrubbed":0,"log_size":5941,"log_dups_size":0,"ondisk_log_size":5941,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:26:44.397215+0000","scrub_duration":16,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29751,"num_write_kb":328170,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"23'613","reported_seq":648,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:55.069405+0000","last_change":"2024-05-09T05:23:55.069438+0000","last_active":"2024-05-09T05:23:55.069405+0000","last_peered":"2024-05-09T05:23:55.069405+0000","last_clean":"2024-05-09T05:23:55.069405+0000","last_became_active":"2024-05-09T05:06:04.059154+0000","last_became_peered":"2024-05-09T05:06:04.059154+0000","last_unstale":"2024-05-09T05:23:55.069405+0000","last_undegraded":"2024-05-09T05:23:55.069405+0000","last_fullsized":"2024-05-09T05:23:55.069405+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'613","last_scrub_stamp":"2024-05-09T05:23:55.069322+0000","last_deep_scrub":"23'613","last_deep_scrub_stamp":"2024-05-09T05:23:55.069322+0000","last_clean_scrub_stamp":"2024-05-09T05:23:55.069322+0000","objects_scrubbed":4,"log_size":613,"log_dups_size":0,"ondisk_log_size":613,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:07:13.045041+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1460,"num_write_kb":471,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.7","version":"23'5503","reported_seq":5536,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:04.142136+0000","last_change":"2024-05-09T05:24:04.142169+0000","last_active":"2024-05-09T05:24:04.142136+0000","last_peered":"2024-05-09T05:24:04.142136+0000","last_clean":"2024-05-09T05:24:04.142136+0000","last_became_active":"2024-05-09T05:06:05.070940+0000","last_became_peered":"2024-05-09T05:06:05.070940+0000","last_unstale":"2024-05-09T05:24:04.142136+0000","last_undegraded":"2024-05-09T05:24:04.142136+0000","last_fullsized":"2024-05-09T05:24:04.142136+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5503","last_scrub_stamp":"2024-05-09T05:24:04.142015+0000","last_deep_scrub":"23'5503","last_deep_scrub_stamp":"2024-05-09T05:24:04.142015+0000","last_clean_scrub_stamp":"2024-05-09T05:24:04.142015+0000","objects_scrubbed":0,"log_size":5503,"log_dups_size":0,"ondisk_log_size":5503,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:42.092290+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":26876,"num_write_kb":308483,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"23'14","reported_seq":49,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:58.681767+0000","last_change":"2024-05-09T05:23:58.681842+0000","last_active":"2024-05-09T05:23:58.681767+0000","last_peered":"2024-05-09T05:23:58.681767+0000","last_clean":"2024-05-09T05:23:58.681767+0000","last_became_active":"2024-05-09T05:06:04.065703+0000","last_became_peered":"2024-05-09T05:06:04.065703+0000","last_unstale":"2024-05-09T05:23:58.681767+0000","last_undegraded":"2024-05-09T05:23:58.681767+0000","last_fullsized":"2024-05-09T05:23:58.681767+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'14","last_scrub_stamp":"2024-05-09T05:23:58.681651+0000","last_deep_scrub":"23'14","last_deep_scrub_stamp":"2024-05-09T05:23:58.681651+0000","last_clean_scrub_stamp":"2024-05-09T05:23:58.681651+0000","objects_scrubbed":4,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:06.699700+0000","scrub_duration":4,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.6","version":"23'4642","reported_seq":4675,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:54.794473+0000","last_change":"2024-05-09T05:23:54.794497+0000","last_active":"2024-05-09T05:23:54.794473+0000","last_peered":"2024-05-09T05:23:54.794473+0000","last_clean":"2024-05-09T05:23:54.794473+0000","last_became_active":"2024-05-09T05:06:05.056117+0000","last_became_peered":"2024-05-09T05:06:05.056117+0000","last_unstale":"2024-05-09T05:23:54.794473+0000","last_undegraded":"2024-05-09T05:23:54.794473+0000","last_fullsized":"2024-05-09T05:23:54.794473+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4642","last_scrub_stamp":"2024-05-09T05:23:54.794382+0000","last_deep_scrub":"23'4642","last_deep_scrub_stamp":"2024-05-09T05:23:54.794382+0000","last_clean_scrub_stamp":"2024-05-09T05:23:54.794382+0000","objects_scrubbed":0,"log_size":4642,"log_dups_size":0,"ondisk_log_size":4642,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:16:22.488252+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22705,"num_write_kb":262889,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"23'6","reported_seq":41,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:01.311745+0000","last_change":"2024-05-09T05:24:01.311804+0000","last_active":"2024-05-09T05:24:01.311745+0000","last_peered":"2024-05-09T05:24:01.311745+0000","last_clean":"2024-05-09T05:24:01.311745+0000","last_became_active":"2024-05-09T05:06:04.068898+0000","last_became_peered":"2024-05-09T05:06:04.068898+0000","last_unstale":"2024-05-09T05:24:01.311745+0000","last_undegraded":"2024-05-09T05:24:01.311745+0000","last_fullsized":"2024-05-09T05:24:01.311745+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6","last_scrub_stamp":"2024-05-09T05:24:01.311593+0000","last_deep_scrub":"23'6","last_deep_scrub_stamp":"2024-05-09T05:24:01.311593+0000","last_clean_scrub_stamp":"2024-05-09T05:24:01.311593+0000","objects_scrubbed":1,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:57:50.073632+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.9","version":"23'5215","reported_seq":5248,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:56.289142+0000","last_change":"2024-05-09T05:23:56.289178+0000","last_active":"2024-05-09T05:23:56.289142+0000","last_peered":"2024-05-09T05:23:56.289142+0000","last_clean":"2024-05-09T05:23:56.289142+0000","last_became_active":"2024-05-09T05:06:05.060475+0000","last_became_peered":"2024-05-09T05:06:05.060475+0000","last_unstale":"2024-05-09T05:23:56.289142+0000","last_undegraded":"2024-05-09T05:23:56.289142+0000","last_fullsized":"2024-05-09T05:23:56.289142+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5215","last_scrub_stamp":"2024-05-09T05:23:56.289009+0000","last_deep_scrub":"23'5215","last_deep_scrub_stamp":"2024-05-09T05:23:56.289009+0000","last_clean_scrub_stamp":"2024-05-09T05:23:56.289009+0000","objects_scrubbed":0,"log_size":5215,"log_dups_size":0,"ondisk_log_size":5215,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T05:34:54.158806+0000","scrub_duration":10,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":25185,"num_write_kb":293431,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2],"acting":[4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.9","version":"23'14","reported_seq":49,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:57.111361+0000","last_change":"2024-05-09T05:23:57.111414+0000","last_active":"2024-05-09T05:23:57.111361+0000","last_peered":"2024-05-09T05:23:57.111361+0000","last_clean":"2024-05-09T05:23:57.111361+0000","last_became_active":"2024-05-09T05:06:04.065506+0000","last_became_peered":"2024-05-09T05:06:04.065506+0000","last_unstale":"2024-05-09T05:23:57.111361+0000","last_undegraded":"2024-05-09T05:23:57.111361+0000","last_fullsized":"2024-05-09T05:23:57.111361+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'14","last_scrub_stamp":"2024-05-09T05:23:57.111267+0000","last_deep_scrub":"23'14","last_deep_scrub_stamp":"2024-05-09T05:23:57.111267+0000","last_clean_scrub_stamp":"2024-05-09T05:23:57.111267+0000","objects_scrubbed":4,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:01.774472+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7],"acting":[1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.8","version":"23'5538","reported_seq":5571,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:59.178588+0000","last_change":"2024-05-09T05:23:59.178620+0000","last_active":"2024-05-09T05:23:59.178588+0000","last_peered":"2024-05-09T05:23:59.178588+0000","last_clean":"2024-05-09T05:23:59.178588+0000","last_became_active":"2024-05-09T05:06:05.065472+0000","last_became_peered":"2024-05-09T05:06:05.065472+0000","last_unstale":"2024-05-09T05:23:59.178588+0000","last_undegraded":"2024-05-09T05:23:59.178588+0000","last_fullsized":"2024-05-09T05:23:59.178588+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5538","last_scrub_stamp":"2024-05-09T05:23:59.178474+0000","last_deep_scrub":"23'5538","last_deep_scrub_stamp":"2024-05-09T05:23:59.178474+0000","last_clean_scrub_stamp":"2024-05-09T05:23:59.178474+0000","objects_scrubbed":0,"log_size":5538,"log_dups_size":0,"ondisk_log_size":5538,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:26:45.494045+0000","scrub_duration":11,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":27899,"num_write_kb":316727,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,1],"acting":[3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.a","version":"23'8","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:54.111789+0000","last_change":"2024-05-09T05:23:54.111845+0000","last_active":"2024-05-09T05:23:54.111789+0000","last_peered":"2024-05-09T05:23:54.111789+0000","last_clean":"2024-05-09T05:23:54.111789+0000","last_became_active":"2024-05-09T05:06:04.059478+0000","last_became_peered":"2024-05-09T05:06:04.059478+0000","last_unstale":"2024-05-09T05:23:54.111789+0000","last_undegraded":"2024-05-09T05:23:54.111789+0000","last_fullsized":"2024-05-09T05:23:54.111789+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'8","last_scrub_stamp":"2024-05-09T05:23:54.111656+0000","last_deep_scrub":"23'8","last_deep_scrub_stamp":"2024-05-09T05:23:54.111656+0000","last_clean_scrub_stamp":"2024-05-09T05:23:54.111656+0000","objects_scrubbed":2,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T08:54:39.333257+0000","scrub_duration":2,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,3],"acting":[1,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.b","version":"23'5078","reported_seq":5111,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:05.156989+0000","last_change":"2024-05-09T05:24:05.156989+0000","last_active":"2024-05-09T05:24:05.156989+0000","last_peered":"2024-05-09T05:24:05.156989+0000","last_clean":"2024-05-09T05:24:05.156989+0000","last_became_active":"2024-05-09T05:06:05.070728+0000","last_became_peered":"2024-05-09T05:06:05.070728+0000","last_unstale":"2024-05-09T05:24:05.156989+0000","last_undegraded":"2024-05-09T05:24:05.156989+0000","last_fullsized":"2024-05-09T05:24:05.156989+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5078","last_scrub_stamp":"2024-05-09T05:24:05.156810+0000","last_deep_scrub":"23'5078","last_deep_scrub_stamp":"2024-05-09T05:24:05.156810+0000","last_clean_scrub_stamp":"2024-05-09T05:24:05.156810+0000","objects_scrubbed":0,"log_size":5078,"log_dups_size":0,"ondisk_log_size":5078,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:05:56.780474+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24494,"num_write_kb":270229,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"23'1100","reported_seq":1135,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:00.311305+0000","last_change":"2024-05-09T05:24:00.311305+0000","last_active":"2024-05-09T05:24:00.311305+0000","last_peered":"2024-05-09T05:24:00.311305+0000","last_clean":"2024-05-09T05:24:00.311305+0000","last_became_active":"2024-05-09T05:06:04.068154+0000","last_became_peered":"2024-05-09T05:06:04.068154+0000","last_unstale":"2024-05-09T05:24:00.311305+0000","last_undegraded":"2024-05-09T05:24:00.311305+0000","last_fullsized":"2024-05-09T05:24:00.311305+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'1100","last_scrub_stamp":"2024-05-09T05:24:00.311146+0000","last_deep_scrub":"23'1100","last_deep_scrub_stamp":"2024-05-09T05:24:00.311146+0000","last_clean_scrub_stamp":"2024-05-09T05:24:00.311146+0000","objects_scrubbed":3,"log_size":1100,"log_dups_size":0,"ondisk_log_size":1100,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:13:00.841027+0000","scrub_duration":15,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194304,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1089,"num_write_kb":4419,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4],"acting":[7,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.a","version":"23'4956","reported_seq":4989,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:03.647345+0000","last_change":"2024-05-09T05:24:03.647385+0000","last_active":"2024-05-09T05:24:03.647345+0000","last_peered":"2024-05-09T05:24:03.647345+0000","last_clean":"2024-05-09T05:24:03.647345+0000","last_became_active":"2024-05-09T05:06:05.062881+0000","last_became_peered":"2024-05-09T05:06:05.062881+0000","last_unstale":"2024-05-09T05:24:03.647345+0000","last_undegraded":"2024-05-09T05:24:03.647345+0000","last_fullsized":"2024-05-09T05:24:03.647345+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4956","last_scrub_stamp":"2024-05-09T05:24:03.647220+0000","last_deep_scrub":"23'4956","last_deep_scrub_stamp":"2024-05-09T05:24:03.647220+0000","last_clean_scrub_stamp":"2024-05-09T05:24:03.647220+0000","objects_scrubbed":0,"log_size":4956,"log_dups_size":0,"ondisk_log_size":4956,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:19:28.196236+0000","scrub_duration":13,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24725,"num_write_kb":283470,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.1d","version":"23'6745","reported_seq":6777,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:04.845631+0000","last_change":"2024-05-09T05:24:04.845661+0000","last_active":"2024-05-09T05:24:04.845631+0000","last_peered":"2024-05-09T05:24:04.845631+0000","last_clean":"2024-05-09T05:24:04.845631+0000","last_became_active":"2024-05-09T05:06:05.064917+0000","last_became_peered":"2024-05-09T05:06:05.064917+0000","last_unstale":"2024-05-09T05:24:04.845631+0000","last_undegraded":"2024-05-09T05:24:04.845631+0000","last_fullsized":"2024-05-09T05:24:04.845631+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6745","last_scrub_stamp":"2024-05-09T05:24:04.845512+0000","last_deep_scrub":"23'6745","last_deep_scrub_stamp":"2024-05-09T05:24:04.845512+0000","last_clean_scrub_stamp":"2024-05-09T05:24:04.845512+0000","objects_scrubbed":0,"log_size":6745,"log_dups_size":0,"ondisk_log_size":6745,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:26:39.531076+0000","scrub_duration":9,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":33228,"num_write_kb":379009,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4],"acting":[5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1c","version":"23'9","reported_seq":43,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:03.297378+0000","last_change":"2024-05-09T05:24:03.297423+0000","last_active":"2024-05-09T05:24:03.297378+0000","last_peered":"2024-05-09T05:24:03.297378+0000","last_clean":"2024-05-09T05:24:03.297378+0000","last_became_active":"2024-05-09T05:06:04.069849+0000","last_became_peered":"2024-05-09T05:06:04.069849+0000","last_unstale":"2024-05-09T05:24:03.297378+0000","last_undegraded":"2024-05-09T05:24:03.297378+0000","last_fullsized":"2024-05-09T05:24:03.297378+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'9","last_scrub_stamp":"2024-05-09T05:24:03.297278+0000","last_deep_scrub":"23'9","last_deep_scrub_stamp":"2024-05-09T05:24:03.297278+0000","last_clean_scrub_stamp":"2024-05-09T05:24:03.297278+0000","objects_scrubbed":3,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:54:55.075223+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5],"acting":[7,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1c","version":"23'4185","reported_seq":4210,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:58.699920+0000","last_change":"2024-05-09T05:23:58.699979+0000","last_active":"2024-05-09T05:23:58.699920+0000","last_peered":"2024-05-09T05:23:58.699920+0000","last_clean":"2024-05-09T05:23:58.699920+0000","last_became_active":"2024-05-09T05:07:01.722785+0000","last_became_peered":"2024-05-09T05:07:01.722785+0000","last_unstale":"2024-05-09T05:23:58.699920+0000","last_undegraded":"2024-05-09T05:23:58.699920+0000","last_fullsized":"2024-05-09T05:23:58.699920+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'4185","last_scrub_stamp":"2024-05-09T05:23:58.699766+0000","last_deep_scrub":"23'4185","last_deep_scrub_stamp":"2024-05-09T05:23:58.699766+0000","last_clean_scrub_stamp":"2024-05-09T05:23:58.699766+0000","objects_scrubbed":0,"log_size":4185,"log_dups_size":0,"ondisk_log_size":4185,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:07:16.675485+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20471,"num_write_kb":229445,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,4],"acting":[2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1d","version":"23'28","reported_seq":62,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:02.335654+0000","last_change":"2024-05-09T05:24:02.335709+0000","last_active":"2024-05-09T05:24:02.335654+0000","last_peered":"2024-05-09T05:24:02.335654+0000","last_clean":"2024-05-09T05:24:02.335654+0000","last_became_active":"2024-05-09T05:06:04.069685+0000","last_became_peered":"2024-05-09T05:06:04.069685+0000","last_unstale":"2024-05-09T05:24:02.335654+0000","last_undegraded":"2024-05-09T05:24:02.335654+0000","last_fullsized":"2024-05-09T05:24:02.335654+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'28","last_scrub_stamp":"2024-05-09T05:24:02.335553+0000","last_deep_scrub":"23'28","last_deep_scrub_stamp":"2024-05-09T05:24:02.335553+0000","last_clean_scrub_stamp":"2024-05-09T05:24:02.335553+0000","objects_scrubbed":7,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:39:09.228826+0000","scrub_duration":6,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":5000,"num_omap_keys":10,"num_objects_repaired":0},"up":[7,6],"acting":[7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1f","version":"23'6872","reported_seq":6905,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:57.757011+0000","last_change":"2024-05-09T05:23:57.757069+0000","last_active":"2024-05-09T05:23:57.757011+0000","last_peered":"2024-05-09T05:23:57.757011+0000","last_clean":"2024-05-09T05:23:57.757011+0000","last_became_active":"2024-05-09T05:06:05.058996+0000","last_became_peered":"2024-05-09T05:06:05.058996+0000","last_unstale":"2024-05-09T05:23:57.757011+0000","last_undegraded":"2024-05-09T05:23:57.757011+0000","last_fullsized":"2024-05-09T05:23:57.757011+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'6872","last_scrub_stamp":"2024-05-09T05:23:57.756873+0000","last_deep_scrub":"23'6872","last_deep_scrub_stamp":"2024-05-09T05:23:57.756873+0000","last_clean_scrub_stamp":"2024-05-09T05:23:57.756873+0000","objects_scrubbed":0,"log_size":6872,"log_dups_size":0,"ondisk_log_size":6872,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T06:41:34.233228+0000","scrub_duration":8,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":34844,"num_write_kb":385600,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1e","version":"23'9","reported_seq":45,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:54.285608+0000","last_change":"2024-05-09T05:23:54.285608+0000","last_active":"2024-05-09T05:23:54.285608+0000","last_peered":"2024-05-09T05:23:54.285608+0000","last_clean":"2024-05-09T05:23:54.285608+0000","last_became_active":"2024-05-09T05:06:04.060910+0000","last_became_peered":"2024-05-09T05:06:04.060910+0000","last_unstale":"2024-05-09T05:23:54.285608+0000","last_undegraded":"2024-05-09T05:23:54.285608+0000","last_fullsized":"2024-05-09T05:23:54.285608+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'9","last_scrub_stamp":"2024-05-09T05:23:54.285490+0000","last_deep_scrub":"23'9","last_deep_scrub_stamp":"2024-05-09T05:23:54.285490+0000","last_clean_scrub_stamp":"2024-05-09T05:23:54.285490+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:41:34.735115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0],"acting":[3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1e","version":"23'5811","reported_seq":5845,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:24:02.214395+0000","last_change":"2024-05-09T05:24:02.214395+0000","last_active":"2024-05-09T05:24:02.214395+0000","last_peered":"2024-05-09T05:24:02.214395+0000","last_clean":"2024-05-09T05:24:02.214395+0000","last_became_active":"2024-05-09T05:06:05.068686+0000","last_became_peered":"2024-05-09T05:06:05.068686+0000","last_unstale":"2024-05-09T05:24:02.214395+0000","last_undegraded":"2024-05-09T05:24:02.214395+0000","last_fullsized":"2024-05-09T05:24:02.214395+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'5811","last_scrub_stamp":"2024-05-09T05:24:02.214245+0000","last_deep_scrub":"23'5811","last_deep_scrub_stamp":"2024-05-09T05:24:02.214245+0000","last_clean_scrub_stamp":"2024-05-09T05:24:02.214245+0000","objects_scrubbed":0,"log_size":5811,"log_dups_size":0,"ondisk_log_size":5811,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T07:45:59.372457+0000","scrub_duration":12,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":29692,"num_write_kb":336226,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,6],"acting":[3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1f","version":"23'747","reported_seq":782,"reported_epoch":23,"state":"active+clean","last_fresh":"2024-05-09T05:23:52.821451+0000","last_change":"2024-05-09T05:23:52.821485+0000","last_active":"2024-05-09T05:23:52.821451+0000","last_peered":"2024-05-09T05:23:52.821451+0000","last_clean":"2024-05-09T05:23:52.821451+0000","last_became_active":"2024-05-09T05:06:04.060620+0000","last_became_peered":"2024-05-09T05:06:04.060620+0000","last_unstale":"2024-05-09T05:23:52.821451+0000","last_undegraded":"2024-05-09T05:23:52.821451+0000","last_fullsized":"2024-05-09T05:23:52.821451+0000","mapping_epoch":18,"log_start":"0'0","ondisk_log_start":"0'0","created":18,"last_epoch_clean":19,"parent":"0.0","parent_split_bits":0,"last_scrub":"23'747","last_scrub_stamp":"2024-05-09T05:23:52.821359+0000","last_deep_scrub":"23'747","last_deep_scrub_stamp":"2024-05-09T05:23:52.821359+0000","last_clean_scrub_stamp":"2024-05-09T05:23:52.821359+0000","objects_scrubbed":7,"log_size":747,"log_dups_size":0,"ondisk_log_size":747,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:12:06.801873+0000","scrub_duration":17,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":4194976,"num_objects":7,"num_object_clones":0,"num_object_copies":14,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":739,"num_write_kb":4528,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7],"acting":[0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1808567,"num_write_kb":20359177,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":362904,"ondisk_log_size":362904,"up":128,"acting":128,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":18590025,"num_objects":103,"num_object_clones":0,"num_object_copies":206,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":103,"num_whiteouts":0,"num_read":36,"num_read_kb":112,"num_write":4539,"num_write_kb":19914,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":25000,"num_omap_keys":50,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":37421056,"data_stored":37180050,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3854,"ondisk_log_size":3854,"up":64,"acting":64,"num_store_stats":8},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":66,"num_read_kb":55,"num_write":101,"num_write_kb":1628,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":74,"ondisk_log_size":74,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":7,"up_from":15,"seq":64424509662,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":729492,"kb_used_data":13132,"kb_used_omap":1,"kb_used_meta":716350,"kb_avail":93642348,"statfs":{"total":96636764160,"available":95889764352,"internally_reserved":0,"allocated":13447168,"data_stored":13261681,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":733542858},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":15,"seq":64424509663,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":702748,"kb_used_data":276,"kb_used_omap":1,"kb_used_meta":702462,"kb_avail":93669092,"statfs":{"total":96636764160,"available":95917150208,"internally_reserved":0,"allocated":282624,"data_stored":90233,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":719321541},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":14,"seq":60129542367,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":384208,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":378174,"kb_avail":93987632,"statfs":{"total":96636764160,"available":96243335168,"internally_reserved":0,"allocated":6168576,"data_stored":5997855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":387250630},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":14,"seq":60129542368,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":698768,"kb_used_data":6024,"kb_used_omap":1,"kb_used_meta":692734,"kb_avail":93673072,"statfs":{"total":96636764160,"available":95921225728,"internally_reserved":0,"allocated":6168576,"data_stored":5997723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":709360069},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":14,"seq":60129542368,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":679136,"kb_used_data":8536,"kb_used_omap":1,"kb_used_meta":670590,"kb_avail":93692704,"statfs":{"total":96636764160,"available":95941328896,"internally_reserved":0,"allocated":8740864,"data_stored":8564586,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1592,"internal_metadata":686684616},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":14,"seq":60129542369,"num_pgs":16,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":400676,"kb_used_data":348,"kb_used_omap":1,"kb_used_meta":400318,"kb_avail":93971164,"statfs":{"total":96636764160,"available":96226471936,"internally_reserved":0,"allocated":356352,"data_stored":175944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":409926085},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":14,"seq":60129542368,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":446280,"kb_used_data":256,"kb_used_omap":1,"kb_used_meta":446014,"kb_avail":93925560,"statfs":{"total":96636764160,"available":96179773440,"internally_reserved":0,"allocated":262144,"data_stored":87251,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":456718789},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":14,"seq":60129542367,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":675468,"kb_used_data":4932,"kb_used_omap":1,"kb_used_meta":670526,"kb_avail":93696372,"statfs":{"total":96636764160,"available":95945084928,"internally_reserved":0,"allocated":5050368,"data_stored":4871265,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":686619077},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4222976,"data_stored":4195178,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1532,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":122880,"data_stored":90225,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8507392,"data_stored":8478867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912004,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":4514,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":5935104,"data_stored":5912136,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12619776,"data_stored":12585594,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T05:24:15.221 DEBUG:teuthology.orchestra.run.smithi001:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2024-05-09T05:24:15.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.375+0000 7efd793ba640 1 Processor -- start 2024-05-09T05:24:15.371 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.375+0000 7efd793ba640 1 -- start start 2024-05-09T05:24:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.376+0000 7efd793ba640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7efd740cf360 0x7efd740b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd793ba640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd740b4570 0x7efd74156b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd793ba640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd7415c9a0 0x7efd7415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd793ba640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7efd740ce5c0 con 0x7efd740b4570 2024-05-09T05:24:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd793ba640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7efd7415d4c0 con 0x7efd740cf360 2024-05-09T05:24:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd793ba640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efd7415d7a0 con 0x7efd7415c9a0 2024-05-09T05:24:15.373 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd727fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd740b4570 0x7efd74156b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd727fc640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7efd740b4570 0x7efd74156b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49824/0 (socket says 172.21.15.1:49824) 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd727fc640 1 -- 172.21.15.1:0/3969714089 learned_addr learned my addr 172.21.15.1:0/3969714089 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd72ffd640 1 --2- 172.21.15.1:0/3969714089 >> v2:172.21.15.196:3300/0 conn(0x7efd740cf360 0x7efd740b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd737fe640 1 --2- 172.21.15.1:0/3969714089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd7415c9a0 0x7efd7415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd727fc640 1 -- 172.21.15.1:0/3969714089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd7415c9a0 msgr2=0x7efd7415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd727fc640 1 --2- 172.21.15.1:0/3969714089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd7415c9a0 0x7efd7415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd727fc640 1 -- 172.21.15.1:0/3969714089 >> v2:172.21.15.196:3300/0 conn(0x7efd740cf360 msgr2=0x7efd740b4b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd727fc640 1 --2- 172.21.15.1:0/3969714089 >> v2:172.21.15.196:3300/0 conn(0x7efd740cf360 0x7efd740b4b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd727fc640 1 -- 172.21.15.1:0/3969714089 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd740ce880 con 0x7efd740b4570 2024-05-09T05:24:15.374 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.377+0000 7efd72ffd640 1 --2- 172.21.15.1:0/3969714089 >> v2:172.21.15.196:3300/0 conn(0x7efd740cf360 0x7efd740b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:24:15.375 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.378+0000 7efd727fc640 1 --2- 172.21.15.1:0/3969714089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd740b4570 0x7efd74156b50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7efd7408f300 tx=0x7efd6802a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db24aed21e13b1a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.378+0000 7efd71ffb640 1 -- 172.21.15.1:0/3969714089 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68037070 con 0x7efd740b4570 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.378+0000 7efd71ffb640 1 -- 172.21.15.1:0/3969714089 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd68032440 con 0x7efd740b4570 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.378+0000 7efd71ffb640 1 -- 172.21.15.1:0/3969714089 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68032810 con 0x7efd740b4570 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 -- 172.21.15.1:0/3969714089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd740b4570 msgr2=0x7efd74156b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 --2- 172.21.15.1:0/3969714089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd740b4570 0x7efd74156b50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7efd7408f300 tx=0x7efd6802a270 comp rx=0 tx=0).stop 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 -- 172.21.15.1:0/3969714089 shutdown_connections 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 --2- 172.21.15.1:0/3969714089 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd7415c9a0 0x7efd7415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 --2- 172.21.15.1:0/3969714089 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd740b4570 0x7efd74156b50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 --2- 172.21.15.1:0/3969714089 >> v2:172.21.15.196:3300/0 conn(0x7efd740cf360 0x7efd740b4b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 -- 172.21.15.1:0/3969714089 >> 172.21.15.1:0/3969714089 conn(0x7efd7406de10 msgr2=0x7efd7406e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 -- 172.21.15.1:0/3969714089 shutdown_connections 2024-05-09T05:24:15.376 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.379+0000 7efd793ba640 1 -- 172.21.15.1:0/3969714089 wait complete. 2024-05-09T05:24:15.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.381+0000 7efd793ba640 1 Processor -- start 2024-05-09T05:24:15.377 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.381+0000 7efd793ba640 1 -- start start 2024-05-09T05:24:15.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd793ba640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd740b4570 0x7efd741f5a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd793ba640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7415c9a0 0x7efd741e3380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd793ba640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd740c83f0 0x7efd741e57e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd793ba640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7efd740b5090 con 0x7efd7415c9a0 2024-05-09T05:24:15.379 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd793ba640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7efd740cee40 con 0x7efd740b4570 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd793ba640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7efd74157090 con 0x7efd740c83f0 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd727fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7415c9a0 0x7efd741e3380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd727fc640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7415c9a0 0x7efd741e3380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49834/0 (socket says 172.21.15.1:49834) 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd727fc640 1 -- 172.21.15.1:0/4191395356 learned_addr learned my addr 172.21.15.1:0/4191395356 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd72ffd640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd740b4570 0x7efd741f5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd737fe640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd740c83f0 0x7efd741e57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd727fc640 1 -- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd740c83f0 msgr2=0x7efd741e57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.382+0000 7efd727fc640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd740c83f0 0x7efd741e57e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd727fc640 1 -- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd740b4570 msgr2=0x7efd741f5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd727fc640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd740b4570 0x7efd741f5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd727fc640 1 -- 172.21.15.1:0/4191395356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd741f5f80 con 0x7efd7415c9a0 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd727fc640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7415c9a0 0x7efd741e3380 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efd6802a2b0 tx=0x7efd680046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd5b7fe640 1 -- 172.21.15.1:0/4191395356 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68041020 con 0x7efd7415c9a0 2024-05-09T05:24:15.380 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd5b7fe640 1 -- 172.21.15.1:0/4191395356 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efd680306c0 con 0x7efd7415c9a0 2024-05-09T05:24:15.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd741e4b10 con 0x7efd7415c9a0 2024-05-09T05:24:15.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd5b7fe640 1 -- 172.21.15.1:0/4191395356 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd68037040 con 0x7efd7415c9a0 2024-05-09T05:24:15.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.383+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd7415a380 con 0x7efd7415c9a0 2024-05-09T05:24:15.381 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.384+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd74050870 con 0x7efd7415c9a0 2024-05-09T05:24:15.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.385+0000 7efd5b7fe640 1 -- 172.21.15.1:0/4191395356 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7efd68044020 con 0x7efd7415c9a0 2024-05-09T05:24:15.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.385+0000 7efd5b7fe640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd600762d0 0x7efd600787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.385+0000 7efd5b7fe640 1 -- 172.21.15.1:0/4191395356 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7efd680b7e80 con 0x7efd7415c9a0 2024-05-09T05:24:15.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.386+0000 7efd72ffd640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd600762d0 0x7efd600787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.382 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.386+0000 7efd72ffd640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd600762d0 0x7efd600787e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efd64003fd0 tx=0x7efd640039f0 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:15.386 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.390+0000 7efd5b7fe640 1 -- 172.21.15.1:0/4191395356 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7efd6804b030 con 0x7efd7415c9a0 2024-05-09T05:24:15.575 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.580+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) -- 0x7efd740cdf30 con 0x7efd7415c9a0 2024-05-09T05:24:15.579 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.583+0000 7efd5b7fe640 1 -- 172.21.15.1:0/4191395356 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v10)=0 v10) ==== 123+0+0 (secure 0 0 0) 0x7efd680817f0 con 0x7efd7415c9a0 2024-05-09T05:24:15.580 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.584+0000 7efd5b7fe640 1 -- 172.21.15.1:0/4191395356 <== mon.0 v2:172.21.15.1:3300/0 8 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7efd68080c00 con 0x7efd7415c9a0 2024-05-09T05:24:15.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd600762d0 msgr2=0x7efd600787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.582 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd600762d0 0x7efd600787e0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7efd64003fd0 tx=0x7efd640039f0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7415c9a0 msgr2=0x7efd741e3380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7415c9a0 0x7efd741e3380 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7efd6802a2b0 tx=0x7efd680046c0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 shutdown_connections 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7efd600762d0 0x7efd600787e0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7efd740c83f0 0x7efd741e57e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7efd7415c9a0 0x7efd741e3380 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 --2- 172.21.15.1:0/4191395356 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7efd740b4570 0x7efd741f5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 >> 172.21.15.1:0/4191395356 conn(0x7efd7406de10 msgr2=0x7efd740d0830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 shutdown_connections 2024-05-09T05:24:15.583 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.586+0000 7efd793ba640 1 -- 172.21.15.1:0/4191395356 wait complete. 2024-05-09T05:24:15.594 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T05:24:15.595 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:24:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.713+0000 7f567aab8640 1 Processor -- start 2024-05-09T05:24:15.709 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.713+0000 7f567aab8640 1 -- start start 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f567aab8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567406f1b0 0x7f56740ce740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f567aab8640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f567415ca90 0x7f567415cea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f567aab8640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f567406f6d0 con 0x7f56740d16b0 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f567aab8640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f56740cd0e0 con 0x7f567415ca90 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f567aab8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f56740cc4d0 con 0x7f567406f1b0 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f5679055640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f567415ca90 0x7f567415cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f5679055640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f567415ca90 0x7f567415cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49844/0 (socket says 172.21.15.1:49844) 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f5679055640 1 -- 172.21.15.1:0/3999096969 learned_addr learned my addr 172.21.15.1:0/3999096969 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:15.711 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.714+0000 7f5678854640 1 --1- 172.21.15.1:0/3999096969 >> v1:172.21.15.1:6789/0 conn(0x7f56740d16b0 0x7f56740d03d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37458/0 (socket says 172.21.15.1:37458) 2024-05-09T05:24:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.715+0000 7f5673fff640 1 --2- 172.21.15.1:0/3999096969 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567406f1b0 0x7f56740ce740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.715+0000 7f5679055640 1 -- 172.21.15.1:0/3999096969 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567406f1b0 msgr2=0x7f56740ce740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.715+0000 7f5679055640 1 --2- 172.21.15.1:0/3999096969 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567406f1b0 0x7f56740ce740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.715+0000 7f5679055640 1 -- 172.21.15.1:0/3999096969 >> v1:172.21.15.1:6789/0 conn(0x7f56740d16b0 legacy=0x7f56740d03d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.715+0000 7f56737fe640 1 -- 172.21.15.1:0/3999096969 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1864277675 0 0) 0x7f567406f6d0 con 0x7f56740d16b0 2024-05-09T05:24:15.712 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.715+0000 7f5679055640 1 -- 172.21.15.1:0/3999096969 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56740cc7b0 con 0x7f567415ca90 2024-05-09T05:24:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.715+0000 7f5679055640 1 --2- 172.21.15.1:0/3999096969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567415ca90 0x7f567415cea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f566c009e50 tx=0x7f566c02a7a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5aa87ddbed1d592f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f56737fe640 1 -- 172.21.15.1:0/3999096969 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c036070 con 0x7f567415ca90 2024-05-09T05:24:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f56737fe640 1 -- 172.21.15.1:0/3999096969 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f566c031600 con 0x7f567415ca90 2024-05-09T05:24:15.714 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f56737fe640 1 -- 172.21.15.1:0/3999096969 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f566c0319f0 con 0x7f567415ca90 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f567aab8640 1 -- 172.21.15.1:0/3999096969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567415ca90 msgr2=0x7f567415cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f567aab8640 1 --2- 172.21.15.1:0/3999096969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567415ca90 0x7f567415cea0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f566c009e50 tx=0x7f566c02a7a0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f567aab8640 1 -- 172.21.15.1:0/3999096969 shutdown_connections 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f567aab8640 1 --2- 172.21.15.1:0/3999096969 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567406f1b0 0x7f56740ce740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f567aab8640 1 --2- 172.21.15.1:0/3999096969 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567415ca90 0x7f567415cea0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f567aab8640 1 -- 172.21.15.1:0/3999096969 >> 172.21.15.1:0/3999096969 conn(0x7f56740b3660 msgr2=0x7f56740b3a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f567aab8640 1 -- 172.21.15.1:0/3999096969 shutdown_connections 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.716+0000 7f567aab8640 1 -- 172.21.15.1:0/3999096969 wait complete. 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.718+0000 7f567aab8640 1 Processor -- start 2024-05-09T05:24:15.715 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.718+0000 7f567aab8640 1 -- start start 2024-05-09T05:24:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.719+0000 7f567aab8640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567406f1b0 0x7f5674159650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.717 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.719+0000 7f567aab8640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56740d16b0 0x7f5674156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.719+0000 7f567aab8640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567415ca90 0x7f5674151dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.719+0000 7f567aab8640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f567415d3e0 con 0x7f567406f1b0 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f567aab8640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f56740ccd70 con 0x7f56740d16b0 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f567aab8640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f56740cec80 con 0x7f567415ca90 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567406f1b0 0x7f5674159650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567406f1b0 0x7f5674159650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49858/0 (socket says 172.21.15.1:49858) 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 -- 172.21.15.1:0/1515228457 learned_addr learned my addr 172.21.15.1:0/1515228457 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5679055640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567415ca90 0x7f5674151dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5673fff640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56740d16b0 0x7f5674156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 -- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567415ca90 msgr2=0x7f5674151dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567415ca90 0x7f5674151dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 -- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56740d16b0 msgr2=0x7f5674156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56740d16b0 0x7f5674156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.718 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 -- 172.21.15.1:0/1515228457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5674157440 con 0x7f567406f1b0 2024-05-09T05:24:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5679055640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567415ca90 0x7f5674151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:24:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f5678854640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567406f1b0 0x7f5674159650 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f566800c410 tx=0x7f566800c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f56717fa640 1 -- 172.21.15.1:0/1515228457 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56680074e0 con 0x7f567406f1b0 2024-05-09T05:24:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f56717fa640 1 -- 172.21.15.1:0/1515228457 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f56680047e0 con 0x7f567406f1b0 2024-05-09T05:24:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56741585b0 con 0x7f567406f1b0 2024-05-09T05:24:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f56717fa640 1 -- 172.21.15.1:0/1515228457 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5668004bf0 con 0x7f567406f1b0 2024-05-09T05:24:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.720+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5674158b70 con 0x7f567406f1b0 2024-05-09T05:24:15.719 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.721+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5674050870 con 0x7f567406f1b0 2024-05-09T05:24:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.722+0000 7f56717fa640 1 -- 172.21.15.1:0/1515228457 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f5668011070 con 0x7f567406f1b0 2024-05-09T05:24:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.723+0000 7f56717fa640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56580762d0 0x7f56580787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.723+0000 7f56717fa640 1 -- 172.21.15.1:0/1515228457 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7f56680978c0 con 0x7f567406f1b0 2024-05-09T05:24:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.723+0000 7f5673fff640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56580762d0 0x7f56580787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:15.720 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.723+0000 7f5673fff640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56580762d0 0x7f56580787e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5660006fd0 tx=0x7f5660008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:15.722 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.727+0000 7f56717fa640 1 -- 172.21.15.1:0/1515228457 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f56680611f0 con 0x7f567406f1b0 2024-05-09T05:24:15.954 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.958+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f56740cf070 con 0x7f567406f1b0 2024-05-09T05:24:15.955 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.959+0000 7f56717fa640 1 -- 172.21.15.1:0/1515228457 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6915 (secure 0 0 0) 0x7f5668061e20 con 0x7f567406f1b0 2024-05-09T05:24:15.955 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:24:15.955 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:33.214091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34160,"mds_1":34157,"mds_2":4311,"mds_3":4287,"mds_4":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4311":{"gid":4311,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[4273,34160,4287,4311,34157]},"id":1}]} 2024-05-09T05:24:15.955 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-05-09T05:24:15.957 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.961+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56580762d0 msgr2=0x7f56580787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.961+0000 7f567aab8640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56580762d0 0x7f56580787e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f5660006fd0 tx=0x7f5660008040 comp rx=0 tx=0).stop 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.961+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567406f1b0 msgr2=0x7f5674159650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.961+0000 7f567aab8640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567406f1b0 0x7f5674159650 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f566800c410 tx=0x7f566800c8d0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.962+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 shutdown_connections 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.962+0000 7f567aab8640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f56580762d0 0x7f56580787e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.962+0000 7f567aab8640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f56740d16b0 0x7f5674156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.962+0000 7f567aab8640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f567415ca90 0x7f5674151dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.962+0000 7f567aab8640 1 --2- 172.21.15.1:0/1515228457 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f567406f1b0 0x7f5674159650 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.962+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 >> 172.21.15.1:0/1515228457 conn(0x7f56740b3660 msgr2=0x7f5674075d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.962+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 shutdown_connections 2024-05-09T05:24:15.958 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:15.962+0000 7f567aab8640 1 -- 172.21.15.1:0/1515228457 wait complete. 2024-05-09T05:24:15.969 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T05:24:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.088+0000 7ffa323b2640 1 Processor -- start 2024-05-09T05:24:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.088+0000 7ffa323b2640 1 -- start start 2024-05-09T05:24:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.090+0000 7ffa323b2640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa2c0ce390 0x7ffa2c16d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.090+0000 7ffa323b2640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa2c0cd7a0 con 0x7ffa2c072e20 2024-05-09T05:24:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.090+0000 7ffa323b2640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ffa2c0c7340 con 0x7ffa2c0746e0 2024-05-09T05:24:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.090+0000 7ffa323b2640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7ffa2c0cafe0 con 0x7ffa2c0ce390 2024-05-09T05:24:16.091 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.090+0000 7ffa3094f640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7ffa2c0ce390 0x7ffa2c16d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.090+0000 7ffa2bfff640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ffa2c072e20 0x7ffa2c0ca8d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37470/0 (socket says 172.21.15.1:37470) 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.090+0000 7ffa2bfff640 1 -- 172.21.15.1:0/315859139 learned_addr learned my addr 172.21.15.1:0/315859139 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.090+0000 7ffa2affd640 1 -- 172.21.15.1:0/315859139 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3037897817 0 0) 0x7ffa2c0cd7a0 con 0x7ffa2c072e20 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.095+0000 7ffa2affd640 1 -- 172.21.15.1:0/315859139 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa14003500 con 0x7ffa2c072e20 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.095+0000 7ffa2affd640 1 -- 172.21.15.1:0/315859139 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 44283988 0 0) 0x7ffa2c0c7340 con 0x7ffa2c0746e0 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.095+0000 7ffa2affd640 1 -- 172.21.15.1:0/315859139 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ffa2c0cd7a0 con 0x7ffa2c0746e0 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.095+0000 7ffa3094f640 1 -- 172.21.15.1:0/315859139 >> v1:172.21.15.196:6789/0 conn(0x7ffa2c0746e0 legacy=0x7ffa2c0cdc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.095+0000 7ffa3094f640 1 -- 172.21.15.1:0/315859139 >> v1:172.21.15.1:6789/0 conn(0x7ffa2c072e20 legacy=0x7ffa2c0ca8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.095+0000 7ffa2affd640 1 -- 172.21.15.1:0/315859139 <== mon.0 v1:172.21.15.1:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) ==== 214+0+0 (unknown 3306113356 0 0) 0x7ffa14003500 con 0x7ffa2c072e20 2024-05-09T05:24:16.092 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.095+0000 7ffa3094f640 1 -- 172.21.15.1:0/315859139 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2c0c7620 con 0x7ffa2c0ce390 2024-05-09T05:24:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.096+0000 7ffa3094f640 1 --2- 172.21.15.1:0/315859139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c0ce390 0x7ffa2c16d0e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b200 tx=0x7ffa2402abe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcea972fe7593b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.093 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.096+0000 7ffa2affd640 1 -- 172.21.15.1:0/315859139 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24031070 con 0x7ffa2c0ce390 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.096+0000 7ffa2affd640 1 -- 172.21.15.1:0/315859139 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ffa24007dd0 con 0x7ffa2c0ce390 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.096+0000 7ffa2affd640 1 -- 172.21.15.1:0/315859139 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa1c001710 con 0x7ffa2c0ce390 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.096+0000 7ffa323b2640 1 -- 172.21.15.1:0/315859139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c0ce390 msgr2=0x7ffa2c16d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.096+0000 7ffa323b2640 1 --2- 172.21.15.1:0/315859139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c0ce390 0x7ffa2c16d0e0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b200 tx=0x7ffa2402abe0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.097+0000 7ffa323b2640 1 -- 172.21.15.1:0/315859139 shutdown_connections 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.097+0000 7ffa323b2640 1 --2- 172.21.15.1:0/315859139 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c0ce390 0x7ffa2c16d0e0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.097+0000 7ffa323b2640 1 -- 172.21.15.1:0/315859139 >> 172.21.15.1:0/315859139 conn(0x7ffa2c070dd0 msgr2=0x7ffa2c0711e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.097+0000 7ffa323b2640 1 -- 172.21.15.1:0/315859139 shutdown_connections 2024-05-09T05:24:16.094 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.097+0000 7ffa323b2640 1 -- 172.21.15.1:0/315859139 wait complete. 2024-05-09T05:24:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.098+0000 7ffa323b2640 1 Processor -- start 2024-05-09T05:24:16.095 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.098+0000 7ffa323b2640 1 -- start start 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa323b2640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa2c072e20 0x7ffa2c0c54b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa323b2640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa2c0746e0 0x7ffa2c15fb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa323b2640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c160e20 0x7ffa2c161230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa323b2640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c0c7be0 con 0x7ffa2c160e20 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa323b2640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ffa2c0b3c60 con 0x7ffa2c072e20 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa323b2640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ffa2c074c60 con 0x7ffa2c0746e0 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c160e20 0x7ffa2c161230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c160e20 0x7ffa2c161230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49888/0 (socket says 172.21.15.1:49888) 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 -- 172.21.15.1:0/2106960943 learned_addr learned my addr 172.21.15.1:0/2106960943 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa2bfff640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa2c072e20 0x7ffa2c0c54b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.097 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa2b7fe640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa2c0746e0 0x7ffa2c15fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 -- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa2c0746e0 msgr2=0x7ffa2c15fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa2c0746e0 0x7ffa2c15fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 -- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa2c072e20 msgr2=0x7ffa2c0c54b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa2c072e20 0x7ffa2c0c54b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 -- 172.21.15.1:0/2106960943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa2c0c59f0 con 0x7ffa2c160e20 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.100+0000 7ffa3094f640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c160e20 0x7ffa2c161230 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b100 tx=0x7ffa24004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.101+0000 7ffa28ff9640 1 -- 172.21.15.1:0/2106960943 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24045070 con 0x7ffa2c160e20 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.101+0000 7ffa28ff9640 1 -- 172.21.15.1:0/2106960943 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ffa2402c680 con 0x7ffa2c160e20 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.101+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa2c1ffd60 con 0x7ffa2c160e20 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.101+0000 7ffa28ff9640 1 -- 172.21.15.1:0/2106960943 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa24031070 con 0x7ffa2c160e20 2024-05-09T05:24:16.098 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.101+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa2c0c49d0 con 0x7ffa2c160e20 2024-05-09T05:24:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.102+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa2c050870 con 0x7ffa2c160e20 2024-05-09T05:24:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.102+0000 7ffa28ff9640 1 -- 172.21.15.1:0/2106960943 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ffa24048070 con 0x7ffa2c160e20 2024-05-09T05:24:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.103+0000 7ffa28ff9640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa180762d0 0x7ffa180787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.103+0000 7ffa28ff9640 1 -- 172.21.15.1:0/2106960943 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7ffa240b7ff0 con 0x7ffa2c160e20 2024-05-09T05:24:16.099 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.103+0000 7ffa2bfff640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa180762d0 0x7ffa180787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.100 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.104+0000 7ffa2bfff640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa180762d0 0x7ffa180787e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c001eb0 tx=0x7ffa1c001790 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.103 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.108+0000 7ffa28ff9640 1 -- 172.21.15.1:0/2106960943 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ffa24081170 con 0x7ffa2c160e20 2024-05-09T05:24:16.334 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.338+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa2c0c2cb0 con 0x7ffa2c160e20 2024-05-09T05:24:16.335 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.339+0000 7ffa28ff9640 1 -- 172.21.15.1:0/2106960943 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) ==== 94+0+6915 (secure 0 0 0) 0x7ffa24085470 con 0x7ffa2c160e20 2024-05-09T05:24:16.335 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:24:16.335 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":4270,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.1:6835/2618026039","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6833","nonce":2618026039},{"type":"v1","addr":"172.21.15.1:6835","nonce":2618026039}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T05:06:04.442853+0000","modified":"2024-05-09T05:06:33.214091+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":34160,"mds_1":34157,"mds_2":4311,"mds_3":4287,"mds_4":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"e","rank":4,"incarnation":15,"state":"up:active","state_seq":10,"addr":"172.21.15.1:6837/3883243235","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6836","nonce":3883243235},{"type":"v1","addr":"172.21.15.1:6837","nonce":3883243235}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4287":{"gid":4287,"name":"c","rank":3,"incarnation":13,"state":"up:active","state_seq":9,"addr":"172.21.15.1:6834/1980556788","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6832","nonce":1980556788},{"type":"v1","addr":"172.21.15.1:6834","nonce":1980556788}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_4311":{"gid":4311,"name":"d","rank":2,"incarnation":11,"state":"up:active","state_seq":7,"addr":"172.21.15.196:6839/2336267187","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6838","nonce":2336267187},{"type":"v1","addr":"172.21.15.196:6839","nonce":2336267187}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34157":{"gid":34157,"name":"b","rank":1,"incarnation":9,"state":"up:active","state_seq":6,"addr":"172.21.15.196:6835/1932237812","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6834","nonce":1932237812},{"type":"v1","addr":"172.21.15.196:6835","nonce":1932237812}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_34160":{"gid":34160,"name":"f","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.196:6837/3384416671","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6836","nonce":3384416671},{"type":"v1","addr":"172.21.15.196:6837","nonce":3384416671}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":34160,"qdb_cluster":[4273,34160,4287,4311,34157]},"id":1}]} 2024-05-09T05:24:16.335 INFO:teuthology.orchestra.run.smithi001.stderr:dumped fsmap epoch 16 2024-05-09T05:24:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa180762d0 msgr2=0x7ffa180787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa180762d0 0x7ffa180787e0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ffa1c001eb0 tx=0x7ffa1c001790 comp rx=0 tx=0).stop 2024-05-09T05:24:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c160e20 msgr2=0x7ffa2c161230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c160e20 0x7ffa2c161230 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7ffa2400b100 tx=0x7ffa24004a40 comp rx=0 tx=0).stop 2024-05-09T05:24:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 shutdown_connections 2024-05-09T05:24:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ffa180762d0 0x7ffa180787e0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.337 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ffa2c160e20 0x7ffa2c161230 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ffa2c0746e0 0x7ffa2c15fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 --2- 172.21.15.1:0/2106960943 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ffa2c072e20 0x7ffa2c0c54b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 >> 172.21.15.1:0/2106960943 conn(0x7ffa2c070dd0 msgr2=0x7ffa2c162360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 shutdown_connections 2024-05-09T05:24:16.338 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.341+0000 7ffa323b2640 1 -- 172.21.15.1:0/2106960943 wait complete. 2024-05-09T05:24:16.349 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T05:24:16.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.467+0000 7fc112f57640 1 Processor -- start 2024-05-09T05:24:16.463 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.467+0000 7fc112f57640 1 -- start start 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc112f57640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc10c072120 0x7fc10c156a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc112f57640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c0caab0 0x7fc10c0caec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc112f57640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc10c0c5e00 con 0x7fc10c15d1d0 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc112f57640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc10c0ca340 con 0x7fc10c072120 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc112f57640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c0ca620 con 0x7fc10c0caab0 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc1114f4640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c0caab0 0x7fc10c0caec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc110cf3640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc10c072120 0x7fc10c156a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc1114f4640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c0caab0 0x7fc10c0caec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.196:3301/0 says I am v2:172.21.15.1:54226/0 (socket says 172.21.15.1:54226) 2024-05-09T05:24:16.465 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc1114f4640 1 -- 172.21.15.1:0/3641455700 learned_addr learned my addr 172.21.15.1:0/3641455700 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.469+0000 7fc110cf3640 1 -- 172.21.15.1:0/3641455700 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c0caab0 msgr2=0x7fc10c0caec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.470+0000 7fc110cf3640 1 --2- 172.21.15.1:0/3641455700 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c0caab0 0x7fc10c0caec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.470+0000 7fc110cf3640 1 -- 172.21.15.1:0/3641455700 >> v1:172.21.15.196:6789/0 conn(0x7fc10c15d1d0 legacy=0x7fc10c070e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.470+0000 7fc10b7fe640 1 -- 172.21.15.1:0/3641455700 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 1968746812 0 0) 0x7fc10c0c5e00 con 0x7fc10c15d1d0 2024-05-09T05:24:16.467 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.470+0000 7fc110cf3640 1 -- 172.21.15.1:0/3641455700 --> v2:172.21.15.196:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c0c60e0 con 0x7fc10c072120 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.470+0000 7fc1114f4640 1 --2- 172.21.15.1:0/3641455700 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c0caab0 0x7fc10c0caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.470+0000 7fc110cf3640 1 --2- 172.21.15.1:0/3641455700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c072120 0x7fc10c156a80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc104009d90 tx=0x7fc10402a660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9ae2effee66b4721 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc10b7fe640 1 -- 172.21.15.1:0/3641455700 <== mon.1 v2:172.21.15.196:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104007740 con 0x7fc10c072120 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc10b7fe640 1 -- 172.21.15.1:0/3641455700 <== mon.1 v2:172.21.15.196:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc1040078e0 con 0x7fc10c072120 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc10b7fe640 1 -- 172.21.15.1:0/3641455700 <== mon.1 v2:172.21.15.196:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc104007cd0 con 0x7fc10c072120 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc112f57640 1 -- 172.21.15.1:0/3641455700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c072120 msgr2=0x7fc10c156a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc112f57640 1 --2- 172.21.15.1:0/3641455700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c072120 0x7fc10c156a80 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fc104009d90 tx=0x7fc10402a660 comp rx=0 tx=0).stop 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc112f57640 1 -- 172.21.15.1:0/3641455700 shutdown_connections 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc112f57640 1 --2- 172.21.15.1:0/3641455700 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c0caab0 0x7fc10c0caec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc112f57640 1 --2- 172.21.15.1:0/3641455700 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c072120 0x7fc10c156a80 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc112f57640 1 -- 172.21.15.1:0/3641455700 >> 172.21.15.1:0/3641455700 conn(0x7fc10c0731d0 msgr2=0x7fc10c0cf9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc112f57640 1 -- 172.21.15.1:0/3641455700 shutdown_connections 2024-05-09T05:24:16.468 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.471+0000 7fc112f57640 1 -- 172.21.15.1:0/3641455700 wait complete. 2024-05-09T05:24:16.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.473+0000 7fc112f57640 1 Processor -- start 2024-05-09T05:24:16.469 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.473+0000 7fc112f57640 1 -- start start 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.474+0000 7fc112f57640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c072120 0x7fc10c156060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.474+0000 7fc112f57640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10c0caab0 0x7fc10c159790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.474+0000 7fc112f57640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c15d1d0 0x7fc10c158c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.474+0000 7fc112f57640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c0cb400 con 0x7fc10c0caab0 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc112f57640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc10c0c66a0 con 0x7fc10c15d1d0 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc112f57640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc10c15d5e0 con 0x7fc10c072120 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10c0caab0 0x7fc10c159790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10c0caab0 0x7fc10c159790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49900/0 (socket says 172.21.15.1:49900) 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 -- 172.21.15.1:0/934196133 learned_addr learned my addr 172.21.15.1:0/934196133 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc110cf3640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c072120 0x7fc10c156060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc1114f4640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c15d1d0 0x7fc10c158c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.472 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 -- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c072120 msgr2=0x7fc10c156060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c072120 0x7fc10c156060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 -- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c15d1d0 msgr2=0x7fc10c158c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c15d1d0 0x7fc10c158c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 -- 172.21.15.1:0/934196133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc10c155d00 con 0x7fc10c0caab0 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc110cf3640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c072120 0x7fc10c156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc10bfff640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10c0caab0 0x7fc10c159790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc0f400c3f0 tx=0x7fc0f400c8b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc1097fa640 1 -- 172.21.15.1:0/934196133 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f4017070 con 0x7fc10c0caab0 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc1097fa640 1 -- 172.21.15.1:0/934196133 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc0f40035f0 con 0x7fc10c0caab0 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc10c157840 con 0x7fc10c0caab0 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.475+0000 7fc1097fa640 1 -- 172.21.15.1:0/934196133 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f4003a00 con 0x7fc10c0caab0 2024-05-09T05:24:16.473 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.476+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc10c157e00 con 0x7fc10c0caab0 2024-05-09T05:24:16.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.476+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc10c159e90 con 0x7fc10c0caab0 2024-05-09T05:24:16.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.477+0000 7fc1097fa640 1 -- 172.21.15.1:0/934196133 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc0f401a070 con 0x7fc10c0caab0 2024-05-09T05:24:16.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.478+0000 7fc1097fa640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ec0762d0 0x7fc0ec0787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.478+0000 7fc1097fa640 1 -- 172.21.15.1:0/934196133 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc0f4096ae0 con 0x7fc10c0caab0 2024-05-09T05:24:16.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.478+0000 7fc110cf3640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ec0762d0 0x7fc0ec0787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.474 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.479+0000 7fc110cf3640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ec0762d0 0x7fc0ec0787e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc104034000 tx=0x7fc10402f000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.478 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.483+0000 7fc1097fa640 1 -- 172.21.15.1:0/934196133 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc0f405fbc0 con 0x7fc10c0caab0 2024-05-09T05:24:16.669 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.674+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fc10c050870 con 0x7fc10c0caab0 2024-05-09T05:24:16.671 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.675+0000 7fc1097fa640 1 -- 172.21.15.1:0/934196133 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) ==== 74+0+16795 (secure 0 0 0) 0x7fc0f4063ec0 con 0x7fc10c0caab0 2024-05-09T05:24:16.671 INFO:teuthology.orchestra.run.smithi001.stdout: 2024-05-09T05:24:16.671 INFO:teuthology.orchestra.run.smithi001.stdout:{"epoch":23,"fsid":"aa0bbdea-07dc-4c3a-9697-b556ae629556","created":"2024-05-09T05:05:41.096226+0000","modified":"2024-05-09T05:07:01.648910+0000","last_up_change":"2024-05-09T05:05:51.871680+0000","last_in_change":"2024-05-09T05:05:43.337985+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":5,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":8,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T05:05:51.558449+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":8,"score_stable":8,"optimal_score":0.25,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T05:06:02.375743+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"20","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T05:06:03.392859+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.3799999952316284,"score_stable":1.3799999952316284,"optimal_score":1,"raw_score_acting":1.3799999952316284,"raw_score_stable":1.3799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"cfa581c7-a28f-41ce-98c2-bf8e56f9af33","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6824","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6825","nonce":1100742567}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6826","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6827","nonce":1100742567}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6830","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6831","nonce":1100742567}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6828","nonce":1100742567},{"type":"v1","addr":"172.21.15.1:6829","nonce":1100742567}]},"public_addr":"172.21.15.1:6825/1100742567","cluster_addr":"172.21.15.1:6827/1100742567","heartbeat_back_addr":"172.21.15.1:6831/1100742567","heartbeat_front_addr":"172.21.15.1:6829/1100742567","state":["exists","up"]},{"osd":1,"uuid":"040c2166-4e3a-4f17-a385-a50c34d987ec","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6811","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6813","nonce":2455090880}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6815","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6817","nonce":2455090880}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6822","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6823","nonce":2455090880}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6819","nonce":2455090880},{"type":"v1","addr":"172.21.15.1:6821","nonce":2455090880}]},"public_addr":"172.21.15.1:6813/2455090880","cluster_addr":"172.21.15.1:6817/2455090880","heartbeat_back_addr":"172.21.15.1:6823/2455090880","heartbeat_front_addr":"172.21.15.1:6821/2455090880","state":["exists","up"]},{"osd":2,"uuid":"4977c108-0fff-47ec-802e-67b4e9464153","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6800","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6801","nonce":239705464}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6802","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6803","nonce":239705464}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6806","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6807","nonce":239705464}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6804","nonce":239705464},{"type":"v1","addr":"172.21.15.1:6805","nonce":239705464}]},"public_addr":"172.21.15.1:6801/239705464","cluster_addr":"172.21.15.1:6803/239705464","heartbeat_back_addr":"172.21.15.1:6807/239705464","heartbeat_front_addr":"172.21.15.1:6805/239705464","state":["exists","up"]},{"osd":3,"uuid":"4f41003a-2ede-4fa9-a0d3-66c790841802","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6808","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6809","nonce":3991537322}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6810","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6812","nonce":3991537322}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6818","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6820","nonce":3991537322}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.1:6814","nonce":3991537322},{"type":"v1","addr":"172.21.15.1:6816","nonce":3991537322}]},"public_addr":"172.21.15.1:6809/3991537322","cluster_addr":"172.21.15.1:6812/3991537322","heartbeat_back_addr":"172.21.15.1:6820/3991537322","heartbeat_front_addr":"172.21.15.1:6816/3991537322","state":["exists","up"]},{"osd":4,"uuid":"62cd371e-e40a-4361-8778-94e828501072","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6808","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6809","nonce":4274960119}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6810","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6811","nonce":4274960119}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6814","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6815","nonce":4274960119}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6812","nonce":4274960119},{"type":"v1","addr":"172.21.15.196:6813","nonce":4274960119}]},"public_addr":"172.21.15.196:6809/4274960119","cluster_addr":"172.21.15.196:6811/4274960119","heartbeat_back_addr":"172.21.15.196:6815/4274960119","heartbeat_front_addr":"172.21.15.196:6813/4274960119","state":["exists","up"]},{"osd":5,"uuid":"33179512-6a79-4c65-a6cc-f26eb69974d5","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6823","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6825","nonce":3481343234}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6826","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6827","nonce":3481343234}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6830","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6831","nonce":3481343234}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6828","nonce":3481343234},{"type":"v1","addr":"172.21.15.196:6829","nonce":3481343234}]},"public_addr":"172.21.15.196:6825/3481343234","cluster_addr":"172.21.15.196:6827/3481343234","heartbeat_back_addr":"172.21.15.196:6831/3481343234","heartbeat_front_addr":"172.21.15.196:6829/3481343234","state":["exists","up"]},{"osd":6,"uuid":"fecedcf1-e751-4020-8a8f-d3e6057c6e29","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":22,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6800","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6801","nonce":3854610795}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6802","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6803","nonce":3854610795}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6806","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6807","nonce":3854610795}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6804","nonce":3854610795},{"type":"v1","addr":"172.21.15.196:6805","nonce":3854610795}]},"public_addr":"172.21.15.196:6801/3854610795","cluster_addr":"172.21.15.196:6803/3854610795","heartbeat_back_addr":"172.21.15.196:6807/3854610795","heartbeat_front_addr":"172.21.15.196:6805/3854610795","state":["exists","up"]},{"osd":7,"uuid":"676efa33-462a-4779-ba5a-5ab31c6896ac","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":15,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6816","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6817","nonce":747967558}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6818","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6819","nonce":747967558}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6822","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6824","nonce":747967558}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.196:6820","nonce":747967558},{"type":"v1","addr":"172.21.15.196:6821","nonce":747967558}]},"public_addr":"172.21.15.196:6817/747967558","cluster_addr":"172.21.15.196:6819/747967558","heartbeat_back_addr":"172.21.15.196:6824/747967558","heartbeat_front_addr":"172.21.15.196:6821/747967558","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.527142+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.487884+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:49.861938+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.539425+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.530681+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.531834+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.475768+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T05:05:48.511300+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"3.1c","mappings":[{"from":5,"to":2}]},{"pgid":"3.24","mappings":[{"from":3,"to":1}]},{"pgid":"3.26","mappings":[{"from":7,"to":2}]},{"pgid":"3.2f","mappings":[{"from":5,"to":2}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T05:24:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ec0762d0 msgr2=0x7fc0ec0787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ec0762d0 0x7fc0ec0787e0 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7fc104034000 tx=0x7fc10402f000 comp rx=0 tx=0).stop 2024-05-09T05:24:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10c0caab0 msgr2=0x7fc10c159790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10c0caab0 0x7fc10c159790 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc0f400c3f0 tx=0x7fc0f400c8b0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 shutdown_connections 2024-05-09T05:24:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc0ec0762d0 0x7fc0ec0787e0 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc10c15d1d0 0x7fc10c158c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.673 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc10c0caab0 0x7fc10c159790 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 --2- 172.21.15.1:0/934196133 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc10c072120 0x7fc10c156060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 >> 172.21.15.1:0/934196133 conn(0x7fc10c0731d0 msgr2=0x7fc10c0cc080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:16.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 shutdown_connections 2024-05-09T05:24:16.674 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.677+0000 7fc112f57640 1 -- 172.21.15.1:0/934196133 wait complete. 2024-05-09T05:24:16.686 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T05:24:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.805+0000 7fc389134640 1 Processor -- start 2024-05-09T05:24:16.801 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.805+0000 7fc389134640 1 -- start start 2024-05-09T05:24:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.806+0000 7fc389134640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3840cf360 0x7fc3840b4b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.806+0000 7fc389134640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7fc38415c9a0 0x7fc38415cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.806+0000 7fc389134640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7fc3840ce5c0 con 0x7fc3840b4570 2024-05-09T05:24:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.806+0000 7fc389134640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7fc38415d4c0 con 0x7fc3840cf360 2024-05-09T05:24:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.806+0000 7fc389134640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7fc38415d7a0 con 0x7fc38415c9a0 2024-05-09T05:24:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc382d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3840cf360 0x7fc3840b4b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc382d9d640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7fc3840cf360 0x7fc3840b4b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49916/0 (socket says 172.21.15.1:49916) 2024-05-09T05:24:16.804 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc382d9d640 1 -- 172.21.15.1:0/4151752819 learned_addr learned my addr 172.21.15.1:0/4151752819 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc38359e640 1 --2- 172.21.15.1:0/4151752819 >> v2:172.21.15.196:3300/0 conn(0x7fc38415c9a0 0x7fc38415cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc382d9d640 1 -- 172.21.15.1:0/4151752819 >> v2:172.21.15.196:3300/0 conn(0x7fc38415c9a0 msgr2=0x7fc38415cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc382d9d640 1 --2- 172.21.15.1:0/4151752819 >> v2:172.21.15.196:3300/0 conn(0x7fc38415c9a0 0x7fc38415cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc382d9d640 1 -- 172.21.15.1:0/4151752819 >> v1:172.21.15.196:6789/0 conn(0x7fc3840b4570 legacy=0x7fc384156b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc382d9d640 1 -- 172.21.15.1:0/4151752819 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3840ce880 con 0x7fc3840cf360 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.807+0000 7fc38359e640 1 --2- 172.21.15.1:0/4151752819 >> v2:172.21.15.196:3300/0 conn(0x7fc38415c9a0 0x7fc38415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.808+0000 7fc382d9d640 1 --2- 172.21.15.1:0/4151752819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3840cf360 0x7fc3840b4b50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc38406eb70 tx=0x7fc37402a270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbc9fe9265abffc9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.808+0000 7fc381d9b640 1 -- 172.21.15.1:0/4151752819 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc374037070 con 0x7fc3840cf360 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.808+0000 7fc381d9b640 1 -- 172.21.15.1:0/4151752819 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc374032440 con 0x7fc3840cf360 2024-05-09T05:24:16.805 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.808+0000 7fc381d9b640 1 -- 172.21.15.1:0/4151752819 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc374032830 con 0x7fc3840cf360 2024-05-09T05:24:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.809+0000 7fc389134640 1 -- 172.21.15.1:0/4151752819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3840cf360 msgr2=0x7fc3840b4b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.809+0000 7fc389134640 1 --2- 172.21.15.1:0/4151752819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3840cf360 0x7fc3840b4b50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc38406eb70 tx=0x7fc37402a270 comp rx=0 tx=0).stop 2024-05-09T05:24:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.809+0000 7fc389134640 1 -- 172.21.15.1:0/4151752819 shutdown_connections 2024-05-09T05:24:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.809+0000 7fc389134640 1 --2- 172.21.15.1:0/4151752819 >> v2:172.21.15.196:3300/0 conn(0x7fc38415c9a0 0x7fc38415cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.809+0000 7fc389134640 1 --2- 172.21.15.1:0/4151752819 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3840cf360 0x7fc3840b4b50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.807 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.809+0000 7fc389134640 1 -- 172.21.15.1:0/4151752819 >> 172.21.15.1:0/4151752819 conn(0x7fc38406de10 msgr2=0x7fc38406e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.809+0000 7fc389134640 1 -- 172.21.15.1:0/4151752819 shutdown_connections 2024-05-09T05:24:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.809+0000 7fc389134640 1 -- 172.21.15.1:0/4151752819 wait complete. 2024-05-09T05:24:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.811+0000 7fc389134640 1 Processor -- start 2024-05-09T05:24:16.808 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.811+0000 7fc389134640 1 -- start start 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc389134640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3840b4570 0x7fc3841e8090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc389134640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc38415c9a0 0x7fc3841f2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc389134640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3841f67b0 0x7fc3840ca830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc389134640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7fc3840b5090 con 0x7fc3841f67b0 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc389134640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7fc3840cee40 con 0x7fc38415c9a0 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc389134640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7fc3840cd3f0 con 0x7fc3840b4570 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3841f67b0 0x7fc3840ca830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38359e640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3841f67b0 0x7fc3840ca830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49930/0 (socket says 172.21.15.1:49930) 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38359e640 1 -- 172.21.15.1:0/349983839 learned_addr learned my addr 172.21.15.1:0/349983839 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:16.810 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc382d9d640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3840b4570 0x7fc3841e8090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38259c640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc38415c9a0 0x7fc3841f2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38359e640 1 -- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3840b4570 msgr2=0x7fc3841e8090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38359e640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3840b4570 0x7fc3841e8090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38359e640 1 -- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc38415c9a0 msgr2=0x7fc3841f2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38359e640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc38415c9a0 0x7fc3841f2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.812+0000 7fc38359e640 1 -- 172.21.15.1:0/349983839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3841f3530 con 0x7fc3841f67b0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.813+0000 7fc38359e640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3841f67b0 0x7fc3840ca830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00cd80 tx=0x7fc37c00a650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.813+0000 7fc36b7fe640 1 -- 172.21.15.1:0/349983839 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c00f680 con 0x7fc3841f67b0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.813+0000 7fc36b7fe640 1 -- 172.21.15.1:0/349983839 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7fc37c00f840 con 0x7fc3841f67b0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.813+0000 7fc36b7fe640 1 -- 172.21.15.1:0/349983839 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc37c00fc50 con 0x7fc3841f67b0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.813+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3841e2f50 con 0x7fc3841f67b0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.813+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3841f5fb0 con 0x7fc3841f67b0 2024-05-09T05:24:16.811 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.814+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc384074ac0 con 0x7fc3841f67b0 2024-05-09T05:24:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.815+0000 7fc36b7fe640 1 -- 172.21.15.1:0/349983839 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7fc37c00e3e0 con 0x7fc3841f67b0 2024-05-09T05:24:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.815+0000 7fc36b7fe640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3700762d0 0x7fc3700787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.815+0000 7fc36b7fe640 1 -- 172.21.15.1:0/349983839 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(23..23 src has 1..23) ==== 5825+0+0 (secure 0 0 0) 0x7fc37c098370 con 0x7fc3841f67b0 2024-05-09T05:24:16.812 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.815+0000 7fc382d9d640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3700762d0 0x7fc3700787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:16.813 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.816+0000 7fc382d9d640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3700762d0 0x7fc3700787e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc374034000 tx=0x7fc37402e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:16.815 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:16.820+0000 7fc36b7fe640 1 -- 172.21.15.1:0/349983839 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7fc384074ac0 con 0x7fc3841f67b0 2024-05-09T05:24:17.046 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.050+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fc384050870 con 0x7fc3841f67b0 2024-05-09T05:24:17.071 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.075+0000 7fc36b7fe640 1 -- 172.21.15.1:0/349983839 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v17) ==== 161+0+0 (secure 0 0 0) 0x7fc37c014030 con 0x7fc3841f67b0 2024-05-09T05:24:17.071 INFO:teuthology.orchestra.run.smithi001.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T05:24:17.073 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.077+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3700762d0 msgr2=0x7fc3700787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.077+0000 7fc389134640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3700762d0 0x7fc3700787e0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc374034000 tx=0x7fc37402e000 comp rx=0 tx=0).stop 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.077+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3841f67b0 msgr2=0x7fc3840ca830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.077+0000 7fc389134640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3841f67b0 0x7fc3840ca830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fc37c00cd80 tx=0x7fc37c00a650 comp rx=0 tx=0).stop 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.078+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 shutdown_connections 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.078+0000 7fc389134640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7fc3700762d0 0x7fc3700787e0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.078+0000 7fc389134640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7fc38415c9a0 0x7fc3841f2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.078+0000 7fc389134640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7fc3841f67b0 0x7fc3840ca830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.078+0000 7fc389134640 1 --2- 172.21.15.1:0/349983839 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7fc3840b4570 0x7fc3841e8090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.078+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 >> 172.21.15.1:0/349983839 conn(0x7fc38406de10 msgr2=0x7fc3840cfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.078+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 shutdown_connections 2024-05-09T05:24:17.074 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.078+0000 7fc389134640 1 -- 172.21.15.1:0/349983839 wait complete. 2024-05-09T05:24:17.089 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T05:24:17.141 INFO:tasks.ceph.mds.d.smithi196.stdout:starting mds.d at 2024-05-09T05:24:17.141 INFO:tasks.ceph.mds.b.smithi196.stdout:starting mds.b at 2024-05-09T05:24:17.142 INFO:tasks.ceph.mds.f.smithi196.stdout:starting mds.f at 2024-05-09T05:24:17.159 INFO:tasks.ceph.mds.c.smithi001.stdout:starting mds.c at 2024-05-09T05:24:17.159 INFO:tasks.ceph.mds.e.smithi001.stdout:starting mds.e at 2024-05-09T05:24:17.213 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.218+0000 7ff5fbb00640 1 Processor -- start 2024-05-09T05:24:17.214 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.218+0000 7ff5fbb00640 1 -- start start 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.219+0000 7ff5fbb00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40b4ec0 0x7ff5f40cf9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.219+0000 7ff5fbb00640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5f415c7b0 con 0x7ff5f4160460 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.219+0000 7ff5fbb00640 1 -- --> v1:172.21.15.196:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7ff5f40cca50 con 0x7ff5f40ce010 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.219+0000 7ff5fbb00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f415d0c0 con 0x7ff5f40b4ec0 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.219+0000 7ff5fa09d640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7ff5f4160460 0x7ff5f4167130 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37490/0 (socket says 172.21.15.1:37490) 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.219+0000 7ff5fa09d640 1 -- 172.21.15.1:0/625538539 learned_addr learned my addr 172.21.15.1:0/625538539 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.219+0000 7ff5f989c640 1 --2- 172.21.15.1:0/625538539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40b4ec0 0x7ff5f40cf9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.220+0000 7ff5f889a640 1 -- 172.21.15.1:0/625538539 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3018546146 0 0) 0x7ff5f415c7b0 con 0x7ff5f4160460 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.220+0000 7ff5f889a640 1 -- 172.21.15.1:0/625538539 --> v1:172.21.15.1:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5e0003520 con 0x7ff5f4160460 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.220+0000 7ff5f889a640 1 -- 172.21.15.1:0/625538539 <== mon.1 v1:172.21.15.196:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 4017699118 0 0) 0x7ff5f40cca50 con 0x7ff5f40ce010 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.220+0000 7ff5f889a640 1 -- 172.21.15.1:0/625538539 --> v1:172.21.15.196:6789/0 -- auth(proto 2 36 bytes epoch 0) -- 0x7ff5f415c7b0 con 0x7ff5f40ce010 2024-05-09T05:24:17.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.220+0000 7ff5f989c640 1 -- 172.21.15.1:0/625538539 >> v1:172.21.15.196:6789/0 conn(0x7ff5f40ce010 legacy=0x7ff5f415fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:17.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.220+0000 7ff5f989c640 1 -- 172.21.15.1:0/625538539 >> v1:172.21.15.1:6789/0 conn(0x7ff5f4160460 legacy=0x7ff5f4167130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:17.217 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.220+0000 7ff5f989c640 1 -- 172.21.15.1:0/625538539 --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f40ccd30 con 0x7ff5f40b4ec0 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5f989c640 1 --2- 172.21.15.1:0/625538539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40b4ec0 0x7ff5f40cf9d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00097d0 tx=0x7ff5f002a290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3685f7f250d531c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5f889a640 1 -- 172.21.15.1:0/625538539 <== mon.2 v2:172.21.15.196:3301/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0037070 con 0x7ff5f40b4ec0 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5f889a640 1 -- 172.21.15.1:0/625538539 <== mon.2 v2:172.21.15.196:3301/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff5f0032440 con 0x7ff5f40b4ec0 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5f889a640 1 -- 172.21.15.1:0/625538539 <== mon.2 v2:172.21.15.196:3301/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f0032810 con 0x7ff5f40b4ec0 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5fbb00640 1 -- 172.21.15.1:0/625538539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40b4ec0 msgr2=0x7ff5f40cf9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5fbb00640 1 --2- 172.21.15.1:0/625538539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40b4ec0 0x7ff5f40cf9d0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00097d0 tx=0x7ff5f002a290 comp rx=0 tx=0).stop 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5fbb00640 1 -- 172.21.15.1:0/625538539 shutdown_connections 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5fbb00640 1 --2- 172.21.15.1:0/625538539 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40b4ec0 0x7ff5f40cf9d0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:17.218 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.221+0000 7ff5fbb00640 1 -- 172.21.15.1:0/625538539 >> 172.21.15.1:0/625538539 conn(0x7ff5f40d2360 msgr2=0x7ff5f40ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.222+0000 7ff5fbb00640 1 -- 172.21.15.1:0/625538539 shutdown_connections 2024-05-09T05:24:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.222+0000 7ff5fbb00640 1 -- 172.21.15.1:0/625538539 wait complete. 2024-05-09T05:24:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.223+0000 7ff5fbb00640 1 Processor -- start 2024-05-09T05:24:17.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.223+0000 7ff5fbb00640 1 -- start start 2024-05-09T05:24:17.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.224+0000 7ff5fbb00640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40b4ec0 0x7ff5f41e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fbb00640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40ce010 0x7ff5f406e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fbb00640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5f4160460 0x7ff5f4151460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fbb00640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f40cc750 con 0x7ff5f4160460 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fbb00640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7ff5f415cd50 con 0x7ff5f40b4ec0 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fbb00640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7ff5f40cff10 con 0x7ff5f40ce010 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5f4160460 0x7ff5f4151460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5f4160460 0x7ff5f4151460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49972/0 (socket says 172.21.15.1:49972) 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 -- 172.21.15.1:0/3992458043 learned_addr learned my addr 172.21.15.1:0/3992458043 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5f989c640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40b4ec0 0x7ff5f41e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5f909b640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40ce010 0x7ff5f406e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 -- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40ce010 msgr2=0x7ff5f406e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40ce010 0x7ff5f406e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 -- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40b4ec0 msgr2=0x7ff5f41e7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:17.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40b4ec0 0x7ff5f41e7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 -- 172.21.15.1:0/3992458043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5f41f2690 con 0x7ff5f4160460 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5f989c640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40b4ec0 0x7ff5f41e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5f909b640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40ce010 0x7ff5f406e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fa09d640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5f4160460 0x7ff5f4151460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00ca80 tx=0x7ff5ec00cf40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/3992458043 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec00abd0 con 0x7ff5f4160460 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/3992458043 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7ff5ec00ad70 con 0x7ff5f4160460 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5f415f550 con 0x7ff5f4160460 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.225+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/3992458043 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5ec012640 con 0x7ff5f4160460 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.226+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5f41e75b0 con 0x7ff5f4160460 2024-05-09T05:24:17.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.226+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff5f415fa70 con 0x7ff5f4160460 2024-05-09T05:24:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.227+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/3992458043 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7ff5ec0127e0 con 0x7ff5f4160460 2024-05-09T05:24:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.228+0000 7ff5ea7fc640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d8076300 0x7ff5d8078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.228+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/3992458043 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6255+0+0 (secure 0 0 0) 0x7ff5ec097f50 con 0x7ff5f4160460 2024-05-09T05:24:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.228+0000 7ff5f989c640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d8076300 0x7ff5d8078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:17.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.229+0000 7ff5f989c640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d8076300 0x7ff5d8078810 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0034000 tx=0x7ff5f002e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:17.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.233+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/3992458043 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7ff5f415fa70 con 0x7ff5f4160460 2024-05-09T05:24:17.460 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:17.464+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7ff5f4050870 con 0x7ff5f4160460 2024-05-09T05:24:18.072 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.077+0000 7ff5ea7fc640 1 -- 172.21.15.1:0/3992458043 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v18) ==== 104+0+0 (secure 0 0 0) 0x7ff5ec061560 con 0x7ff5f4160460 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.079+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d8076300 msgr2=0x7ff5d8078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.079+0000 7ff5fbb00640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d8076300 0x7ff5d8078810 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0034000 tx=0x7ff5f002e000 comp rx=0 tx=0).stop 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.079+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5f4160460 msgr2=0x7ff5f4151460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.079+0000 7ff5fbb00640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5f4160460 0x7ff5f4151460 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff5ec00ca80 tx=0x7ff5ec00cf40 comp rx=0 tx=0).stop 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.080+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 shutdown_connections 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.080+0000 7ff5fbb00640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7ff5d8076300 0x7ff5d8078810 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.080+0000 7ff5fbb00640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7ff5f4160460 0x7ff5f4151460 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.080+0000 7ff5fbb00640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7ff5f40ce010 0x7ff5f406e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.080+0000 7ff5fbb00640 1 --2- 172.21.15.1:0/3992458043 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7ff5f40b4ec0 0x7ff5f41e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.080+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 >> 172.21.15.1:0/3992458043 conn(0x7ff5f40d2360 msgr2=0x7ff5f40b5530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.080+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 shutdown_connections 2024-05-09T05:24:18.076 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.080+0000 7ff5fbb00640 1 -- 172.21.15.1:0/3992458043 wait complete. 2024-05-09T05:24:18.091 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T05:24:18.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.223+0000 7f02ae8d9640 1 Processor -- start 2024-05-09T05:24:18.219 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.223+0000 7f02ae8d9640 1 -- start start 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02ae8d9640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f02a8072400 0x7f02a8154530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02ae8d9640 1 --2- >> v2:172.21.15.196:3300/0 conn(0x7f02a8070f00 0x7f02a8071310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02ae8d9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a80cae10 0x7f02a80cb220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02ae8d9640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f02a80cb760 con 0x7f02a8072400 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02ae8d9640 1 -- --> v2:172.21.15.196:3300/0 -- mon_getmap magic: 0 -- 0x7f02a80ca3c0 con 0x7f02a8070f00 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02ae8d9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f02a80cbf90 con 0x7f02a80cae10 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02a7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f02a8072400 0x7f02a8154530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02a7fff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f02a8072400 0x7f02a8154530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49984/0 (socket says 172.21.15.1:49984) 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.224+0000 7f02a7fff640 1 -- 172.21.15.1:0/3236000612 learned_addr learned my addr 172.21.15.1:0/3236000612 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02ace76640 1 --2- 172.21.15.1:0/3236000612 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a80cae10 0x7f02a80cb220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02a77fe640 1 --2- 172.21.15.1:0/3236000612 >> v2:172.21.15.196:3300/0 conn(0x7f02a8070f00 0x7f02a8071310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:18.221 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02a7fff640 1 -- 172.21.15.1:0/3236000612 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a80cae10 msgr2=0x7f02a80cb220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:18.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02a7fff640 1 --2- 172.21.15.1:0/3236000612 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a80cae10 0x7f02a80cb220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02a7fff640 1 -- 172.21.15.1:0/3236000612 >> v2:172.21.15.196:3300/0 conn(0x7f02a8070f00 msgr2=0x7f02a8071310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:18.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02a7fff640 1 --2- 172.21.15.1:0/3236000612 >> v2:172.21.15.196:3300/0 conn(0x7f02a8070f00 0x7f02a8071310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02a7fff640 1 -- 172.21.15.1:0/3236000612 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a815a780 con 0x7f02a8072400 2024-05-09T05:24:18.222 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02a77fe640 1 --2- 172.21.15.1:0/3236000612 >> v2:172.21.15.196:3300/0 conn(0x7f02a8070f00 0x7f02a8071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:24:18.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.225+0000 7f02a7fff640 1 --2- 172.21.15.1:0/3236000612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a8072400 0x7f02a8154530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f02a8073f10 tx=0x7f029802a1e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9db7c2fb9fe7be2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:18.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02a6ffd640 1 -- 172.21.15.1:0/3236000612 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298037070 con 0x7f02a8072400 2024-05-09T05:24:18.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02a6ffd640 1 -- 172.21.15.1:0/3236000612 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f0298032440 con 0x7f02a8072400 2024-05-09T05:24:18.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02a6ffd640 1 -- 172.21.15.1:0/3236000612 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0298032830 con 0x7f02a8072400 2024-05-09T05:24:18.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 -- 172.21.15.1:0/3236000612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a8072400 msgr2=0x7f02a8154530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:18.223 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/3236000612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a8072400 0x7f02a8154530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f02a8073f10 tx=0x7f029802a1e0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 -- 172.21.15.1:0/3236000612 shutdown_connections 2024-05-09T05:24:18.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/3236000612 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a80cae10 0x7f02a80cb220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/3236000612 >> v2:172.21.15.196:3300/0 conn(0x7f02a8070f00 0x7f02a8071310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/3236000612 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a8072400 0x7f02a8154530 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 -- 172.21.15.1:0/3236000612 >> 172.21.15.1:0/3236000612 conn(0x7f02a80d0470 msgr2=0x7f02a80cc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:18.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 -- 172.21.15.1:0/3236000612 shutdown_connections 2024-05-09T05:24:18.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.226+0000 7f02ae8d9640 1 -- 172.21.15.1:0/3236000612 wait complete. 2024-05-09T05:24:18.224 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.228+0000 7f02ae8d9640 1 Processor -- start 2024-05-09T05:24:18.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.228+0000 7f02ae8d9640 1 -- start start 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.229+0000 7f02ae8d9640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a8070f00 0x7f02a81515b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ae8d9640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02a8072400 0x7f02a8149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ae8d9640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a80cae10 0x7f02a8149a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ae8d9640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f02a80b2630 con 0x7f02a80cae10 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ae8d9640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f02a80ca8f0 con 0x7f02a8072400 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ae8d9640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f02a80b27b0 con 0x7f02a8070f00 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a80cae10 0x7f02a8149a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a80cae10 0x7f02a8149a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49988/0 (socket says 172.21.15.1:49988) 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 -- 172.21.15.1:0/1593274306 learned_addr learned my addr 172.21.15.1:0/1593274306 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:18.227 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02a7fff640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a8070f00 0x7f02a81515b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02a77fe640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02a8072400 0x7f02a8149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 -- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a8070f00 msgr2=0x7f02a81515b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a8070f00 0x7f02a81515b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 -- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02a8072400 msgr2=0x7f02a8149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02a8072400 0x7f02a8149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 -- 172.21.15.1:0/1593274306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02a814f4c0 con 0x7f02a80cae10 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ace76640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a80cae10 0x7f02a8149a50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f02a0007d70 tx=0x7f02a000e4b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02a4ff9640 1 -- 172.21.15.1:0/1593274306 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0014420 con 0x7f02a80cae10 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02a4ff9640 1 -- 172.21.15.1:0/1593274306 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f02a0019070 con 0x7f02a80cae10 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02a4ff9640 1 -- 172.21.15.1:0/1593274306 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02a0014840 con 0x7f02a80cae10 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.230+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02a814f760 con 0x7f02a80cae10 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.231+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02a8149f90 con 0x7f02a80cae10 2024-05-09T05:24:18.228 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.231+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02a8050870 con 0x7f02a80cae10 2024-05-09T05:24:18.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.232+0000 7f02a4ff9640 1 -- 172.21.15.1:0/1593274306 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f02a00149e0 con 0x7f02a80cae10 2024-05-09T05:24:18.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.233+0000 7f02a4ff9640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0294076300 0x7f0294078810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:18.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.233+0000 7f02a4ff9640 1 -- 172.21.15.1:0/1593274306 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(24..24 src has 1..24) ==== 6255+0+0 (secure 0 0 0) 0x7f02a0097f80 con 0x7f02a80cae10 2024-05-09T05:24:18.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.233+0000 7f02a7fff640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0294076300 0x7f0294078810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:18.229 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.234+0000 7f02a7fff640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0294076300 0x7f0294078810 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0298034000 tx=0x7f029802e000 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:18.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.237+0000 7f02a4ff9640 1 -- 172.21.15.1:0/1593274306 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f02a0060f40 con 0x7f02a80cae10 2024-05-09T05:24:18.424 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:18.428+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f02a80cbcd0 con 0x7f02a80cae10 2024-05-09T05:24:19.200 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.204+0000 7f02a4ff9640 1 -- 172.21.15.1:0/1593274306 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) ==== 181+0+0 (secure 0 0 0) 0x7f02a0065240 con 0x7f02a80cae10 2024-05-09T05:24:19.200 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_metadata' removed 2024-05-09T05:24:19.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0294076300 msgr2=0x7f0294078810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:19.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0294076300 0x7f0294078810 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f0298034000 tx=0x7f029802e000 comp rx=0 tx=0).stop 2024-05-09T05:24:19.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a80cae10 msgr2=0x7f02a8149a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:19.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a80cae10 0x7f02a8149a50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f02a0007d70 tx=0x7f02a000e4b0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 shutdown_connections 2024-05-09T05:24:19.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f0294076300 0x7f0294078810 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f02a8072400 0x7f02a8149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.203 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f02a80cae10 0x7f02a8149a50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 --2- 172.21.15.1:0/1593274306 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f02a8070f00 0x7f02a81515b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 >> 172.21.15.1:0/1593274306 conn(0x7f02a80d0470 msgr2=0x7f02a80c5290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:19.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 shutdown_connections 2024-05-09T05:24:19.204 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.207+0000 7f02ae8d9640 1 -- 172.21.15.1:0/1593274306 wait complete. 2024-05-09T05:24:19.218 DEBUG:teuthology.orchestra.run.smithi001:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T05:24:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.355+0000 7f32c1bc7640 1 Processor -- start 2024-05-09T05:24:19.351 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.355+0000 7f32c1bc7640 1 -- start start 2024-05-09T05:24:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32c1bc7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc0c7280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32c1bc7640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32bc0c77c0 0x7f32bc0c7bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32c1bc7640 1 -- --> v1:172.21.15.1:6789/0 -- auth(proto 0 30 bytes epoch 0) -- 0x7f32bc15c7b0 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32c1bc7640 1 -- --> v2:172.21.15.1:3300/0 -- mon_getmap magic: 0 -- 0x7f32bc15d0c0 con 0x7f32bc0c77c0 2024-05-09T05:24:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32c1bc7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f32bc0cca50 con 0x7f32bc0ce010 2024-05-09T05:24:19.353 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32bbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32bc0c77c0 0x7f32bc0c7bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32bb7fe640 1 --1- >> v1:172.21.15.1:6789/0 conn(0x7f32bc0b4ec0 0x7f32bc0cf9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.1:6789/0 says I am v1:172.21.15.1:37496/0 (socket says 172.21.15.1:37496) 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32bb7fe640 1 -- 172.21.15.1:0/1645995112 learned_addr learned my addr 172.21.15.1:0/1645995112 (peer_addr_for_me v1:172.21.15.1:0/0) 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32bbfff640 1 --2- >> v2:172.21.15.1:3300/0 conn(0x7f32bc0c77c0 0x7f32bc0c7bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:49994/0 (socket says 172.21.15.1:49994) 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32bbfff640 1 -- 172.21.15.1:0/1645995112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 msgr2=0x7f32bc0c7280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32baffd640 1 --2- 172.21.15.1:0/1645995112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc0c7280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32bbfff640 1 --2- 172.21.15.1:0/1645995112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc0c7280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32bbfff640 1 -- 172.21.15.1:0/1645995112 >> v1:172.21.15.1:6789/0 conn(0x7f32bc0b4ec0 legacy=0x7f32bc0cf9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32ba7fc640 1 -- 172.21.15.1:0/1645995112 <== mon.0 v1:172.21.15.1:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) ==== 33+0+0 (unknown 3104443899 0 0) 0x7f32bc15c7b0 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32bbfff640 1 -- 172.21.15.1:0/1645995112 --> v2:172.21.15.1:3300/0 -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32bc0ccd30 con 0x7f32bc0c77c0 2024-05-09T05:24:19.354 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.357+0000 7f32baffd640 1 --2- 172.21.15.1:0/1645995112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc0c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T05:24:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.358+0000 7f32bbfff640 1 --2- 172.21.15.1:0/1645995112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0c77c0 0x7f32bc0c7bd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009dc0 tx=0x7f32b402a710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f1d6227e7bb35e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.358+0000 7f32ba7fc640 1 -- 172.21.15.1:0/1645995112 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b4007740 con 0x7f32bc0c77c0 2024-05-09T05:24:19.355 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.358+0000 7f32ba7fc640 1 -- 172.21.15.1:0/1645995112 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f32b40078e0 con 0x7f32bc0c77c0 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.358+0000 7f32ba7fc640 1 -- 172.21.15.1:0/1645995112 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32b4007cd0 con 0x7f32bc0c77c0 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.359+0000 7f32c1bc7640 1 -- 172.21.15.1:0/1645995112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0c77c0 msgr2=0x7f32bc0c7bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.359+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/1645995112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0c77c0 0x7f32bc0c7bd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f32b4009dc0 tx=0x7f32b402a710 comp rx=0 tx=0).stop 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.359+0000 7f32c1bc7640 1 -- 172.21.15.1:0/1645995112 shutdown_connections 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.359+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/1645995112 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc0c7280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.359+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/1645995112 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0c77c0 0x7f32bc0c7bd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.359+0000 7f32c1bc7640 1 -- 172.21.15.1:0/1645995112 >> 172.21.15.1:0/1645995112 conn(0x7f32bc0d2360 msgr2=0x7f32bc0ce650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.359+0000 7f32c1bc7640 1 -- 172.21.15.1:0/1645995112 shutdown_connections 2024-05-09T05:24:19.356 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.359+0000 7f32c1bc7640 1 -- 172.21.15.1:0/1645995112 wait complete. 2024-05-09T05:24:19.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.361+0000 7f32c1bc7640 1 Processor -- start 2024-05-09T05:24:19.357 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.361+0000 7f32c1bc7640 1 -- start start 2024-05-09T05:24:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32c1bc7640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0b4ec0 0x7f32bc1f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32c1bc7640 1 --2- >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32bc0c77c0 0x7f32bc150aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32c1bc7640 1 --2- >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc14c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32c1bc7640 1 -- --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc0cc750 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32c1bc7640 1 -- --> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] -- mon_getmap magic: 0 -- 0x7f32bc15cd50 con 0x7f32bc0c77c0 2024-05-09T05:24:19.359 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32c1bc7640 1 -- --> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] -- mon_getmap magic: 0 -- 0x7f32bc0c8110 con 0x7f32bc0ce010 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32bb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0b4ec0 0x7f32bc1f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32bb7fe640 1 --2- >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0b4ec0 0x7f32bc1f5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.1:3300/0 says I am v2:172.21.15.1:50008/0 (socket says 172.21.15.1:50008) 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32bb7fe640 1 -- 172.21.15.1:0/812085084 learned_addr learned my addr 172.21.15.1:0/812085084 (peer_addr_for_me v2:172.21.15.1:0/0) 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32bbfff640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc14c890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.362+0000 7f32baffd640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32bc0c77c0 0x7f32bc150aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32bb7fe640 1 -- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 msgr2=0x7f32bc14c890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32bb7fe640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc14c890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32bb7fe640 1 -- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32bc0c77c0 msgr2=0x7f32bc150aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32bb7fe640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32bc0c77c0 0x7f32bc150aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32bb7fe640 1 -- 172.21.15.1:0/812085084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32bc15c7b0 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32bb7fe640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0b4ec0 0x7f32bc1f5b50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c410 tx=0x7f32ac00c8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32a3fff640 1 -- 172.21.15.1:0/812085084 <== mon.0 v2:172.21.15.1:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac0074e0 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32a3fff640 1 -- 172.21.15.1:0/812085084 <== mon.0 v2:172.21.15.1:3300/0 2 ==== config(1 keys) ==== 35+0+0 (secure 0 0 0) 0x7f32ac0047e0 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.360 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32a3fff640 1 -- 172.21.15.1:0/812085084 <== mon.0 v2:172.21.15.1:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32ac004bf0 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32bc150fe0 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.363+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32bc1f2900 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.361 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.364+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f32bc050870 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.365+0000 7f32a3fff640 1 -- 172.21.15.1:0/812085084 <== mon.0 v2:172.21.15.1:3300/0 4 ==== mgrmap(e 5) ==== 98201+0+0 (secure 0 0 0) 0x7f32ac011070 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.365+0000 7f32a3fff640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32a80762d0 0x7f32a80787e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T05:24:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.366+0000 7f32a3fff640 1 -- 172.21.15.1:0/812085084 <== mon.0 v2:172.21.15.1:3300/0 5 ==== osd_map(25..25 src has 1..25) ==== 5817+0+0 (secure 0 0 0) 0x7f32ac0972e0 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.366+0000 7f32baffd640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32a80762d0 0x7f32a80787e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T05:24:19.362 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.366+0000 7f32baffd640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32a80762d0 0x7f32a80787e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f32bc153810 tx=0x7f32b0008040 comp rx=0 tx=0).ready entity=mgr.4099 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T05:24:19.366 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.370+0000 7f32a3fff640 1 -- 172.21.15.1:0/812085084 <== mon.0 v2:172.21.15.1:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+190839 (secure 0 0 0) 0x7f32ac060450 con 0x7f32bc0b4ec0 2024-05-09T05:24:19.556 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:19.560+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 --> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7f32bc0cd2b0 con 0x7f32bc0b4ec0 2024-05-09T05:24:20.207 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.211+0000 7f32a3fff640 1 -- 172.21.15.1:0/812085084 <== mon.0 v2:172.21.15.1:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) ==== 169+0+0 (secure 0 0 0) 0x7f32ac064750 con 0x7f32bc0b4ec0 2024-05-09T05:24:20.207 INFO:teuthology.orchestra.run.smithi001.stderr:pool 'cephfs_data' removed 2024-05-09T05:24:20.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.220+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32a80762d0 msgr2=0x7f32a80787e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:20.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.220+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32a80762d0 0x7f32a80787e0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f32bc153810 tx=0x7f32b0008040 comp rx=0 tx=0).stop 2024-05-09T05:24:20.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.220+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0b4ec0 msgr2=0x7f32bc1f5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T05:24:20.216 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.220+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0b4ec0 0x7f32bc1f5b50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f32ac00c410 tx=0x7f32ac00c8d0 comp rx=0 tx=0).stop 2024-05-09T05:24:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.228+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 shutdown_connections 2024-05-09T05:24:20.225 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.228+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:6832/1287021241,v1:172.21.15.196:6833/1287021241] conn(0x7f32a80762d0 0x7f32a80787e0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.228+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:3300/0,v1:172.21.15.196:6789/0] conn(0x7f32bc0c77c0 0x7f32bc150aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.228+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.196:3301/0,v1:172.21.15.196:6790/0] conn(0x7f32bc0ce010 0x7f32bc14c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.228+0000 7f32c1bc7640 1 --2- 172.21.15.1:0/812085084 >> [v2:172.21.15.1:3300/0,v1:172.21.15.1:6789/0] conn(0x7f32bc0b4ec0 0x7f32bc1f5b50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T05:24:20.226 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.228+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 >> 172.21.15.1:0/812085084 conn(0x7f32bc0d2360 msgr2=0x7f32bc154890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T05:24:20.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.231+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 shutdown_connections 2024-05-09T05:24:20.233 INFO:teuthology.orchestra.run.smithi001.stderr:2024-05-09T05:24:20.238+0000 7f32c1bc7640 1 -- 172.21.15.1:0/812085084 wait complete. 2024-05-09T05:24:20.254 INFO:teuthology.misc:Shutting down mds daemons... 2024-05-09T05:24:20.255 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2024-05-09T05:24:20.255 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.270 INFO:tasks.ceph.mds.a:Stopped 2024-05-09T05:24:20.270 DEBUG:tasks.ceph.mds.c:waiting for process to exit 2024-05-09T05:24:20.270 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.330 INFO:tasks.ceph.mds.c:Stopped 2024-05-09T05:24:20.330 DEBUG:tasks.ceph.mds.e:waiting for process to exit 2024-05-09T05:24:20.330 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.384 INFO:tasks.ceph.mds.e:Stopped 2024-05-09T05:24:20.385 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2024-05-09T05:24:20.385 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.399 INFO:tasks.ceph.mds.b:Stopped 2024-05-09T05:24:20.399 DEBUG:tasks.ceph.mds.d:waiting for process to exit 2024-05-09T05:24:20.399 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.454 INFO:tasks.ceph.mds.d:Stopped 2024-05-09T05:24:20.454 DEBUG:tasks.ceph.mds.f:waiting for process to exit 2024-05-09T05:24:20.454 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.507 INFO:tasks.ceph.mds.f:Stopped 2024-05-09T05:24:20.507 INFO:teuthology.misc:Shutting down osd daemons... 2024-05-09T05:24:20.508 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2024-05-09T05:24:20.508 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.556 INFO:tasks.ceph.osd.0:Stopped 2024-05-09T05:24:20.556 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2024-05-09T05:24:20.556 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.611 INFO:tasks.ceph.osd.1:Stopped 2024-05-09T05:24:20.612 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2024-05-09T05:24:20.612 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.672 INFO:tasks.ceph.osd.2:Stopped 2024-05-09T05:24:20.673 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2024-05-09T05:24:20.673 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.728 INFO:tasks.ceph.osd.3:Stopped 2024-05-09T05:24:20.728 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2024-05-09T05:24:20.728 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.800 INFO:tasks.ceph.osd.4:Stopped 2024-05-09T05:24:20.801 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2024-05-09T05:24:20.801 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.856 INFO:tasks.ceph.osd.5:Stopped 2024-05-09T05:24:20.857 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2024-05-09T05:24:20.857 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.918 INFO:tasks.ceph.osd.6:Stopped 2024-05-09T05:24:20.918 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2024-05-09T05:24:20.918 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:20.979 INFO:tasks.ceph.osd.7:Stopped 2024-05-09T05:24:20.979 INFO:teuthology.misc:Shutting down mgr daemons... 2024-05-09T05:24:20.979 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2024-05-09T05:24:20.980 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:21.006 INFO:tasks.ceph.mgr.y:Stopped 2024-05-09T05:24:21.007 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2024-05-09T05:24:21.007 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:21.034 INFO:tasks.ceph.mgr.x:Stopped 2024-05-09T05:24:21.034 INFO:teuthology.misc:Shutting down mon daemons... 2024-05-09T05:24:21.034 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2024-05-09T05:24:21.035 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:21.059 INFO:tasks.ceph.mon.a:Stopped 2024-05-09T05:24:21.060 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2024-05-09T05:24:21.060 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:21.086 INFO:tasks.ceph.mon.b:Stopped 2024-05-09T05:24:21.086 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2024-05-09T05:24:21.086 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T05:24:21.136 INFO:tasks.ceph.mon.c:Stopped 2024-05-09T05:24:21.137 INFO:tasks.ceph:Checking cluster log for badness... 2024-05-09T05:24:21.137 DEBUG:teuthology.orchestra.run.smithi001:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'do not have an application enabled' | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'Degraded data redundancy' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(POOL_' | egrep -v '\(CACHE_POOL_' | egrep -v '\(SMALLER_PGP_NUM\)' | egrep -v '\(OBJECT_' | egrep -v '\(SLOW_OPS\)' | egrep -v '\(REQUEST_SLOW\)' | egrep -v '\(TOO_FEW_PGS\)' | egrep -v 'slow request' | egrep -v 'timeout on replica' | egrep -v 'late reservation from' | egrep -v MON_DOWN | egrep -v OSDMAP_FLAGS | egrep -v OSD_DOWN | egrep -v PG_DEGRADED | egrep -v PG_AVAILABILITY | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'mons down' | egrep -v 'mon down' | egrep -v 'out of quorum' | egrep -v 'overall HEALTH_' | egrep -v '\(MON_DOWN\)' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-09T05:24:21.224 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:24:21.224 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2024-05-09T05:24:21.588 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:24:21.589 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2024-05-09T05:24:21.668 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:24:21.668 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2024-05-09T05:24:21.751 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:24:21.751 DEBUG:teuthology.orchestra.run.smithi001:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2024-05-09T05:24:21.850 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:24:21.850 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2024-05-09T05:24:22.104 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:24:22.104 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2024-05-09T05:24:22.223 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:24:22.223 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2024-05-09T05:24:22.295 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:24:22.295 DEBUG:teuthology.orchestra.run.smithi196:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2024-05-09T05:24:22.393 INFO:tasks.ceph:Archiving mon data... 2024-05-09T05:24:22.395 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/data/mon.a.tgz 2024-05-09T05:24:22.395 DEBUG:teuthology.orchestra.run.smithi001:> mktemp 2024-05-09T05:24:22.417 INFO:teuthology.orchestra.run.smithi001.stdout:/tmp/tmp.kxyo1Z7mcD 2024-05-09T05:24:22.417 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.kxyo1Z7mcD 2024-05-09T05:24:22.572 DEBUG:teuthology.orchestra.run.smithi001:> sudo chmod 0666 /tmp/tmp.kxyo1Z7mcD 2024-05-09T05:24:22.676 DEBUG:teuthology.orchestra.remote:smithi001:/tmp/tmp.kxyo1Z7mcD is 400KB 2024-05-09T05:24:22.776 DEBUG:teuthology.orchestra.run.smithi001:> rm -fr /tmp/tmp.kxyo1Z7mcD 2024-05-09T05:24:22.794 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/data/mon.b.tgz 2024-05-09T05:24:22.794 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-05-09T05:24:22.811 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.5QPZ40rlWI 2024-05-09T05:24:22.811 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.5QPZ40rlWI 2024-05-09T05:24:22.954 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.5QPZ40rlWI 2024-05-09T05:24:23.039 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.5QPZ40rlWI is 402KB 2024-05-09T05:24:23.139 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.5QPZ40rlWI 2024-05-09T05:24:23.154 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/data/mon.c.tgz 2024-05-09T05:24:23.154 DEBUG:teuthology.orchestra.run.smithi196:> mktemp 2024-05-09T05:24:23.209 INFO:teuthology.orchestra.run.smithi196.stdout:/tmp/tmp.fXxov8tfN4 2024-05-09T05:24:23.209 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.fXxov8tfN4 2024-05-09T05:24:23.340 DEBUG:teuthology.orchestra.run.smithi196:> sudo chmod 0666 /tmp/tmp.fXxov8tfN4 2024-05-09T05:24:23.423 DEBUG:teuthology.orchestra.remote:smithi196:/tmp/tmp.fXxov8tfN4 is 402KB 2024-05-09T05:24:23.526 DEBUG:teuthology.orchestra.run.smithi196:> rm -fr /tmp/tmp.fXxov8tfN4 2024-05-09T05:24:23.541 INFO:tasks.ceph:Cleaning ceph cluster... 2024-05-09T05:24:23.541 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T05:24:23.544 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2024-05-09T05:24:23.638 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T05:24:23.660 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-05-09T05:24:23.660 INFO:tasks.ceph:Archiving crash dumps... 2024-05-09T05:24:23.662 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/remote/smithi001/crash 2024-05-09T05:24:23.662 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T05:24:23.707 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/lib/ceph/crash to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/remote/smithi196/crash 2024-05-09T05:24:23.708 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-05-09T05:24:23.732 INFO:tasks.ceph:Shutting down logrotate 2024-05-09T05:24:23.732 DEBUG:teuthology.orchestra.run.smithi001:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:24:23.752 DEBUG:teuthology.orchestra.run.smithi196:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2024-05-09T05:24:23.828 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-09T05:24:23.862 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm /etc/logrotate.d/ceph-test.conf 2024-05-09T05:24:23.887 INFO:tasks.ceph:Compressing logs... 2024-05-09T05:24:23.888 DEBUG:teuthology.orchestra.run.smithi001:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T05:24:23.905 DEBUG:teuthology.orchestra.run.smithi196:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T05:24:23.937 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.29394.log 2024-05-09T05:24:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64252.log 2024-05-09T05:24:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.tmp-client.admin.29394.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40817.log 2024-05-09T05:24:23.938 INFO:teuthology.orchestra.run.smithi001.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.29394.log.gz 2024-05-09T05:24:23.938 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.95529.log 2024-05-09T05:24:23.971 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35659.log 2024-05-09T05:24:23.971 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T05:24:23.971 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35659.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.f.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.35659.log.gz 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph.audit.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35528.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.7.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35528.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.b.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.35528.log.gz 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.5.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35619.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35739.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35619.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.4.log 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.35619.log.gz 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35739.log: 94.0% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T05:24:23.972 INFO:teuthology.orchestra.run.smithi196.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.35739.log.gz 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35572.log 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35699.log 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35572.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35448.log 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.35572.log.gz 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35699.log: gzip -5 --verbose -- /var/log/ceph/ceph-mgr.x.log 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35448.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.35699.log.gz 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.35448.log.gz 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.c.log 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr: 92.6%gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35488.log 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr: -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-mgr.x.log: /var/log/ceph/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.6.log 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-client.admin.35488.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.35488.log.gz 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mds.d.log 2024-05-09T05:24:23.973 INFO:teuthology.orchestra.run.smithi196.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-mon.b.log 2024-05-09T05:24:24.035 INFO:teuthology.orchestra.run.smithi196.stderr:/var/log/ceph/ceph-osd.6.log: /var/log/ceph/ceph-mds.d.log: /var/log/ceph/ceph-mon.b.log: 93.4% -- replaced with /var/log/ceph/ceph-mds.b.log.gz 2024-05-09T05:24:24.067 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64252.log: /var/log/ceph/ceph-client.admin.40817.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110129.log 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.64252.log.gz 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.95529.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44001.log 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110129.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.95529.log.gz 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59113.log 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110129.log.gz 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44001.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41125.log 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41442.log 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41125.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.59113.log.gz 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50521.log 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41442.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52494.log 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50521.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44230.log 2024-05-09T05:24:24.068 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.44001.log.gz 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49756.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44230.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42119.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42304.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50605.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42304.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44799.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49199.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44799.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50008.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49199.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110477.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46508.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50008.log: /var/log/ceph/ceph-client.admin.110477.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51273.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46508.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110477.log.gz 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.82904.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51273.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43651.log 2024-05-09T05:24:24.069 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.82904.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55864.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43651.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.82904.log.gz 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99518.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55864.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49612.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.99518.log: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.55864.log.gz 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.52620.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.99518.log.gz 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49612.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40691.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52077.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40691.log: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52620.log.gz 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44841.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42430.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44841.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41454.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42430.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45513.log 2024-05-09T05:24:24.070 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41454.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106675.log 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45513.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41114.log 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.106675.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109586.log 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41114.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53961.log 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.106675.log.gz 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109586.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48575.log 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53961.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.109586.log.gz 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102188.log 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53961.log.gz/var/log/ceph/ceph-client.admin.48575.log: 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.40817.log.gz 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41657.log 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44230.log.gz 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43651.log.gz 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.102188.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87754.log 2024-05-09T05:24:24.071 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.45513.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41114.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41657.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.102188.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43751.log 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40691.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41125.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87754.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53090.log 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.41442.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41454.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.87754.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43751.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43567.log 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.48575.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.41657.log.gz 2024-05-09T05:24:24.072 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53090.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.59031.log 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.53090.log.gz 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43567.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79715.log 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.59031.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35674.log 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.59031.log.gz 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53588.log 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35674.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109730.log 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.79715.log.gz 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53588.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108577.log 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53588.log.gz 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109730.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50665.log 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109730.log.gz 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43751.log.gz 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.108577.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42803.log 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50665.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.108577.log.gz 2024-05-09T05:24:24.073 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111208.log 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50605.log.gz 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50918.log 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42803.log: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.43567.log.gz 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111208.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110729.log 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.111208.log.gz 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50918.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.56022.log 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.50665.log.gz 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110729.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.0.43333.log 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110729.log.gz 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56022.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111288.log 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.56022.log.gz 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.0.43333.log: 91.1%gzip -- replaced with /var/log/ceph/ceph-client.admin.42803.log.gz 2024-05-09T05:24:24.074 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.49451.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111288.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50352.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.111288.log.gz 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49451.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99745.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50352.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48617.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.99745.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49013.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48617.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.99745.log.gz 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92447.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44398.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49013.log: /var/log/ceph/ceph-client.admin.92447.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40353.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50521.log.gz 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44398.log: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.92447.log.gz 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40353.log: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.51273.log.gz 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54411.log 2024-05-09T05:24:24.075 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.48617.log.gz 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109544.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54411.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49924.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109544.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54411.log.gzgzip 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.108694.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49924.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.109544.log.gz 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49531.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.108694.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41121.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40353.log.gz 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49531.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.108694.log.gz 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97273.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50437.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41121.log: gzip -5/var/log/ceph/ceph-client.admin.97273.log: --verbose -- /var/log/ceph/ceph-client.admin.54063.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50437.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66434.log 2024-05-09T05:24:24.076 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.97273.log.gz 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5/var/log/ceph/ceph-client.admin.54063.log: --verbose -- /var/log/ceph/ceph-client.admin.66855.log 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66434.log: gzip 88.7% -- replaced with /var/log/ceph/ceph-client.admin.54063.log.gz -5 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.51003.log 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.66434.log.gz 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100333.log 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.66855.log.gz 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51003.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52203.log 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.100333.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.2.log 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr: 87.3% -- replaced with /var/log/ceph/ceph.log.gz 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52203.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54930.log 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.100333.log.gz 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.41121.log.gz 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41461.log 2024-05-09T05:24:24.077 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.2.log: /var/log/ceph/ceph-client.admin.54930.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48971.log 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54930.log.gz 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41461.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49798.log 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.49199.log.gz 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48971.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76716.log 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109502.log 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr: 93.5% -- replaced with /var/log/ceph/ceph-client.admin.49612.log.gz 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51315.log 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52494.log.gz 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76716.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51003.log.gz 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109502.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52413.log 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.76716.log.gz 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.109502.log.gz 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51315.log: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.41461.log.gz 2024-05-09T05:24:24.078 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51620.log 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52413.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89815.log 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51620.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44757.log 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89815.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52119.log 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43483.log 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44757.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.89815.log.gz 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52119.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53048.log 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43483.log: gzip -5 --verbose 88.8% -- /var/log/ceph/ceph-client.admin.51417.log -- replaced with /var/log/ceph/ceph-client.admin.49531.log.gz 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53048.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49756.log.gz 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 88.8% --verbose -- replaced with /var/log/ceph/ceph-client.admin.49798.log.gz -- 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph.audit.log 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53048.log.gz 2024-05-09T05:24:24.079 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51417.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110645.log 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66671.log 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110645.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51315.log.gz 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51782.log 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.50918.log.gz 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66671.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.110645.log.gz 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51147.log 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51782.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51620.log.gz 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.43483.log.gz 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54888.log 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.66671.log.gz 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.51417.log.gz 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54285.log 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51147.log: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52077.log.gz 2024-05-09T05:24:24.080 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44170.log 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54888.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.52203.log.gz 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52119.log.gz 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54285.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50563.log 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54888.log.gz 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54285.log.gz 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44170.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110603.log 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50563.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109904.log 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110603.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110897.log 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109904.log: gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.51501.log -- replaced with /var/log/ceph/ceph-client.admin.110603.log.gz 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110897.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41795.log 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.109904.log.gz 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110897.log.gz 2024-05-09T05:24:24.081 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48701.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41795.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109400.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48701.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54846.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109400.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53132.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54846.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.109400.log.gz 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49966.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4%gzip -- replaced with /var/log/ceph/ceph-client.admin.54846.log.gz -5 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-client.admin.52035.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53132.log: /var/log/ceph/ceph-client.admin.49966.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53318.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52035.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48323.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.53132.log.gz 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53318.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51884.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.53318.log.gz 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48323.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76980.log 2024-05-09T05:24:24.082 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52922.log 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76980.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109846.log 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% -- replaced with /var/log/ceph/ceph-client.admin.76980.log.gz 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.44170.log.gz 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52922.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40267.log 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52922.log.gz 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49696.log 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51147.log.gz 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109846.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48701.log.gz 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40267.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48887.log 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109846.log.gz 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49696.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42346.log 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48887.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.3.log 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.35674.log.gz 2024-05-09T05:24:24.083 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42346.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46863.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.51501.log.gz 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.68265.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46863.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50707.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.68265.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42536.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.52035.log.gz 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50707.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.40267.log.gz 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41124.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr: 92.1% -- replaced with /var/log/ceph/ceph-client.admin.68265.log.gz 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111330.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49073.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111330.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52578.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.111330.log.gz 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49073.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74571.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54105.log 2024-05-09T05:24:24.084 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.74571.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.44441.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.52578.log.gz 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43525.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.74571.log.gz 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.54105.log.gz 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42169.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43525.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42905.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42169.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76793.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42905.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50310.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76793.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44585.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50310.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41117.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.76793.log.gz 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44585.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48929.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41117.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44631.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48929.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49654.log 2024-05-09T05:24:24.085 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44631.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50833.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50050.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49654.log: /var/log/ceph/ceph-client.admin.50833.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110771.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50050.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89747.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110771.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49241.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89747.log: gzip 88.5% -5 --verbose -- replaced with /var/log/ceph/ceph-client.admin.110771.log.gz -- 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.76561.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.89747.log.gz 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49241.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97441.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.76561.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54804.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.97441.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45237.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.76561.log.gz 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.97441.log.gz 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54804.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41118.log 2024-05-09T05:24:24.086 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45237.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.c.log 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54804.log.gz 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41795.log.gz 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41118.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51189.log 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42211.log 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose/var/log/ceph/ceph-client.admin.51189.log: -- /var/log/ceph/ceph-client.admin.51459.log 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42211.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41792.log 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89884.log 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74410.log 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41792.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.42905.log.gz 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89884.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.1.log 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.43525.log.gz 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74410.log: gzip 89.4% -5 -- replaced with /var/log/ceph/ceph-client.admin.89884.log.gz --verbose 2024-05-09T05:24:24.087 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-mgr.y.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41458.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.74410.log.gz 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44543.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42494.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41458.log: /var/log/ceph/ceph-client.admin.44543.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99366.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42494.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51926.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.99366.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43832.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.96055.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.99366.log.gz 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51926.log: /var/log/ceph/ceph-client.admin.43832.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110813.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.96055.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49283.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110813.log: gzip -5 --verbose -- /var/log/ceph/ceph-osd.0.log 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr: 91.7% -- replaced with /var/log/ceph/ceph-client.admin.96055.log.gz 2024-05-09T05:24:24.088 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110813.log.gz 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49283.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.58343.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55943.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.58343.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42077.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.58343.log.gz 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.55943.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.87380.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42077.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.83641.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.55943.log.gz 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87380.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.90622.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.87380.log.gz 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.83641.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110264.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109772.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.90622.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.83641.log.gz 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110264.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110561.log 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.90622.log.gz 2024-05-09T05:24:24.089 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110264.log.gz 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109460.log 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109772.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.109772.log.gz 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110350.log 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110561.log: /var/log/ceph/ceph-client.admin.109460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49325.log 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110561.log.gz 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.109460.log.gz 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49451.log.gz 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110350.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42035.log 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.110350.log.gz 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49325.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50352.log.gz 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53672.log 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr: 90.3% -- replaced with /var/log/ceph/ceph-client.admin.50707.log.gz 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46927.log 2024-05-09T05:24:24.090 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42035.log: 89.7% -- replaced with /var/log/ceph/ceph.audit.log.gz 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53672.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46978.log 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.41118.log.gz 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46927.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.41117.log.gz 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.53672.log.gz 87.5% 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.41124.log.gz 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54537.log 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110222.log 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.46978.log: /var/log/ceph/ceph-client.admin.54537.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41458.log.gz 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52329.log 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110222.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44086.log 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.54537.log.gz 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52329.log: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.110222.log.gz 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40607.log 2024-05-09T05:24:24.091 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.41792.log.gz 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49115.log 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44086.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.49325.log.gz 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53444.log 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40607.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.50310.log.gz 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.49654.log.gz 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50833.log.gz 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49115.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54147.log 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40309.log 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.53444.log.gz 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54147.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53216.log 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.54147.log.gz 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40309.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44128.log 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53216.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.57616.log 2024-05-09T05:24:24.092 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44128.log: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53216.log.gz 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41457.log 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.46863.log.gz 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.57616.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97603.log 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.57616.log.gz 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41457.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53630.log 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41785.log 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.97603.log: /var/log/ceph/ceph-client.admin.53630.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52964.log 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.97603.log.gz 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose 89.7% -- -- replaced with /var/log/ceph/ceph-client.admin.53630.log.gz 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr: /var/log/ceph/ceph-client.admin.56843.log 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41785.log: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.51459.log.gz 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52964.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52704.log 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.56843.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.52964.log.gz 2024-05-09T05:24:24.093 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44272.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.56843.log.gz 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52704.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74684.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44272.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40565.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr: 90.8% -- replaced with /var/log/ceph/ceph-client.admin.52704.log.gz 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.45237.log.gz/var/log/ceph/ceph-client.admin.74684.log: 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53402.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.74684.log.gz 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41460.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40565.log: /var/log/ceph/ceph-client.admin.53402.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51105.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41460.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64099.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.53402.log.gz 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91283.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51105.log: gzip/var/log/ceph/ceph-client.admin.64099.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.35408.log 2024-05-09T05:24:24.094 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.91283.log: gzip -5 --verbose -- /var/log/ceph/ceph-mon.a.log 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.64099.log.gz 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.91283.log.gz 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79038.log 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35408.log: /var/log/ceph/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43875.log 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.79038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105324.log 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43875.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106743.log 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.79038.log.gz 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111166.log 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.105324.log: /var/log/ceph/ceph-client.admin.106743.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53486.log 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.105324.log.gz 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111166.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64019.log 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.106743.log.gz 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.46978.log.gz 2024-05-09T05:24:24.095 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53486.log: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.49073.log.gz 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.111166.log.gz 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- 89.0% /var/log/ceph/ceph-client.admin.106506.log 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.49115.log.gz 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 88.7% -5 -- replaced with /var/log/ceph/ceph-client.admin.53486.log.gz --verbose 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr: -- /var/log/ceph/ceph-client.admin.41789.log 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64019.log: /var/log/ceph/ceph-client.admin.106506.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50395.log 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.64019.log.gz 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.106506.log.gz 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41789.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54327.log 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50395.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54744.log 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54327.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74262.log 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54744.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.54327.log.gz 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109358.log 2024-05-09T05:24:24.096 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54744.log.gz 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40775.log 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr: 87.2% -- replaced with /var/log/ceph/ceph-client.admin.74262.log.gz 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109358.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110038.log 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.109358.log.gz/var/log/ceph/ceph-client.admin.40775.log: 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54453.log 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110038.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54231.log 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54453.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110038.log.gz 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.105868.log 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.54453.log.gz 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97687.log 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.105868.log: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.54231.log.gz 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54681.log 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48323.log.gz 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.97687.log: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.105868.log.gz 2024-05-09T05:24:24.097 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89605.log 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.97687.log.gz 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.49966.log.gz 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54681.log: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.50008.log.gz 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52287.log 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50050.log.gz 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7%/var/log/ceph/ceph-client.admin.89605.log: -- replaced with /var/log/ceph/ceph-client.admin.54681.log.gz 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109628.log 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52287.log: gzip -5 --verbose -- 88.7% /var/log/ceph/ceph-client.admin.44883.log 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.89605.log.gz 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109628.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.50437.log.gz 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110172.log 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.109628.log.gz 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.91896.log 2024-05-09T05:24:24.098 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44883.log: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.51189.log.gz 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.51105.log.gz 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose --/var/log/ceph/ceph-client.admin.110172.log: /var/log/ceph/ceph-client.admin.53756.log 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr: 91.3% -- replaced with /var/log/ceph/ceph-client.admin.52329.log.gz 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.91896.log: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110172.log.gz 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108779.log 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53756.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50791.log 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.91896.log.gz 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.108779.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.77670.log 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr: 91.5% -- replaced with /var/log/ceph/ceph-client.admin.53756.log.gz 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.108779.log.gz 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50791.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106819.log 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110855.log 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.77670.log: /var/log/ceph/ceph-client.admin.106819.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99676.log 2024-05-09T05:24:24.099 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.77670.log.gz 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.106819.log.gz 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110855.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50749.log 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110855.log.gz 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.99676.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54597.log 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.99676.log.gz 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.50749.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.48449.log 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54597.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41444.log 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48449.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41123.log 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.54597.log.gz 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41444.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110435.log 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53798.log 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41123.log: /var/log/ceph/ceph-client.admin.110435.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.92521.log 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr: 90.3% 88.3% -- replaced with /var/log/ceph/ceph-client.admin.110435.log.gz 2024-05-09T05:24:24.100 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53798.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48533.log 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr: -- replaced with /var/log/ceph/ceph-client.admin.46927.log.gz 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.49241.log.gz 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.92521.log: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.49283.log.gz 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.53798.log.gz 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109178.log 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48533.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.92521.log.gz 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43917.log 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109178.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111124.log 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.43917.log: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.109178.log.gz 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40649.log 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111124.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52245.log 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40649.log: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.111124.log.gz 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.35777.log 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52245.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52837.log 2024-05-09T05:24:24.101 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.35777.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50139.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52837.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48491.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9%/var/log/ceph/ceph-client.admin.50139.log: -- replaced with /var/log/ceph/ceph-client.admin.52837.log.gz 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40207.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48491.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41796.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.42035.log.gz 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42077.log.gz 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40207.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.78307.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41796.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44501.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.78307.log: gzip -5 --verbose -- /var/log/ceph/ceph-mds.e.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44501.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52371.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.78307.log.gz 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42605.log 2024-05-09T05:24:24.102 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52371.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.46809.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66775.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42605.log: /var/log/ceph/ceph-client.admin.46809.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41459.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66775.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52746.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41459.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54639.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.66775.log.gz 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52746.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.97526.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:gzip/var/log/ceph/ceph-client.admin.54639.log: -5 --verbose -- /var/log/ceph/ceph-client.admin.41113.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.52746.log.gz 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.97526.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50479.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.54639.log.gz 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.97526.log.gz 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41113.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111082.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50479.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44314.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111082.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40901.log 2024-05-09T05:24:24.103 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.111082.log.gz 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87605.log 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44314.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42737.log 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40901.log: /var/log/ceph/ceph-client.admin.87605.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108849.log 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110687.log 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.87605.log.gz 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42737.log: /var/log/ceph/ceph-client.admin.108849.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.101571.log 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110519.log 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110687.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.108849.log.gz 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.101571.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109274.log 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.110687.log.gz 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110519.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109316.log 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr: 90.4% -- replaced with /var/log/ceph/ceph-client.admin.101571.log.gz 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.110519.log.gz 2024-05-09T05:24:24.104 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109274.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48407.log 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109274.log.gz 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.109316.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.45555.log 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48407.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52161.log 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.43832.log.gz 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.43875.log.gz 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr: 91.8% 89.9% -- replaced with /var/log/ceph/ceph-client.admin.43917.log.gz -- replaced with /var/log/ceph/ceph-client.admin.109316.log.gz 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.45555.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50876.log 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52161.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53840.log 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50876.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50961.log 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53840.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54021.log 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50961.log: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.53840.log.gz 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87678.log 2024-05-09T05:24:24.105 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54021.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51375.log 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.54021.log.gz 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87678.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53360.log 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.44086.log.gz 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.87678.log.gz 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42262.log 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51375.log: 91.6% -- replaced with /var/log/ceph/ceph-client.admin.44128.log.gz 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53360.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52880.log 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr: 92.3% -- replaced with /var/log/ceph/ceph-client.admin.35408.log.gz 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.53360.log.gz 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42262.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54972.log 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52880.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40481.log 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.52880.log.gz 2024-05-09T05:24:24.106 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54972.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54495.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40481.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.111040.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.54972.log.gz 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54495.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.74484.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50749.log.gz 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.111040.log: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.54495.log.gz 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48365.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.111040.log.gz 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44673.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.74484.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48761.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48365.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.74484.log.gz 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44673.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51662.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48761.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.73574.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.42388.log 2024-05-09T05:24:24.107 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.73574.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.67595.log 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42388.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.76884.log 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.73574.log.gz 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53006.log 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.67595.log: /var/log/ceph/ceph-client.admin.76884.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.89960.log 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.67595.log.gz 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53006.log: gzip -5 --verbose 89.4% -- -- replaced with /var/log/ceph/ceph-client.admin.76884.log.gz /var/log/ceph/ceph-client.admin.44356.log 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.53006.log.gz 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52536.log 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.89960.log: /var/log/ceph/ceph-client.admin.44356.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48659.log 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.89960.log.gz 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40207.log.gz 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52536.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49409.log 2024-05-09T05:24:24.108 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5/var/log/ceph/ceph-client.admin.48659.log: --verbose -- /var/log/ceph/ceph-client.admin.49882.log 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49409.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53276.log 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110998.log 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.40309.log.gz 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49882.log: /var/log/ceph/ceph-client.admin.53276.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54189.log 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.53276.log.gz 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110998.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40105.log 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54189.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.110998.log.gzgzip 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr: -5 --verbose -- /var/log/ceph/ceph-client.admin.51824.log 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.42211.log.gz 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40105.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41794.log 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.54189.log.gz 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.42304.log.gz 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42346.log.gz 2024-05-09T05:24:24.109 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.42430.log.gz 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5/var/log/ceph/ceph-client.admin.51824.log: --verbose -- /var/log/ceph/ceph-client.admin.50223.log 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41794.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49157.log 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50223.log: 90.7% -- replaced with /var/log/ceph/ceph-client.admin.42536.log.gz 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69860.log 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.42494.log.gz 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.109670.log 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49157.log: /var/log/ceph/ceph-client.admin.69860.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63860.log 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40481.log.gz 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:gzip 88.4%/var/log/ceph/ceph-client.admin.109670.log: -- replaced with /var/log/ceph/ceph-client.admin.69860.log.gz -5 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr: --verbose -- /var/log/ceph/ceph-mds.a.log 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.109670.log.gz 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110087.log 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63860.log: /var/log/ceph/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43959.log 2024-05-09T05:24:24.110 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.63860.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40565.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40607.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110087.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55784.log 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 88.2% -- replaced with /var/log/ceph/ceph-client.admin.40649.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.110087.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.40775.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5/var/log/ceph/ceph-client.admin.43959.log: --verbose -- /var/log/ceph/ceph-client.admin.42671.log 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.41123.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41113.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41444.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55784.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.100954.log 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41457.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.55784.log.gz 2024-05-09T05:24:24.111 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41460.log.gz 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.42671.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48803.log 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.49409.log.gz 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.100954.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40943.log 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.100954.log.gz 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48803.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40397.log 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39990.log 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40943.log: /var/log/ceph/ceph-client.admin.40397.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.52662.log 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39990.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.87528.log 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.52662.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.108928.log 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr: 90.2% -- replaced with /var/log/ceph/ceph-client.admin.50791.log.gz 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.87528.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52662.log.gz 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43693.log 2024-05-09T05:24:24.112 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.87528.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50092.log 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.108928.log: /var/log/ceph/ceph-client.admin.43693.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.106908.log 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.108928.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50092.log: 89.8% -- replaced with /var/log/ceph/ceph-client.admin.48407.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.48449.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.48491.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48365.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53174.log 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.50139.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.106908.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44715.log 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.106908.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53174.log: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.50876.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.53174.log.gz 2024-05-09T05:24:24.113 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.50395.log.gz 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.50223.log.gz 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.48845.log 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.52287.log.gz 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr: 87.9% -- replaced with /var/log/ceph/ceph-client.admin.52245.log.gz 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53714.log 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.44715.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41993.log 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.48845.log: /var/log/ceph/ceph-client.admin.53714.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110958.log 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.41993.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.53714.log.gz 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.44044.log 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110393.log 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51375.log.gz 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110958.log: /var/log/ceph/ceph-client.admin.44044.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.102277.log 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.110958.log.gz 2024-05-09T05:24:24.114 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110393.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.64180.log 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.35777.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.110393.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.51662.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.55619.log 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.102277.log: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.52161.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49367.log 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.64180.log: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.49157.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.102277.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.55619.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.99596.log 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.64180.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.55619.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40733.log 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49367.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.50961.log.gz 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.99596.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51231.log 2024-05-09T05:24:24.115 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.99596.log.gz 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.66584.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40733.log: /var/log/ceph/ceph-client.admin.51231.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51578.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.66584.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.54369.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51578.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.79634.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.66584.log.gz 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.49840.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.54369.log: /var/log/ceph/ceph-client.admin.79634.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.39909.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr: 90.0% -- replaced with /var/log/ceph/ceph-client.admin.54369.log.gz 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.49840.log: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.79634.log.gz 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.69773.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.63070.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.39909.log: /var/log/ceph/ceph-client.admin.69773.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.84307.log 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.69773.log.gz 2024-05-09T05:24:24.116 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.63070.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40439.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.63070.log.gz 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.84307.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.53528.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40439.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.110308.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.53528.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40523.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr: 93.7% -- replaced with /var/log/ceph/ceph-client.admin.84307.log.gz 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.53528.log.gz 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.110308.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.51968.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.110308.log.gz 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.40859.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40523.log: gzip -5 --verbose/var/log/ceph/ceph-client.admin.51968.log: -- /var/log/ceph/ceph-client.admin.51063.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.40859.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.50181.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.51063.log: gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.41797.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.43609.log 2024-05-09T05:24:24.117 INFO:teuthology.orchestra.run.smithi001.stderr:/var/log/ceph/ceph-client.admin.50181.log: /var/log/ceph/ceph-client.admin.41797.log: /var/log/ceph/ceph-client.admin.43609.log: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.50479.log.gz 2024-05-09T05:24:24.118 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.46809.log.gz 2024-05-09T05:24:24.119 INFO:teuthology.orchestra.run.smithi196.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.d.log.gz 2024-05-09T05:24:24.124 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.42388.log.gz 2024-05-09T05:24:24.124 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.43693.log.gz 2024-05-09T05:24:24.124 INFO:teuthology.orchestra.run.smithi001.stderr: 90.9% -- replaced with /var/log/ceph/ceph-client.admin.42737.log.gz 2024-05-09T05:24:24.124 INFO:teuthology.orchestra.run.smithi001.stderr: 91.1% -- replaced with /var/log/ceph/ceph-client.admin.42671.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.43959.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.44841.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.44883.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.39990.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40105.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40397.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 87.6% -- replaced with /var/log/ceph/ceph-client.admin.46508.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.48533.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.48761.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 88.4% -- replaced with /var/log/ceph/ceph-client.admin.41993.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.49367.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 87.8% -- replaced with /var/log/ceph/ceph-client.admin.40859.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 87.5% 87.7% -- replaced with /var/log/ceph/ceph-client.admin.40901.log.gz -- replaced with /var/log/ceph/ceph-client.admin.40943.log.gz 2024-05-09T05:24:24.125 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.41459.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.41789.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.41794.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.41785.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.41796.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.41797.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.51063.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.50092.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49840.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49882.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.50181.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.52536.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.51231.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.48659.log.gz 2024-05-09T05:24:24.126 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.42119.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.42169.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 89.6% -- replaced with /var/log/ceph/ceph-client.admin.44356.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.44314.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44272.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 89.3% -- replaced with /var/log/ceph/ceph-client.admin.44441.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 92.0% -- replaced with /var/log/ceph/ceph-client.admin.44398.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.44501.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44673.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% 90.3% -- replaced with /var/log/ceph/ceph-client.admin.44631.log.gz -- replaced with /var/log/ceph/ceph-client.admin.44543.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 90.1% -- replaced with /var/log/ceph/ceph-client.admin.44585.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 87.4% -- replaced with /var/log/ceph/ceph-client.admin.44715.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.44757.log.gz 2024-05-09T05:24:24.127 INFO:teuthology.orchestra.run.smithi001.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.44799.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.45555.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 91.9% -- replaced with /var/log/ceph/ceph-client.admin.49696.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.51782.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 91.8% -- replaced with /var/log/ceph/ceph-client.admin.51578.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.51824.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 89.9% -- replaced with /var/log/ceph/ceph-client.admin.51884.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 89.1% -- replaced with /var/log/ceph/ceph-client.admin.51926.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.51968.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 89.5% -- replaced with /var/log/ceph/ceph-client.admin.52371.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 88.6% -- replaced with /var/log/ceph/ceph-client.admin.52413.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 88.1% -- replaced with /var/log/ceph/ceph-client.admin.42262.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.43609.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 91.0% -- replaced with /var/log/ceph/ceph-client.admin.42605.log.gz 2024-05-09T05:24:24.128 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.44044.log.gz 2024-05-09T05:24:24.129 INFO:teuthology.orchestra.run.smithi001.stderr: 88.7% -- replaced with /var/log/ceph/ceph-client.admin.40439.log.gz 2024-05-09T05:24:24.129 INFO:teuthology.orchestra.run.smithi001.stderr: 88.5% -- replaced with /var/log/ceph/ceph-client.admin.40523.log.gz 2024-05-09T05:24:24.129 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.40733.log.gz 2024-05-09T05:24:24.144 INFO:teuthology.orchestra.run.smithi001.stderr: 87.7% -- replaced with /var/log/ceph/ceph-client.admin.48803.log.gz 2024-05-09T05:24:24.145 INFO:teuthology.orchestra.run.smithi001.stderr: 89.0% -- replaced with /var/log/ceph/ceph-client.admin.48845.log.gz 2024-05-09T05:24:24.145 INFO:teuthology.orchestra.run.smithi001.stderr: 89.2% -- replaced with /var/log/ceph/ceph-client.admin.48887.log.gz 2024-05-09T05:24:24.145 INFO:teuthology.orchestra.run.smithi001.stderr: 88.9% -- replaced with /var/log/ceph/ceph-client.admin.48929.log.gz 2024-05-09T05:24:24.145 INFO:teuthology.orchestra.run.smithi001.stderr: 88.8% -- replaced with /var/log/ceph/ceph-client.admin.49013.log.gz 2024-05-09T05:24:24.145 INFO:teuthology.orchestra.run.smithi001.stderr: 89.7% -- replaced with /var/log/ceph/ceph-client.admin.48971.log.gz 2024-05-09T05:24:24.145 INFO:teuthology.orchestra.run.smithi001.stderr: 88.3% -- replaced with /var/log/ceph/ceph-client.admin.49924.log.gz 2024-05-09T05:24:24.145 INFO:teuthology.orchestra.run.smithi001.stderr: 89.4% -- replaced with /var/log/ceph/ceph-client.admin.50563.log.gz 2024-05-09T05:24:24.154 INFO:teuthology.orchestra.run.smithi196.stderr: 91.6% -- replaced with /var/log/ceph/ceph-mgr.x.log.gz 2024-05-09T05:24:24.192 INFO:teuthology.orchestra.run.smithi001.stderr: 88.0% -- replaced with /var/log/ceph/ceph-client.admin.39909.log.gz 2024-05-09T05:24:24.259 INFO:teuthology.orchestra.run.smithi001.stderr: 94.3% -- replaced with /var/log/ceph/ceph-mgr.y.log.gz 2024-05-09T05:24:24.345 INFO:teuthology.orchestra.run.smithi196.stderr: 91.8% -- replaced with /var/log/ceph/ceph-mon.c.log.gz 2024-05-09T05:24:24.370 INFO:teuthology.orchestra.run.smithi001.stderr: 93.6% -- replaced with /var/log/ceph/ceph-mds.a.log.gz 2024-05-09T05:24:24.407 INFO:teuthology.orchestra.run.smithi196.stderr: 91.4% -- replaced with /var/log/ceph/ceph-mon.b.log.gz 2024-05-09T05:24:25.001 INFO:teuthology.orchestra.run.smithi001.stderr: 93.4% -- replaced with /var/log/ceph/ceph-mds.c.log.gz 2024-05-09T05:24:25.038 INFO:teuthology.orchestra.run.smithi001.stderr: 93.5% -- replaced with /var/log/ceph/ceph-mds.e.log.gz 2024-05-09T05:24:26.787 INFO:teuthology.orchestra.run.smithi001.stderr: 90.8% -- replaced with /var/log/ceph/ceph-mon.a.log.gz 2024-05-09T05:24:29.106 INFO:teuthology.orchestra.run.smithi196.stderr: 93.3% -- replaced with /var/log/ceph/ceph-mds.f.log.gz 2024-05-09T05:24:35.829 INFO:teuthology.orchestra.run.smithi001.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.2.log.gz 2024-05-09T05:24:37.326 INFO:teuthology.orchestra.run.smithi196.stderr: 93.0% -- replaced with /var/log/ceph/ceph-osd.4.log.gz 2024-05-09T05:24:38.119 INFO:teuthology.orchestra.run.smithi196.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.6.log.gz 2024-05-09T05:24:40.284 INFO:teuthology.orchestra.run.smithi001.stderr: 93.3% -- replaced with /var/log/ceph/ceph-osd.1.log.gz 2024-05-09T05:24:44.211 INFO:teuthology.orchestra.run.smithi196.stderr: 93.2% -- replaced with /var/log/ceph/ceph-osd.7.log.gz 2024-05-09T05:24:44.271 INFO:teuthology.orchestra.run.smithi196.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.5.log.gz 2024-05-09T05:24:44.273 INFO:teuthology.orchestra.run.smithi196.stderr: 2024-05-09T05:24:44.274 INFO:teuthology.orchestra.run.smithi196.stderr:real 0m20.331s 2024-05-09T05:24:44.274 INFO:teuthology.orchestra.run.smithi196.stderr:user 1m11.550s 2024-05-09T05:24:44.274 INFO:teuthology.orchestra.run.smithi196.stderr:sys 0m2.632s 2024-05-09T05:24:45.476 INFO:teuthology.orchestra.run.smithi001.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.0.log.gz 2024-05-09T05:24:49.483 INFO:teuthology.orchestra.run.smithi001.stderr: 93.1% -- replaced with /var/log/ceph/ceph-osd.3.log.gz 2024-05-09T05:27:27.302 INFO:teuthology.orchestra.run.smithi001.stderr: 95.6% -- replaced with /var/log/ceph/ceph-client.0.43333.log.gz 2024-05-09T05:27:27.303 INFO:teuthology.orchestra.run.smithi001.stderr: 2024-05-09T05:27:27.304 INFO:teuthology.orchestra.run.smithi001.stderr:real 3m3.386s 2024-05-09T05:27:27.304 INFO:teuthology.orchestra.run.smithi001.stderr:user 3m42.923s 2024-05-09T05:27:27.304 INFO:teuthology.orchestra.run.smithi001.stderr:sys 0m9.055s 2024-05-09T05:27:27.304 INFO:tasks.ceph:Archiving logs... 2024-05-09T05:27:27.304 DEBUG:teuthology.misc:Transferring archived files from smithi001:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/remote/smithi001/log 2024-05-09T05:27:27.305 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T05:27:58.858 DEBUG:teuthology.misc:Transferring archived files from smithi196:/var/log/ceph to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/remote/smithi196/log 2024-05-09T05:27:58.860 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T05:28:09.545 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T05:28:09.555 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T05:28:09.555 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T05:28:09.559 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T05:28:09.652 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T05:28:09.662 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T05:28:09.662 DEBUG:teuthology.orchestra.run.smithi001:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T05:28:09.666 DEBUG:teuthology.orchestra.run.smithi196:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T05:28:09.681 INFO:teuthology.orchestra.run.smithi001.stderr:bash: line 1: ntpq: command not found 2024-05-09T05:28:09.682 INFO:teuthology.orchestra.run.smithi196.stderr:bash: line 1: ntpq: command not found 2024-05-09T05:28:09.687 INFO:teuthology.orchestra.run.smithi196.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T05:28:09.687 INFO:teuthology.orchestra.run.smithi196.stdout:=============================================================================== 2024-05-09T05:28:09.687 INFO:teuthology.orchestra.run.smithi196.stdout:^* hv01.front.sepia.ceph.com 3 6 377 64 +3028us[+3036us] +/- 64ms 2024-05-09T05:28:09.687 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 64 -1638us[-1629us] +/- 55ms 2024-05-09T05:28:09.688 INFO:teuthology.orchestra.run.smithi196.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 -1673us[-1673us] +/- 91ms 2024-05-09T05:28:09.688 INFO:teuthology.orchestra.run.smithi196.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:28:09.695 INFO:teuthology.orchestra.run.smithi001.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T05:28:09.696 INFO:teuthology.orchestra.run.smithi001.stdout:=============================================================================== 2024-05-09T05:28:09.696 INFO:teuthology.orchestra.run.smithi001.stdout:^* hv01.front.sepia.ceph.com 3 6 377 0 +3052us[+2894us] +/- 65ms 2024-05-09T05:28:09.696 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 1 -1603us[-1761us] +/- 56ms 2024-05-09T05:28:09.696 INFO:teuthology.orchestra.run.smithi001.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 0 -1660us[-1819us] +/- 91ms 2024-05-09T05:28:09.696 INFO:teuthology.orchestra.run.smithi001.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T05:28:09.697 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T05:28:09.707 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T05:28:09.707 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T05:28:09.717 DEBUG:teuthology.orchestra.run.smithi001:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T05:28:09.854 DEBUG:teuthology.orchestra.run.smithi196:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T05:28:09.922 DEBUG:teuthology.orchestra.run.smithi001:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T05:28:09.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:28:09.956 DEBUG:teuthology.orchestra.run.smithi196:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T05:28:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:28:09.986 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T05:28:09.997 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T05:28:10.008 INFO:teuthology.task.internal:Duration was 1742.775112 seconds 2024-05-09T05:28:10.008 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T05:28:10.022 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T05:28:10.023 DEBUG:teuthology.orchestra.run.smithi001:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T05:28:10.026 DEBUG:teuthology.orchestra.run.smithi196:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T05:28:10.067 INFO:teuthology.orchestra.run.smithi196.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T05:28:10.088 INFO:teuthology.orchestra.run.smithi001.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T05:28:10.485 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T05:28:10.486 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi001.front.sepia.ceph.com 2024-05-09T05:28:10.486 DEBUG:teuthology.orchestra.run.smithi001:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T05:28:10.511 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi196.front.sepia.ceph.com 2024-05-09T05:28:10.511 DEBUG:teuthology.orchestra.run.smithi196:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T05:28:10.535 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T05:28:10.535 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T05:28:10.553 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T05:28:10.602 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T05:28:10.602 DEBUG:teuthology.orchestra.run.smithi001:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T05:28:10.624 DEBUG:teuthology.orchestra.run.smithi196:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T05:28:10.751 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T05:28:10.761 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T05:28:10.762 DEBUG:teuthology.orchestra.run.smithi001:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T05:28:10.791 DEBUG:teuthology.orchestra.run.smithi196:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T05:28:10.816 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T05:28:10.827 DEBUG:teuthology.orchestra.run.smithi001:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T05:28:10.833 DEBUG:teuthology.orchestra.run.smithi196:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T05:28:10.861 INFO:teuthology.orchestra.run.smithi001.stdout:kernel.core_pattern = core 2024-05-09T05:28:10.880 INFO:teuthology.orchestra.run.smithi196.stdout:kernel.core_pattern = core 2024-05-09T05:28:10.905 DEBUG:teuthology.orchestra.run.smithi001:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T05:28:10.931 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:28:10.932 DEBUG:teuthology.orchestra.run.smithi196:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T05:28:10.959 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T05:28:10.959 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T05:28:10.970 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T05:28:10.971 DEBUG:teuthology.misc:Transferring archived files from smithi001:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/remote/smithi001 2024-05-09T05:28:10.971 DEBUG:teuthology.orchestra.run.smithi001:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T05:28:11.009 DEBUG:teuthology.misc:Transferring archived files from smithi196:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/pdonnell-2024-05-08_22:06:20-fs-wip-pdonnell-testing-20240508.183908-debug-distro-default-smithi/7699092/remote/smithi196 2024-05-09T05:28:11.009 DEBUG:teuthology.orchestra.run.smithi196:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T05:28:11.042 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T05:28:11.042 DEBUG:teuthology.orchestra.run.smithi001:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T05:28:11.045 DEBUG:teuthology.orchestra.run.smithi196:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T05:28:11.092 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T05:28:11.104 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T05:28:11.104 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T05:28:11.115 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T05:28:11.116 DEBUG:teuthology.orchestra.run.smithi001:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T05:28:11.118 DEBUG:teuthology.orchestra.run.smithi196:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T05:28:11.131 INFO:teuthology.orchestra.run.smithi001.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 05:28 /home/ubuntu/cephtest 2024-05-09T05:28:11.147 INFO:teuthology.orchestra.run.smithi196.stdout: 266049 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 05:28 /home/ubuntu/cephtest 2024-05-09T05:28:11.149 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T05:28:11.160 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T05:28:11.185 INFO:teuthology.run:Summary data: description: fs/thrash/workloads/{begin/{0-install 1-ceph 2-logrotate 3-modules} clusters/1a5s-mds-1c-client conf/{client mds mgr mon osd} distro/{centos_latest} mount/fuse msgr-failures/none objectstore-ec/bluestore-bitmap overrides/{client-shutdown frag ignorelist_health ignorelist_wrongly_marked_down pg_health prefetch_dirfrags/no prefetch_dirfrags/yes prefetch_entire_dirfrags/no prefetch_entire_dirfrags/yes races session_timeout thrashosds-health} ranks/5 tasks/{1-thrash/mon 2-workunit/suites/ffsb}} duration: 1742.7751116752625 flavor: default owner: scheduled_pdonnell@teuthology success: true 2024-05-09T05:28:11.185 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T05:28:11.271 INFO:teuthology.run:pass